Skip to content

Add workflows for performance testing - #1297

Merged
Joao-Dionisio merged 5 commits into
masterfrom
benchmark-workflow
Nov 20, 2026
Merged

Joao-Dionisio merged 5 commits into
masterfrom
benchmark-workflow

Conversation

@Joao-Dionisio

@Joao-Dionisio Joao-Dionisio commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Typing /benchmark starts the runs. Right now, they take around 15 minutes and should be discussed in a separate PR, this one is just for the infrastructure.

image

One can run specific tests and set time limits (here just the tests under model and a 10s time limit). A comment is automatically posted and updated every 5 minutes until the run finishes. This comment points to the comment that originated the run and to the run itself.

image image image

Both base and release are cached (on PR merges and releases) to reduce building time in future runs. Further, for these two, only the timing experiments are run, the deterministic results (instructions from callgrind) are stored.

Significant regressions and improvements are signaled and presented in a new comment once the run finishes. The comparison to the latest release is minimized at the bottom.

image

You can see how it would work in here. (Pay more attention to the latest runs, since the others correspond to earlier versions).

@Joao-Dionisio Joao-Dionisio added this to the v7.0.0 milestone Nov 17, 2026
@Joao-Dionisio

Joao-Dionisio commented Nov 18, 2026 •

Copy link
Copy Markdown
Member Author

@svigerske as our resident jenkins expert, is there some functionality you think should be added? I don't mean a review, just what you would expect something like this to provide that it currently isn't from the PR description :)

EDIT: I'm adding the functionality to compare PySCIPOpts with different SCIP versions in a bit.
EDIT2: Performance variability is quite difficult to mitigate here, that's why I'm counting instructions in almost all tests.

@Joao-Dionisio Joao-Dionisio modified the milestones: v7.0.0, v6.4.0 Nov 19, 2026
@Joao-Dionisio

Copy link
Copy Markdown
Member Author

Well, since it's opt in only, I suppose there's not much harm in merging.

@Joao-Dionisio
Joao-Dionisio merged commit cd4eb5c into master Nov 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant