Repository navigation
Conversation
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The llm-d report adapter checks stack metadata only in the first stage file of each repetition. A later stage can name a different backend or component version and still contribute metrics to a comparison labeled with the first stage's backend.
Check the existing canonical stack metadata in every stage file before aggregation. The error identifies the mismatched file. Metric definitions, duplicate-rate warm-up handling and output formats remain unchanged.
Regression fixtures cover backend drift in warm-up and measured stages, non-engine component drift, and drift in a later repetition. A consistent two-repetition fixture checks median output rates, warm-up exclusion, the backend label and Prism bundles. Five regression subcases failed with the original adapter and pass with the fix.
Validation
git diff --checkpassed.These checks ran locally on macOS with isolated Python 3.12.13. Shell tests use mocks and fail-closed tool-presence stubs. No cloud resources, GPUs or live clusters were provisioned.
The existing main-branch Inference tooling run fails during the shell-test step before Python tests. That hosted CI failure is separate from the passing local results above.