Skip to content

reject mixed stack metadata within reporting repetitions - #7

Open
anxkhn wants to merge 1 commit into
agentgateway:mainfrom
anxkhn:fix/report-stage-stack-consistency
Open

anxkhn wants to merge 1 commit into
agentgateway:mainfrom
anxkhn:fix/report-stage-stack-consistency

Conversation

@anxkhn

@anxkhn anxkhn commented Oct 12, 2026 •

Copy link
Copy Markdown

Summary

The llm-d report adapter checks stack metadata only in the first stage file of each repetition. A later stage can name a different backend or component version and still contribute metrics to a comparison labeled with the first stage's backend.

Check the existing canonical stack metadata in every stage file before aggregation. The error identifies the mismatched file. Metric definitions, duplicate-rate warm-up handling and output formats remain unchanged.

Regression fixtures cover backend drift in warm-up and measured stages, non-engine component drift, and drift in a later repetition. A consistent two-repetition fixture checks median output rates, warm-up exclusion, the backend label and Prism bundles. Five regression subcases failed with the original adapter and pass with the fix.

Validation

  • Reporting suite: 9 tests passed, including Markdown, PNG, CSV and Prism checks.
  • llm-d Python suite: 15 tests passed.
  • All five mocked shell harness suites passed.
  • ShellCheck and Bash syntax passed for all 15 shell files.
  • Python syntax passed for all 18 Python files.
  • Standalone repository reference check and git diff --check passed.

These checks ran locally on macOS with isolated Python 3.12.13. Shell tests use mocks and fail-closed tool-presence stubs. No cloud resources, GPUs or live clusters were provisioned.

The existing main-branch Inference tooling run fails during the shell-test step before Python tests. That hosted CI failure is separate from the passing local results above.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant