Skip to content

perf(bench): add a production readEvents scaling benchmark #2476

Backlog

Description

@spboyer

Summary

Add one production readEvents scaling workload to the existing performance runner. The current built-in configLoad, charterCompile, routing, model-selection and export/import cases largely simulate JSON/string/regex work rather than invoke the corresponding product implementations.

Proposed primary owner: unit-contract-tester.

Evidence and scope

Audited revision: 47082a81799710c3e457b594123d190c00596ebf, observed 2026-11-12.

  • Built-in benchmark cases were exercised for three baseline runs of 1,000 iterations per case. Their output is not evidence of equivalent production-path performance.
  • Existing entry points include scripts/run-benchmarks.mjs and scripts/measure-cold-start.mjs. No workflow invocation of those performance runners was found at the audited revision. Smoke/output-shape tests do not demonstrate sensitivity to realistic product regressions.
  • The local packaged build could not run: TypeScript/Vitest were unavailable and locked dependency restoration returned a configured-feed E404 for hono-4.13.17.tgz. Missing dist also blocked the built benchmark and cold-start runners. This is a measurement limitation, not evidence that the CLI is slow.

The audit's fallback used Node's built-in TypeScript transformation and isolated relative-import resolution. Those function-level diagnostics exclude import/setup time and must not certify packaged startup, application throughput, total allocations, token efficiency, or whole-agent-session memory.

Scope

Parent: github/squad#2481.

One production API only: LocalSquadBackend.readEvents with the local file-state backend. Add a fixture/case to the existing runner foundation; do not build a new benchmark framework. Packaged help/version startup is split into github/squad#2482.

Out of scope: config, charter, routing, model selection, memory search, shell persistence, backend optimization, CI gates, nightly automation and cross-platform rollout. The optimization belongs to github/squad#2479.

Measurable acceptance and tests

  • Run the built production method, not a JSON/string simulation or source-loader shim. Fixture sizes: 100/1,000/10,000 valid events; limit: 1 with one-event-tail and empty-tail cursors, six cases total.
  • For each case, collect three rounds of ten timed observations after two untimed warmups. Exclude fixture/import setup and retain all 30 raw times plus p50/p95, exact SHA, Node/platform/build mode and cache policy.
  • Assert event count, final sequence, cursor and resumability in the fixtures. A deliberately broken result must fail the run rather than produce a successful timing.
  • Deterministic output tests cover sample counts/quantiles, nonfinite or missing data, fixture failure and a nonzero runner exit. Existing runner consumers/output contracts remain compatible.
  • Exercise the comparison/reporting path with a calibrated test-only slowdown of at least 2x; its reported median ratio must exceed the declared comparison threshold. This proves regression sensitivity, not production optimization benefit; do not add a blocking CI threshold in this issue.
  • Check in the portable fixture and document one exact build/run command with a passing sample artifact. Missing dependencies/dist remain a failing or blocked measurement, never a successful substitute.

Done means one trustworthy six-case production benchmark and its runner/result tests, not coverage of every product path.

Related work and non-goals

Follow-up to the already-merged runner foundation github/squad#1111, not another request to add it. github/squad#2449 is a distinct model-quality paired A/B workload.

Methodology: dotnet/aspnetcore#70219, microsoft/waza#846, and the packaged-path correction in microsoft/aspire#16704. No product optimization or end-to-end speedup has been demonstrated by this issue.

Activity

  1. changed the title [-]perf: benchmark real production paths and retain regression evidence[/-] [+]perf(bench): add a production readEvents scaling benchmark[/+] on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions