fix(benchmarks): wait for scripted event delivery - #6487
Conversation
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f5c90d1613
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
philazar
left a comment
There was a problem hiding this comment.
lgtm. Agent reviewer found no bugs and I didnt find anything either
wpfleger96
left a comment
There was a problem hiding this comment.
Reviewed the exact head and traced the receipt contract through the benchmark runtime, completed-batch delivery accounting, the completed-before-control race, and successful non-cancelling steer acknowledgments. The scripted-event gate now requires every expected event receipt plus a terminal turn, while ordinary solo tasks keep the existing first-turn completion behavior. CI is green, and a mutation run that restored the premature ends > 0 shortcut failed both new regressions (2 failed, 85 passed), confirming the tests observe the fix at the intended seam.
…uctions-prompt * origin/main: fix(benchmarks): wait for scripted event delivery (#6487) Polish mobile channel navigation and message sends (#6488) Clarify huddle message destination (#6496) feat(archive): add observer-frame retention schema and gated DB adapter (#5719) fix(desktop): restore human barge-in over agent TTS in huddles (#6431) chore(release): release Buzz Desktop version 0.5.18 (#6489) fix(desktop): simplify duplicate agent provenance (#6401) test(benchmarks): expand Buzz-native dataset (#6448) fix(desktop): sender names in notifications + macOS click-through routing (#6427) docs: clarify two-layer moderation ownership (#6481) Fix mobile thread tail and iOS channel header (#6399) chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392) polish(desktop): finish Projects navigation and context chrome (#6429) fix(desktop): clarify add agents channel action (#6374) Repair stale large channel roster snapshots (#6251) Signed-off-by: Salman Mohammed <smohammed@squareup.com>
* origin/main: (88 commits) docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946) fix(benchmarks): wait for scripted event delivery (#6487) Polish mobile channel navigation and message sends (#6488) Clarify huddle message destination (#6496) feat(archive): add observer-frame retention schema and gated DB adapter (#5719) fix(desktop): restore human barge-in over agent TTS in huddles (#6431) chore(release): release Buzz Desktop version 0.5.18 (#6489) fix(desktop): simplify duplicate agent provenance (#6401) test(benchmarks): expand Buzz-native dataset (#6448) fix(desktop): sender names in notifications + macOS click-through routing (#6427) docs: clarify two-layer moderation ownership (#6481) Fix mobile thread tail and iOS channel header (#6399) chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392) polish(desktop): finish Projects navigation and context chrome (#6429) fix(desktop): clarify add agents channel action (#6374) Repair stale large channel roster snapshots (#6251) feat(desktop-messages): show compact Buzz link metadata (#6252) feat(workflows): reply in-thread from send_message action (#6178) perf(desktop): split discover_acp_providers into cheap and forced paths (#6330) fix(desktop): restore recent channel sorting (#6402) ... Co-authored-by: Bradley Axen <baxen@squareup.com> Signed-off-by: Bradley Axen <baxen@squareup.com> # Conflicts: # crates/buzz-acp/src/base_prompt.md
* origin/main: (65 commits) fix(composer): preserve caret when inserting mentions mid-message (#6531) chore(deps): update rust crate async-trait to v0.1.92 (#6094) chore(deps): update dependency sonner to v2.0.8 (#6093) chore(deps): update rust crate http-body-util to v0.1.4 (#5452) chore(deps): update rust crate http to v1.4.2 (#5451) chore(deps): update rust crate futures-util to v0.3.33 (#5448) chore(deps): update rust crate futures to v0.3.33 (#5445) chore(deps): update dependency @tauri-apps/api to v2.11.1 (#5444) chore(deps): update ubuntu:24.04 docker digest to 561618e (#5442) chore(deps): update swatinem/rust-cache digest to 6323deb (#5441) fix(desktop): restore true zoom by scaling the root rem (#6514) chore(desktop): drop unused ORIGINAL_CONTENT from empty-edit-delete spec (#6517) feat(workflows): clarify workflow setup and activation (#6470) perf(desktop): stop the Projects fan refetching on re-entry and running after leave (#6458) perf(desktop): keep the member roster off the channel-switch path (#6456) fix(deletion): allow IRSA S3 credentials (#6495) docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946) fix(benchmarks): wait for scripted event delivery (#6487) Polish mobile channel navigation and message sends (#6488) Clarify huddle message destination (#6496) ... Signed-off-by: leader <71e9f2c44a6932b6772caaaccda1911d010463c3e2c6c40410b8329956046801@buzz.block.builderlab.xyz> Co-authored-by: Tom Brow <tomb@block.xyz> Signed-off-by: Tom Brow <tomb@block.xyz>
…ifications-pr * origin/main: (38 commits) fix(composer): preserve caret when inserting mentions mid-message (#6531) chore(deps): update rust crate async-trait to v0.1.92 (#6094) chore(deps): update dependency sonner to v2.0.8 (#6093) chore(deps): update rust crate http-body-util to v0.1.4 (#5452) chore(deps): update rust crate http to v1.4.2 (#5451) chore(deps): update rust crate futures-util to v0.3.33 (#5448) chore(deps): update rust crate futures to v0.3.33 (#5445) chore(deps): update dependency @tauri-apps/api to v2.11.1 (#5444) chore(deps): update ubuntu:24.04 docker digest to 561618e (#5442) chore(deps): update swatinem/rust-cache digest to 6323deb (#5441) fix(desktop): restore true zoom by scaling the root rem (#6514) chore(desktop): drop unused ORIGINAL_CONTENT from empty-edit-delete spec (#6517) feat(workflows): clarify workflow setup and activation (#6470) perf(desktop): stop the Projects fan refetching on re-entry and running after leave (#6458) perf(desktop): keep the member roster off the channel-switch path (#6456) fix(deletion): allow IRSA S3 credentials (#6495) docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946) fix(benchmarks): wait for scripted event delivery (#6487) Polish mobile channel navigation and message sends (#6488) Clarify huddle message destination (#6496) ... Signed-off-by: Tom Brow <tomb@block.xyz>
## Why The Buzz-native runtime could stop a solo agent after one turn even when a scripted follow-up event was queued or already running. That could turn a harness timing race into a benchmark failure. Follow-up to block#6448. ## What - Treat solo tasks with scripted events separately from ordinary one-turn tasks. - Wait until turn counts and authored message IDs remain idle and unchanged for several polls before stopping the agent. - Remove the message-count shortcut that could stop an active turn. - Add focused regressions for a delayed follow-up turn and an already-running turn with a `DONE:` message. ## Testing - `pytest -q tests/test_container_runtime.py -k "solo_turn_end or scripted_events"` — 3 passed - `ruff check` on the three touched Python files - `ruff format --check` on the three touched Python files - Local `buzz-native-solo-luna.yaml` Harbor run, one attempt per affected task: - `cross-thread-requests` — reward 1.0; all 6 task-specific checks passed - `interleaved-agent-reports` — reward 1.0; all 7 task-specific checks passed - 2 completed trials, 0 exceptions, 56 seconds total ## Risk Low. This only changes completion detection for solo benchmark tasks with scripted events. Ordinary solo tasks retain immediate completion after their first turn ends. --- **Update Aug 21, 13:07 EDT:** Replaced the fixed settle delay with exact delivery receipts after [review feedback](block#6487 (comment)). - ACP now records the event IDs delivered by each completed turn; the runtime also recognizes existing successful-steer receipts. - Scripted trials stop only after every expected event ID is acknowledged and no turn is active. Missing receipts wait for the trial budget instead of producing partial evidence. - TDD regression delayed delivery beyond the old five-poll boundary and failed before the implementation; the receipt parser and Rust receipt format are pinned independently. - Focused verification: 31 container-runtime tests passed, 6 ACP delivery tests passed, plus Ruff, rustfmt, and Clippy. - Receipt-gated Luna run: 2 completed trials and 0 exceptions. `interleaved-agent-reports` scored 1.0. `cross-thread-requests` completed both calculations and thread isolation but Luna omitted the user mention on ALPHA, so that model-output dimension scored 0. - Risk remains low: the ACP production change adds delivery receipt logging and restores a missing turn-end log on a completed-before-control race; it does not change queue dispatch behavior. Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
Why
The Buzz-native runtime could stop a solo agent after one turn even when a scripted follow-up event was queued or already running. That could turn a harness timing race into a benchmark failure.
Follow-up to #6448.
What
DONE:message.Testing
pytest -q tests/test_container_runtime.py -k "solo_turn_end or scripted_events"— 3 passedruff checkon the three touched Python filesruff format --checkon the three touched Python filesbuzz-native-solo-luna.yamlHarbor run, one attempt per affected task:cross-thread-requests— reward 1.0; all 6 task-specific checks passedinterleaved-agent-reports— reward 1.0; all 7 task-specific checks passedRisk
Low. This only changes completion detection for solo benchmark tasks with scripted events. Ordinary solo tasks retain immediate completion after their first turn ends.
Update Aug 21, 13:07 EDT: Replaced the fixed settle delay with exact delivery receipts after review feedback.
interleaved-agent-reportsscored 1.0.cross-thread-requestscompleted both calculations and thread isolation but Luna omitted the user mention on ALPHA, so that model-output dimension scored 0.Generated with Codex