Skip to content

test(chaos): strengthen fork and liveness checks - #4676

Open
neekolas wants to merge 3 commits into
mainfrom
codex/chaos-suite-checks
Open

neekolas wants to merge 3 commits into
mainfrom
codex/chaos-suite-checks

Conversation

@neekolas

@neekolas neekolas commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The chaos suite could lose useful contention as membership churn exhausted groups, and unresolved obligations or changing error text could weaken its liveness verdicts.

Rotate writers across independent devices, add same-inbox contention, restore multi-inbox groups after churn, and reject normal rounds with no group writes. Keep unresolved work inconclusive at the run limit and preserve nested blocking causes and unpublished intents in bounded inspection output.

Validation on main 1e27174993:

  • 84 chaos tests and 31 MLS membership recovery tests passed. New tests were verified against weakened implementations.
  • A 50-round baseline passed.
  • A 30-minute all-fault run passed: 139 rounds, 3,336 operations, 210 fault windows covering all 14 fault types; no forks or bricked groups. Membership restoration ran before contention resumed.

This changes only the chaos suite. The core fixes for #4664 are already on main.

Further audit found two pre-existing timing gaps for follow-up: the roll-call retry loop reuses elapsed ages across serial RPCs, and checkpoint recovery can compose a 30.25-second stream-recovery budget and a 60-second checkpoint under a 90-second parent RPC deadline. These boundary cases were not reproduced in acceptance runs. The hosted baseline passed all 50 rounds; its slowest round was 50 seconds.

@macroscopeapp

macroscopeapp Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — This PR materially changes chaos-runner recovery, device ordering/contention, liveness budgets, and inconclusive run-limit behavior rather than only adding tests. The changes are isolated from production SDK paths, but recovery and ordering guarantees require human review.

No code changes detected at 034b62e. Prior analysis still applies.

You can add or adjust custom eligibility rules. Learn more.

@neekolas
neekolas force-pushed the codex/chaos-suite-checks branch from aa83187 to 795e598 Compare October 23, 2026 14:19
)
});
ensure!(
recovering || has_group_writes,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: This only validates that an Add was scheduled, not that it succeeded or restored a multi-inbox group. Failed operations are counted as ordinary errors; with all groups cold and the cap reached, the remaining targetless work can be sync-only, so repeated failed Adds can still let a baseline reach its limit without resuming contention. Could the recovery path require a successful Add and confirm a hot group in the next snapshot, or fail as a harness error if repair does not take effect?

Posted via Macroscope — Code Quality

@neekolas
neekolas force-pushed the codex/chaos-suite-checks branch from 795e598 to 034b62e Compare October 26, 2026 23:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant