Repository navigation
Enforce D41 hash-chained latch journal, pre-parse raw response persistence, and shared-v2 closure - #5212
Draft
realmarcin wants to merge 55 commits into
Draft
Enforce D41 hash-chained latch journal, pre-parse raw response persistence, and shared-v2 closure#5212realmarcin wants to merge 55 commits into
realmarcin wants to merge 55 commits into
Conversation
Owner decision D33 (2026-10-25): admission check now. At the 128k report
cap (D28) a 1M window leaves 872k tokens of input. Kids First's measured
regate input was already 880,265 tokens at the 96k cap, and the regate
also carries the report it rejected. The receipt completion and the typed
audit stages count before they send; the report family did not.
Why: a request that cannot fit should be refused before it is sent, with
its numbers, and not discovered as a provider error after the usage call
has begun. On the regate path that error was caught by
`_regenerate_report`, so the run completed with the unregated report and
no recorded reason.
On shared generation 1, `_call_with_usage` now counts the exact request
`_call_with_retry` will send for reconcile_full, report, report_regate and
report_after_repair. It uses the receipt completion's helpers: the
single-attempt client, `messages.count_tokens` and `_context_check`,
against the registration's `audit_limits` window, as the typed stages do.
When count + max_tokens exceeds context_limit_tokens it raises
UsageLedgerError before the usage call begins. Nothing is sent, and there
is no pending call, usage row, abandoned attempt or consumed regate
allowance. `_generate_phase` and `_regenerate_report` both re-raise it, so
MAX_ATTEMPTS is not spent, and a resume recounts the same request and
refuses it again. The batch canary records it as it records the other
admission refusals ("it raised UsageLedgerError"). The dollar readout is
unchanged and prices nothing for a refused call. An admitted count is
pinned as `{project}_{phase}_context.json` in the snapshot index and the
provenance record's intermediates. The receipt completion module gains
`_measure_context`, the count before its verdict, and a `label` on its
messages so the refusal names the phase and its numbers; its own messages
are unchanged. Shared generation 2 already counts every ordinary call and
is untouched.
reconcile_full is included although #5157 names only the report family.
D28 raised it to 128k with the report (`SHARED_PHASE_MAX_TOKENS`), and it
carries the same records, audit findings and schema context, so the same
872k room applies. An over-window reconcile_full already ended the run
with a provider error; the check changes only that it stops before
sending, with the count, at the cost of one unbilled count call.
A refused regate or report refresh after repair now stops the run, where
before the request was sent and any error it drew left the stale report
in place. Re-sizing the AI_READI and KIDS_FIRST inputs, #5157's other
proposed fix, is still open.
No model-facing bytes change. The CHORUS fixture's reconcile_full, report
and regate requests hash the same at the base f8c5ac7 and here, and the
same with the admission bypassed (tests/test_report_admission.py).
Refs #5157, #5112
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…specific schema trim (#5148)
…ed ledger rows (D41)
…ed-v2 closure (D41)
…eview) Why: review R9 of 225a3cc (verdict "fix first") found four problems. A refused regate or refresh after repair raised out of the run with nothing recorded. The regate re-refused on every resume, so the run could never publish, and the refresh left a resume that died on the receipt-completion boundary. The CHORUS parity test hashed the author's checkout path, so it failed in every other checkout and in CI. A single transient error on the unbilled count stopped a run whose dispatch would have retried it. A refused request now raises ContextNotAdmitted, a UsageLedgerError that carries what the refusal rests on: the phase, the reason, the count, max_tokens, the window and its basis, and the request hash. The refused count is pinned as {project}_{phase}_context_refused.json. reconcile_full and the first report still stop the run before dispatch, because there is no artifact to keep. A refused regate is now treated as a regate that failed (#967). The report as written stays, and report_gate records regenerated false, regeneration_attempted false (the allowance is unspent), the reason and a regate_refused block. The run then writes its record before the post-record checks decide (#1818). A refused refresh after repair is now latched as a terminal evidence refusal, report_refresh_not_admitted with its numbers. A resume re-raises it before any drift logic (#1820). The report as written describes the records before repair, and the gate's source review refuses it in any case, which is how the parent ended the same request when the provider refused it. report_regenerated_after_repair also states whether the refresh was written. The count now uses the same transport as the call it admits: the caller's client, with the transient errors _call_with_retry retries retried up to MAX_ATTEMPTS, with the same backoff and rate-limit pause. Both now use one _transient classification. Non-transient errors and unusable counts still refuse at once. The parity test now normalises the checkout the schema resolved from as well as the run root. The constants were re-derived from f8c5ac7 in a separate tree and match the reviewer's. Typed stage counts now carry their stage's label, and an over-window count states its numbers. New tests cover the regate, refresh, count-retry and resume paths. Not done. Progress is not pinned between repair and refresh: the refresh trigger and the repair log live in that invocation's memory, so a pin alone would let a resumed run skip the refresh and drop the log. A refused regate is not kept in the ledger; a resumed gate recounts the same request, unbilled, and reaches the same recorded outcome. Refs #5157, #967 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… refuse D4D_RECEIPT_FULL_MAX_TOKENS (A14 R2 F1)
…ntity # Conflicts: # src/data_sheets_schema/api_runner.py
Fix F1 guard-owned fields and routing snapshots, F2 terminal startup diagnostics, F3 merge-base parity, F4 acknowledged interrupted updates, and F6 validator refresh latches. Implement D56 ineffective-override refusal and D57 externally anchored resume heads. Address F9 structural diagnostics and F11 repeated report rejection. Document F7 BC1 merge-order re-pinning without editing shared_generation.py; retain F10 and B history. Preserve serial offline regression, tamper and byte-parity evidence, including failed-before results and corrected fixture reruns. Claude re-review remains required under D55. Refs #5138, #5142, #5212.
realmarcin
added a commit
that referenced
this pull request
Oct 28, 2026
…uard rulings) After claude's independent review of both D41 guard candidates (d4d-executions/plans/reviews/d42_claude_independent_review_2026-10-27.md), the owner decided at 2026-10-28T07:02Z: - D55, the D42 pick: "A, fixed first (Recommended)". codex's design (#5212) goes forward once the review's F1-F4, F6 and F7 are fixed and re-reviewed. agy's B does not. - D56, override semantics: "No: terminal ends the run (Recommended)". An override never continues a run past a terminal or control-state discrepancy, and an override that can authorize nothing is refused before it is recorded. This amends the H3 spec. - D57, journal anchoring: "Record head at launch (Recommended)". The runner prints the journal head at each start and stop, claude's monitor record keeps it, and a resume must present the expected head. Adds section 4 rows and section 5.12, and updates the header, status, section 5 title and overview. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
realmarcin
added a commit
that referenced
this pull request
Oct 28, 2026
…uard rulings) After claude's independent review of both D41 guard candidates (d4d-executions/plans/reviews/d42_claude_independent_review_2026-10-27.md), the owner decided at 2026-10-28T07:02Z: - D55, the D42 pick: "A, fixed first (Recommended)". codex's design (#5212) goes forward once the review's F1-F4, F6 and F7 are fixed and re-reviewed. agy's B does not. - D56, override semantics: "No: terminal ends the run (Recommended)". An override never continues a run past a terminal or control-state discrepancy, and an override that can authorize nothing is refused before it is recorded. This amends the H3 spec. - D57, journal anchoring: "Record head at launch (Recommended)". The runner prints the journal head at each start and stop, claude's monitor record keeps it, and a resume must present the expected head. Adds section 4 rows and section 5.12, and updates the header, status, section 5 title and overview. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 13c81fd)
R6: compose local origin/claude/5157-report-admission at aaa6f5b. Preserve its report refresh implementation and tests; adapt resume tests to externally monitored D57 heads in the follow-up fix.
R2: refused starts emit no normal start/stop anchor; a forged chain cannot replace the external monitor head. R1: document and test SIGTERM reconciliation as the supported recovery path, without changing process-global signal policy. R5: exercise A13 registration hashes and D57 journal heads together, and move the scope-context monitor away from A14 additions. R6: opt #5157 report tests into the external append monitor. R7: compare identical current-tree builders and resources with/without the transport callback, independent of Git events and deleted refs. Document R3 assisted interrupted recovery, R4 guarded label reuse, R8 per-file flat anchors and R9 old-generation refusal. Preserve serial test launcher and SIGTERM fixture provenance; 127 core tests pass.
R6/R7: the merged report suite passed 61 cases, including all seven formerly unmonitored resumes, but exposed historical CHORUS wire hashes changed by A13. Replace that pin with the existing same-tree admission-on/off comparison and retain the exact phase roster and counted-versus-dispatched request checks. D41 parity now runs the actual journal check on candidate transports, omits it on the baseline, and verifies that a real terminal latch prevents dispatch. Preserve serial selection, capture archiving and artifact verification helpers. No production or protected source changed.
Make the omission request receiver an explicit frozen Prepared value without changing payload JSON or relaxing the specificity scanner. Complete selected-tree fixtures with REQUEST_CODE and retain refusal of missing selected dependencies. Assert the exact shared-v1 source-hash delta for the three historical-validation dependencies. Validation: 422 required-module tests, 309 A14/S7 tests, and 123 neutrality tests pass; the final nine targeted cases also pass. Tests ran serially, offline, with external pytest basetemp. Record before/after CHORUS request builds and retain the provenance helper. Five of eight complete bodies and the inner omission request are literally unchanged. The three typed bodies differ only in registration_sha256; all eight match with that fingerprint held fixed. Strict fresh-registration byte parity remains an unresolved requirement, explicitly documented for review. No scanner exemption or source-identity bypass is introduced.
R1-R9: preserve the merge resolution, supported SIGTERM fallback, refused-anchor security regression, #5157 monitoring and same-tree parity outcomes. Record 127 core, 29 F2, 4 routing, 6 parity, 373 provenance and 396 receipt passes; report coverage is 61 plus one corrected rerun, and launcher/CLI/lock coverage is 144 plus two process-inspection reruns. Preserve failed iterations, exact commands, XML, source pins and wire captures in byte-verified archives. Verify every retained batch matches final production Python sources. Record the clean A14 merge-tree result and retain CI, independent review and the existing BC1 composition re-pin as explicit handoff boundaries.
R2: shorten every full hexadecimal hash in refused-start diagnostics and raised exceptions to an explicitly unverified eight-character prefix. Suppress the original exception context so CLI and log tracebacks cannot disclose the forged resume anchor. Capture stdout, stderr, exception text, tracebacks and logs; replay every emitted 64-hex token. Reproduce the reviewer d-prime forged chain with zero calls on refusal and a three-dispatch positive control using the unprinted forged head. Offline validation: 538 latch-journal/D41/F2 and regression tests, all four routing API cases, and six parity tests passed. Preserve the launcher and structured validation evidence. No push or provider calls.
PR #5210 / #5157, finding M1: use the same-tree admission-on/off wire test from #5212 head 6041adf. Omit only the D57 monitor opt-in because this packet has no latch-journal fixture; #5212 retains that opt-in when refreshed. The exact-wire canary passes without changing protected production code or prompt pins.
Refs #5212, #5162. Merge the exact Wave6 #5212 refresh and reviewed A14 head without rebasing either parent. Resolve only tests/test_source_heading_runtime_api.py by retaining the exact A14 descriptor delta assertion and the D57 journal_launch_monitor marker. Protected source is Git auto-merged only; no manual protected-file or decision edits.
realmarcin
added a commit
that referenced
this pull request
Oct 29, 2026
Addresses these P3 findings on 314189e: - Claude F8 / Codex F4: the decision table no longer presents the D42 guard pick as open. D41-D43 and D48 cite D55, and the D55 row states #5212's actual state (draft at db88ca6; fix packet 4f20a30 not merge-ready after its re-review, R5). - Claude F15: the Inputs cite 13c81fd (D55-D57), 4c6c5ac (D58-D60) and 8c91e37 (D61, on #5164's branch), add D62 and D63 rows, and say D62/D63 are so far recorded in STATUS.md only. - Claude F7 (part): D62 and D63 enter the decision table. - Claude F9: a step-ID glossary separates plan IDs from board IDs; unprefixed IDs are plan IDs except A21 (the re-canary) and the A14 digest. The re-canary fills plan A10's role, A10c its contingency. The D28, D35, D39, D44/D45 rows and section 11 now say "board A17". - Claude F21: A1 is superseded as a sizing input; D63 authorizes part of A1b; the plan-amendment note follows D50 (claude prepares, agy pushes). - Claude F22 (item 1): "named under D23" becomes D36. - Claude F23: the header proposes what performs P4 (an owner word naming this file's sha256, recorded in the decision record), as P4 item 16, and flags the generation plan's "merge" wording. Inputs also name the BC1 fit analysis, the D59 review and both P3 passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 29, 2026
…nting Use the public selected omission request for both identity and roster. Start the direct form fixture's fresh accounting before creating records, retaining the historical-output guard. Exact-parent controls distinguish the integration mismatch from the inherited fixture defect. Refs #5212, #5162.
Retain original 548-pass/four-failure suite, exact-parent controls, disproved intermediate test hypothesis, final 50 passing scoped checks, actual canary bytes and diagnostic limits. Preserve logs/XML verbatim including original whitespace and verify published packet candidates from local Git blobs. Refs #5212, #5162.
This was referenced Oct 29, 2026
D41 parity helper answers the unprojected omission request, which A14's strict binding refuses
#5384
Open
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements D41 (
H3) hash-chained latch journal, pre-parse raw response persistence, andshared-v2ROUTING_CODEclosure onclaude/d41-latch-guard(db88ca68c19e09142b7c218d0d0596f321e78440, reconciling8ab03fda923d2c8caa9a2e65e558be73e92500e9and9fa5d391b612cdf82f40798ecbbc1b9839b387fc, stacked onclaude/dc-a13-receipt-floor-v301264370e2ca51312cb534d42a4c7b9f2721b1c6):src/data_sheets_schema/latch_journal.py&src/data_sheets_schema/api_runner.py: Enforces a tamper-evident hash-chained latch journal (latch_journal.json), persists raw responses before gate/parse evaluation (_write_raw_response_before_parse), verifies phase artifacts on entry (_verify_phase_artifacts), and refuses runs whose terminal or rejection state moved backward without a qualified override.src/data_sheets_schema/shared_generation.py: Appliesnotes/d41_shared_generation_patch_proposal.diff(ROUTING_CODE,STAGE_ORDER,STAGE_BUILDERS, anddescriptor(version=2)closure).d41_latch_journal_8ab03fda9_review_2026-10-26.mdFindings F1–F6.Verification
pytest tests/test_d41_latch_guard.py tests/test_api_runner_ctl.py tests/test_build_api_registrations.py tests/test_shared_generation_scope_context.py: 210 passed.