From 5d648779ffd4df583a2419e3a1c43f90fbed6e43 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sun, 25 Oct 2026 15:40:49 +0100 Subject: [PATCH 1/6] docs(project): add production qualification implementation plan --- ...5-codeapi-production-qualification-plan.md | 63 +++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 docs/project/2026-10-25-codeapi-production-qualification-plan.md diff --git a/docs/project/2026-10-25-codeapi-production-qualification-plan.md b/docs/project/2026-10-25-codeapi-production-qualification-plan.md new file mode 100644 index 00000000..454f18fa --- /dev/null +++ b/docs/project/2026-10-25-codeapi-production-qualification-plan.md @@ -0,0 +1,63 @@ +# CodeAPI production load and sandbox recovery qualification + +## Approval summary + +Approval mode: executable batch. The user approved the next bounded production-load and Spot-recovery test package after the sustained-load tooling was merged. Execute a finite production repeat of the accepted paced workload using the existing deployment, capacity, signing profile and synthetic caller contract. Separately qualify sandbox-runner process replacement in staging. Preserve the distinction between process recovery and a real Spot node interruption. + +Production acceptance reuses the existing direct API/artifact contract and the three eight-minute sustained intervals. The existing shared envelope remains authoritative: at most 400 HTTP calls, 360 execution-route reservations, 1080000 requested guest milliseconds, four in-flight requests, and a 1800-second invocation bound. Two stable synthetic callers share one ledger. Stop admission on the first failure or stale guard, settle admitted work, and clean only owned synthetic artifacts and local processes. Preserve every failed invocation separately; there are no unchanged automatic retries. + +The authorized production effects are synthetic API calls, read-only observations using existing connectivity, owned loopback Service forwards, exact existing read-only signing scope, and deletion of newly owned test artifacts. No deployment, capacity, rate, secret-policy or cloud-resource change is proposed. Production disruption is excluded. A staging interruption requires a concrete target, disruption type and recovery boundary; prepare its exact approval package before a destructive action. The earlier general approval does not silently authorize an Azure node eviction or deletion. + +Done means production finite acceptance or a precise preserved blocker, reviewed staging recovery procedure, verified owned cleanup or explicitly preserved cleanup uncertainty blocking acceptance, and honest evidence scope. Source changes, review and ordinary draft delivery are authorized. A staging pod-replacement experiment can demonstrate controller/process recovery only. Genuine node loss, running-job interruption and cloud-capacity recovery remain separately named tests. + +## Execution details + +### Working context and ownership + +Target: fork main, starting from the merged qualification package. Branch: `rs/codeapi-production-qualification`; worktree: `trees/storage-reliability-investigation/trees/codeapi-production-qualification`. Artifact root: `docs/project/`. Main owns the plan, trusted-source review, secrets, live requests, interruption authority and final proof. Execution mode: standard. One bounded trusted source executor owns only the named private operator adaptation below; it has no secret, live-system or forge access. A read-only trusted planning pass challenges this finite procedure before commit; Claude reviews the production operator adaptation and the recovery semantics. Existing qualifying source reviews and deterministic checks are reused where their contract is unchanged. + +All actual environment bindings, workload observations, identities, measurements and receipts stay in ignored `docs/project/_local/reviews/2026-10-25-production-qualification/`. The tracked plan and draft description contain intended behavior and source verification only. No browser or UI changes are included. + +### Production workload and guard + +Reuse `scripts/qualification/codeapi.mjs` and its existing tests unchanged. Reuse the accepted staging observer and final-health predicates, adapting only the environment context, exact loopback ports, existing production PEM signing profile and applicable production canary inventory. Preserve original workload identity, readiness, restart, immutable-image, Argo source/producing revision, KEDA ownership, queue, retirement and timestamp checks. Keep the original five-second freshness bins and interval-boundary proof. A mismatch or unknown metric identity stops admission instead of relaxing a predicate. + +Before any synthetic request, require both existing contexts reachable, all intended production workloads ready, healthy schedulable serving nodes, no active maintenance, stable source/configuration bindings, empty fresh queues and sandbox activity, ordinary principal rate limits sufficient for the exact fixed offers, and a fresh successful applicable producing storage canary. Capture a complete new environment baseline; earlier staging or production receipts cannot substitute. Preserve the production runner's existing concurrency as measured from the allowlisted configuration, without changing it. + +Use only the existing `codeapi-prd` profile's exact approved signing secret, mapped into the child process and removed from its environment before parsing. Reuse the approved in-memory public-verifier match to select the signing key ID. Never print, hash, serialize or save private keys, verifier values or tokens. Use the existing issuer/audience and two stable synthetic principals with fresh short-lived tokens; no identity rotation or trust expansion. + +The complete original request ledger, workload schedule, observation acceptance and final-health rules in the merged sustained-load plan are inherited unchanged, except the explicit environment/profile/canary binding adaptation above. Client `passed` alone is insufficient: external observer coverage, all interval boundaries and final acceptance must pass independently. No repeated workload produces files; the direct contract owns the three temporary objects. Maintain global/per-caller in-flight limits, missed-slot stop behavior, independent interval guards, exact outputs and fair offered allocation. Record client timing, resource and queue observations without treating fixed offered load as saturation capacity or an SLA. A failed ordinary request, unexpected 429, output mismatch, cleanup uncertainty, stale guard, source movement, maintenance or original workload turnover makes the invocation incomplete and holds dependent tests. + +After client terminal settlement and its owned artifact cleanup, require fresh queue/sandbox drain in sixty seconds, final bindings and original health, fresh owner-bound canary production, and all owned processes/forwards stopped within the existing 120-second post-run bound. No further CodeAPI request is permitted in that read-only evidence/shutdown period. Canary evidence cannot extend the bound. Unknown or failed artifact cleanup blocks acceptance; preserve the uncertainty without broad discovery or unapproved repair. Applicable production canary evidence must be at most twenty minutes old. Do not create manual canary jobs or alter schedules to manufacture acceptance. + +### Recovery sequence and scope + +First prepare a read-only staging inventory: runner Deployment/ReplicaSet/pod ownership, existing HPA/KEDA bounds, node pool ownership and disruption budgets, original API/worker health, active queues and sandbox metrics. Bind one proposed target by UID and resourceVersion in private receipts. Use current official eviction semantics and a nonmutating server dry-run where permitted. Respect disruption budgets and normal termination; never force deletion or bypass admission. + +Before actual interruption, obtain one coherent approval naming the concrete pod eviction or separate cloud interruption, expected temporary service effect, exact UID/guards, admission count, recovery deadline, success evidence and recovery path. The approved package already allows all independent source preparation and production-load work. Do not repeat generic approval for those actions. + +A pod-replacement proposal must keep production untouched and cloud capacity unchanged. Run it only with empty queues and no active sandbox executions, so it measures idle replacement. Success requires replacement UID on the intended unchanged owner/template, ready serving endpoints, preserved desired state and a bounded exact-result direct smoke. Do not claim active-job survival, cancellation under node loss, capacity reacquisition or Spot eviction tolerance from this proof. A genuine Spot interruption needs a separate explicit cloud target, cost/outage envelope and safe recovery plan through the existing DF Cloud/Pulumi/GitOps route. + +### Delegation Map and bounded slices + +| Slice | Owner | Dependency | Acceptance | +| --- | --- | --- | --- | +| Persist finite plan and authority | Main with read-only planner | User direction and existing qualification contracts | Frozen draft passes planning challenge and is committed before implementation | +| Adapt private production operator | Bounded trusted source executor; main integrates | Named plan and unchanged client contract | Environment-only diff, existing observer/signer fixtures pass, integrated review before admission | +| Run one finite production qualification | Main exclusively | Source/operator checks, review and complete fresh preflight | Same workload/observer/final-health acceptance and verified owned cleanup, or preserved failure | +| Prepare staging recovery approval package | Main exclusively with Claude advice | Read-only ownership/disruption inventory and official semantics | Exact target/action/guards/limits/recovery prepared; interruption waits for specific authority | +| Review and deliver the package | Main with integrated reviewer | Prior slices terminal | Sanitized tracked plan, private receipts, ordinary push and draft; honest unmet gates | + +Only this plan is a new tracked path. Private operator paths are `run_prd.py`, `final-health.py`, `sandbox-metric-cohort.py`, `observer-contract.test.py`, `signing-wrapper.mjs`, `signing-smoke.test.mjs`, `analyze-results.py`, and `operator-adaptation.md` beneath the existing ignored package directory. Main additionally owns `preflight.py`, `preflight.test.py`, `recovery-approval-package.md`, and `continuation.md` there. Allowlisted baseline and receipt JSON files are generated evidence, not a new framework. No executable production source changes are planned. + +Feature-wide test portfolio: `none` for unchanged public client scheduling/budgets, with the merged twenty-case suite and producing hosted run reused. `extend existing` for the private observer/signing fixtures named above to protect context, destination, profile, original-pod stop and production-canary changes. `add new` only for the small main-owned preflight fixture if its allowlisted configuration/rate/node checks are not already protected. Live proof owns image/owner/source binding, interval coverage and final canary/cleanup acceptance; fixture success cannot substitute. + +### Verification and delivery + +All offline adaptation checks, the planning pass and integrated source/operator review must pass before the first production request. Reuse unchanged public-client verification. The private operator adaptation must have a reviewed mechanical diff against the accepted source and offline checks for exact context/port/profile/canary bindings, original-pod stop behavior, source/replica-owner invariants and failure cleanup. Preserve existing behavioral fixtures; add only checks that protect changed environment binding behavior. No new runtime dependency, maintained framework or deployed service change is required. + +Commit this generic plan after planning review and before implementation. Private operational files remain ignored. Run focused repository formatting and diff checks for tracked text; inspect staged material for secrets and real-environment data before every commit or draft update. Complete integrated review and ordinary branch push/draft delivery. Ready/merge, publication and environment rollout remain outside this package. Preserve all historical artifacts, existing user connectivity and unrelated work. + +## Progress + +Planning started after direct approval. Existing merged source qualification is reused. Current live preflight, trusted planning challenge, operator adaptation and recovery approval package are pending. No synthetic execution or disruption has begun in this package. From e73c7c212a5c528fde0f1873ee10ac678fc4a10e Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sun, 25 Oct 2026 15:47:53 +0100 Subject: [PATCH 2/6] docs(project): clarify shared production qualification limits --- .../2026-10-25-codeapi-production-qualification-plan.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/project/2026-10-25-codeapi-production-qualification-plan.md b/docs/project/2026-10-25-codeapi-production-qualification-plan.md index 454f18fa..08b6a97b 100644 --- a/docs/project/2026-10-25-codeapi-production-qualification-plan.md +++ b/docs/project/2026-10-25-codeapi-production-qualification-plan.md @@ -22,13 +22,13 @@ All actual environment bindings, workload observations, identities, measurements Reuse `scripts/qualification/codeapi.mjs` and its existing tests unchanged. Reuse the accepted staging observer and final-health predicates, adapting only the environment context, exact loopback ports, existing production PEM signing profile and applicable production canary inventory. Preserve original workload identity, readiness, restart, immutable-image, Argo source/producing revision, KEDA ownership, queue, retirement and timestamp checks. Keep the original five-second freshness bins and interval-boundary proof. A mismatch or unknown metric identity stops admission instead of relaxing a predicate. -Before any synthetic request, require both existing contexts reachable, all intended production workloads ready, healthy schedulable serving nodes, no active maintenance, stable source/configuration bindings, empty fresh queues and sandbox activity, ordinary principal rate limits sufficient for the exact fixed offers, and a fresh successful applicable producing storage canary. Capture a complete new environment baseline; earlier staging or production receipts cannot substitute. Preserve the production runner's existing concurrency as measured from the allowlisted configuration, without changing it. +Before any synthetic request, require both existing contexts reachable, all intended production workloads ready, healthy schedulable serving nodes, no active maintenance, stable source/configuration bindings, empty fresh queues and sandbox activity, ordinary principal rate limits sufficient for the exact fixed offers, the deployed default execution profile, applicable canary cadence compatible with the final age bound, and a fresh successful applicable producing storage canary. Capture a complete new environment baseline; earlier staging or production receipts cannot substitute. Preserve the production runner's existing concurrency as measured from the allowlisted configuration, without changing it. Use only the existing `codeapi-prd` profile's exact approved signing secret, mapped into the child process and removed from its environment before parsing. Reuse the approved in-memory public-verifier match to select the signing key ID. Never print, hash, serialize or save private keys, verifier values or tokens. Use the existing issuer/audience and two stable synthetic principals with fresh short-lived tokens; no identity rotation or trust expansion. -The complete original request ledger, workload schedule, observation acceptance and final-health rules in the merged sustained-load plan are inherited unchanged, except the explicit environment/profile/canary binding adaptation above. Client `passed` alone is insufficient: external observer coverage, all interval boundaries and final acceptance must pass independently. No repeated workload produces files; the direct contract owns the three temporary objects. Maintain global/per-caller in-flight limits, missed-slot stop behavior, independent interval guards, exact outputs and fair offered allocation. Record client timing, resource and queue observations without treating fixed offered load as saturation capacity or an SLA. A failed ordinary request, unexpected 429, output mismatch, cleanup uncertainty, stale guard, source movement, maintenance or original workload turnover makes the invocation incomplete and holds dependent tests. +The complete original request ledger, workload schedule, observation acceptance and final-health rules in the merged sustained-load plan are inherited unchanged, except the explicit environment/profile/canary binding adaptation above. Client `passed` alone is insufficient: external observer coverage, all interval boundaries and final acceptance must pass independently. No repeated workload produces files; the direct contract owns the three temporary objects. Maintain global/per-caller in-flight limits, missed-slot stop behavior, independent interval guards, exact outputs and fair offered allocation. Record client timing, resource and queue observations without treating fixed offered load as saturation capacity or an SLA. The loopback authenticated Service path bypasses ingress/edge policy and qualifies that service path only. A failed ordinary request, unexpected 429, output mismatch, cleanup uncertainty, stale guard, source movement, maintenance or original workload turnover makes the invocation incomplete and holds dependent tests. Shared-user contention can legitimately miss the fixed slot or guest-time contract; classify that as incomplete offered-load coverage, not a service outage. Empty-queue preflight that cannot be satisfied is an honest blocker. No unchanged repeat follows without a new user decision. Queue ceilings limit aggregate impact but do not isolate the synthetic workload from users. Equal allocation/completion does not prove scheduler fairness or a service objective. -After client terminal settlement and its owned artifact cleanup, require fresh queue/sandbox drain in sixty seconds, final bindings and original health, fresh owner-bound canary production, and all owned processes/forwards stopped within the existing 120-second post-run bound. No further CodeAPI request is permitted in that read-only evidence/shutdown period. Canary evidence cannot extend the bound. Unknown or failed artifact cleanup blocks acceptance; preserve the uncertainty without broad discovery or unapproved repair. Applicable production canary evidence must be at most twenty minutes old. Do not create manual canary jobs or alter schedules to manufacture acceptance. +After client terminal settlement and its owned artifact cleanup, require fresh queue/sandbox drain in sixty seconds, final bindings and original health, fresh owner-bound canary production, and all owned processes/forwards stopped within the existing 120-second post-run bound. No further CodeAPI request is permitted in that read-only evidence/shutdown period. Canary evidence cannot extend the bound. Unknown or failed artifact cleanup blocks acceptance; preserve the uncertainty and exact run/principal references privately without broad discovery or unapproved repair. Use one graceful stop signal and allow the client settlement/cleanup bounds before escalation; forced termination cannot be reported as complete cleanup. Applicable production canary evidence must be at most twenty minutes old. Do not create manual canary jobs or alter schedules to manufacture acceptance. ### Recovery sequence and scope From 1327ba17272b65ea051903e1f5e5b17382caea63 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sun, 25 Oct 2026 16:13:56 +0100 Subject: [PATCH 3/6] docs(project): record production qualification source review --- .../project/2026-10-25-codeapi-production-qualification-plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/project/2026-10-25-codeapi-production-qualification-plan.md b/docs/project/2026-10-25-codeapi-production-qualification-plan.md index 08b6a97b..7800d242 100644 --- a/docs/project/2026-10-25-codeapi-production-qualification-plan.md +++ b/docs/project/2026-10-25-codeapi-production-qualification-plan.md @@ -60,4 +60,4 @@ Commit this generic plan after planning review and before implementation. Privat ## Progress -Planning started after direct approval. Existing merged source qualification is reused. Current live preflight, trusted planning challenge, operator adaptation and recovery approval package are pending. No synthetic execution or disruption has begun in this package. +The generic procedure is delivered in [draft PR #50](https://github.com/uzh-bf/code-interpreter/pull/50). Native planning review and trusted integrated operator review are complete; the focused stop-path finding was corrected and its behavioral regressions pass. Public qualification client behavior remains unchanged. The latest operational checkpoint, actual acceptance, measurements, cleanup and disruption authority are maintained exclusively in the existing ignored package directory; consult its continuation before any later live action. Source delivery and CI do not establish environment acceptance. The staging interruption proposal retains its exact-action human gate. From a28ed0b6616e7a6e9ee9efcc1f7c8986033c40c6 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sun, 25 Oct 2026 16:48:58 +0100 Subject: [PATCH 4/6] fix(qualification): synchronize sustained timing fixtures --- ...5-codeapi-production-qualification-plan.md | 2 +- scripts/qualification/codeapi.test.mjs | 82 +++++++++++++++---- 2 files changed, 69 insertions(+), 15 deletions(-) diff --git a/docs/project/2026-10-25-codeapi-production-qualification-plan.md b/docs/project/2026-10-25-codeapi-production-qualification-plan.md index 7800d242..0f0224e1 100644 --- a/docs/project/2026-10-25-codeapi-production-qualification-plan.md +++ b/docs/project/2026-10-25-codeapi-production-qualification-plan.md @@ -48,7 +48,7 @@ A pod-replacement proposal must keep production untouched and cloud capacity unc | Prepare staging recovery approval package | Main exclusively with Claude advice | Read-only ownership/disruption inventory and official semantics | Exact target/action/guards/limits/recovery prepared; interruption waits for specific authority | | Review and deliver the package | Main with integrated reviewer | Prior slices terminal | Sanitized tracked plan, private receipts, ordinary push and draft; honest unmet gates | -Only this plan is a new tracked path. Private operator paths are `run_prd.py`, `final-health.py`, `sandbox-metric-cohort.py`, `observer-contract.test.py`, `signing-wrapper.mjs`, `signing-smoke.test.mjs`, `analyze-results.py`, and `operator-adaptation.md` beneath the existing ignored package directory. Main additionally owns `preflight.py`, `preflight.test.py`, `recovery-approval-package.md`, and `continuation.md` there. Allowlisted baseline and receipt JSON files are generated evidence, not a new framework. No executable production source changes are planned. +Only this plan is a new tracked path. A reproduced qualification-fixture timing defect discovered in the required source checks adds the existing `scripts/qualification/codeapi.test.mjs` to this package: make synthetic transport/virtual-time settlement deterministic without changing the deployed client or its behavioral limits. Main integrates a bounded trusted executor correction and reruns the affected checks before draft completion. Private operator paths are `run_prd.py`, `final-health.py`, `sandbox-metric-cohort.py`, `observer-contract.test.py`, `signing-wrapper.mjs`, `signing-smoke.test.mjs`, `analyze-results.py`, and `operator-adaptation.md` beneath the existing ignored package directory. Main additionally owns `preflight.py`, `preflight.test.py`, `recovery-approval-package.md`, and `continuation.md` there. Allowlisted baseline and receipt JSON files are generated evidence, not a new framework. No executable production source changes are planned. A private observer correction may accept only a bounded, freshly proven owned HPA transition when Argo aggregate health lags complete healthy runner status; require direct unchanged non-runner Deployment proof and preserve all unrelated-health failures. Review changed predicates before any corrected live attempt; preserve each actual invocation separately. Feature-wide test portfolio: `none` for unchanged public client scheduling/budgets, with the merged twenty-case suite and producing hosted run reused. `extend existing` for the private observer/signing fixtures named above to protect context, destination, profile, original-pod stop and production-canary changes. `add new` only for the small main-owned preflight fixture if its allowlisted configuration/rate/node checks are not already protected. Live proof owns image/owner/source binding, interval coverage and final canary/cleanup acceptance; fixture success cannot substitute. diff --git a/scripts/qualification/codeapi.test.mjs b/scripts/qualification/codeapi.test.mjs index 6d58482f..2fd28b21 100644 --- a/scripts/qualification/codeapi.test.mjs +++ b/scripts/qualification/codeapi.test.mjs @@ -655,7 +655,8 @@ test('corrupt resource results stop admission and final-idle guard failure canno (options.corruptResource ? 'resource_output_mismatch' : 'guard_rejected') - ) + ), + JSON.stringify({ options, failures: result.failures }) ); assert.equal( events.some(x => x.type === 'resource_phase' && x.name === 'soak'), @@ -891,7 +892,10 @@ async function sustainedRun( let elapsed = 0, delayed = false, zeroPauses = 0, - overrunAdded = false; + overrunAdded = false, + heldRequestCount = 0; + const ordinaryRequests = new Set(), + heldRequests = []; const writeGuard = async observedAt => { const temporary = `${c.guardPath}.${randomUUID()}`; await writeFile( @@ -912,6 +916,10 @@ async function sustainedRun( pause: async ms => { zeroPauses += ms === 0 ? 1 : 0; await delay(pauseMs); + if (ordinaryRequests.size) { + await Promise.allSettled(ordinaryRequests); + await new Promise(resolve => setImmediate(resolve)); + } if (lateSlot && !delayed && ms === 0 && zeroPauses === 2) { elapsed += 1001; delayed = true; @@ -920,6 +928,10 @@ async function sustainedRun( const missing = missingBoundary && elapsed >= 480000 && elapsed <= 490000; await writeGuard(epoch + (missing ? 479999 : elapsed)); + if (heldRequests.length && heldRequestCount >= holdSlots) + setImmediate(() => { + for (const release of heldRequests.splice(0)) release(); + }); }, request: async (...args) => { let input; @@ -929,6 +941,13 @@ async function sustainedRun( const label = input?.code && /nonce="[^"]*:(\d+):(\d+):([ab]):\d+"/.exec(input.code); + const perform = async () => { + await writeGuard(epoch + elapsed); + const response = await exchange(...args); + if (input?.code?.includes('qualification-resource')) + elapsed += 1; + return response; + }; if ( holdFirst && label && @@ -936,20 +955,22 @@ async function sustainedRun( (!holdCaller || label[3] === holdCaller) && Number(label[2]) < holdSlots ) { - return new Promise(resolve => { - setTimeout(async () => { - resolve(await exchange(...args)); - }, 500); + heldRequestCount++; + return new Promise((resolve, reject) => { + heldRequests.push(() => perform().then(resolve, reject)); }); } if (overrun && !overrunAdded && input?.code?.includes(':1:47:')) { elapsed += 11001; overrunAdded = true; } - await writeGuard(epoch + elapsed); - const response = await exchange(...args); - if (input?.code?.includes('qualification-resource')) elapsed += 1; - return response; + const pending = perform(); + ordinaryRequests.add(pending); + try { + return await pending; + } finally { + ordinaryRequests.delete(pending); + } }, }); const events = (await readFile(c.receiptPath, 'utf8')) @@ -962,7 +983,14 @@ async function sustainedRun( (await readFile(c.receiptPath, 'utf8')).includes('synthetic-secret'), false ); - return { result, events, server, elapsed, runId: c.runId }; + return { + result, + events, + server, + elapsed, + runId: c.runId, + heldRequestCount, + }; } test('sustained schedule binds nonces, kinds, callers and the full ledger', async t => { @@ -1128,12 +1156,38 @@ test('late, blocked, missing-boundary and overrun intervals stop without catch-u [{ missingBoundary: true }, 'sustained_boundary_missing', 1], [{ overrun: true }, 'sustained_response_overrun', 1], ]) { - const { result, events } = await sustainedRun(t, options); - assert.equal(result.status, 'failed'); - assert.ok(result.failures.some(x => x.reason === reason)); + const { result, events, heldRequestCount } = await sustainedRun( + t, + options + ); + assert.equal( + result.status, + 'failed', + JSON.stringify({ + options, + failures: result.failures, + heldRequestCount, + }) + ); + assert.ok( + result.failures.some(x => x.reason === reason), + JSON.stringify({ + options, + expectedReason: reason, + failures: result.failures, + }) + ); assert.equal( events.filter(x => x.type === 'sustained_phase').length, phases ); } }); + +test('ordinary transport latency settles before the virtual sustained schedule advances', async t => { + const { result, events } = await sustainedRun(t, { + serverOptions: { sustainedDelay: { ms: 40 } }, + }); + assert.equal(result.status, 'passed'); + assert.equal(events.filter(x => x.type === 'sustained_phase').length, 3); +}); From 3ff85d514f800082f5a62d9b14d2db850569861d Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sun, 25 Oct 2026 16:57:11 +0100 Subject: [PATCH 5/6] fix(qualification): publish fixture guards atomically --- scripts/qualification/codeapi.test.mjs | 46 ++++++++++++-------------- 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/scripts/qualification/codeapi.test.mjs b/scripts/qualification/codeapi.test.mjs index 2fd28b21..42b5092f 100644 --- a/scripts/qualification/codeapi.test.mjs +++ b/scripts/qualification/codeapi.test.mjs @@ -535,6 +535,12 @@ test('resource workload oracle has independent known answers and bounded allocat assert.throws(() => resourceWorkload('cpu', 'invalid\nnonce')); }); +async function publishFixtureGuard(path, value) { + const temporary = `${path}.${randomUUID()}`; + await writeFile(temporary, JSON.stringify(value)); + await rename(temporary, path); +} + async function resourceRun( t, { @@ -564,26 +570,20 @@ async function resourceRun( elapsed += 11000; delayed = true; } - await writeFile( - c.guardPath, - JSON.stringify({ - runId: c.runId, - observedAt: epoch + elapsed, - ok: !(finalFailure && elapsed >= 672000), - }) - ); + await publishFixtureGuard(c.guardPath, { + runId: c.runId, + observedAt: epoch + elapsed, + ok: !(finalFailure && elapsed >= 672000), + }); }, request: async (...args) => { if (lateAnchor && args[1] === '/health') elapsed += 35000; // This fixture observes local fake time only, while production uses the external observer. - await writeFile( - c.guardPath, - JSON.stringify({ - runId: c.runId, - observedAt: epoch + elapsed, - ok: true, - }) - ); + await publishFixtureGuard(c.guardPath, { + runId: c.runId, + observedAt: epoch + elapsed, + ok: true, + }); return exchange(...args); }, }); @@ -896,14 +896,12 @@ async function sustainedRun( heldRequestCount = 0; const ordinaryRequests = new Set(), heldRequests = []; - const writeGuard = async observedAt => { - const temporary = `${c.guardPath}.${randomUUID()}`; - await writeFile( - temporary, - JSON.stringify({ runId: c.runId, observedAt, ok: true }) - ); - await rename(temporary, c.guardPath); - }; + const writeGuard = observedAt => + publishFixtureGuard(c.guardPath, { + runId: c.runId, + observedAt, + ok: true, + }); const result = await runQualification({ ...c, baseUrl: server.baseUrl, From 0ab1323a89b2893a0a58afeacd1d792ccaf8ba14 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sun, 25 Oct 2026 17:02:23 +0100 Subject: [PATCH 6/6] fix(qualification): fence pending fixture dispatches --- scripts/qualification/codeapi.test.mjs | 42 +++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 4 deletions(-) diff --git a/scripts/qualification/codeapi.test.mjs b/scripts/qualification/codeapi.test.mjs index 42b5092f..5f1cfe82 100644 --- a/scripts/qualification/codeapi.test.mjs +++ b/scripts/qualification/codeapi.test.mjs @@ -884,6 +884,7 @@ async function sustainedRun( overrun = false, earlyWake = false, pauseMs = 20, + credentialDelayMs = 0, } = {} ) { const c = await files(t), @@ -893,7 +894,11 @@ async function sustainedRun( delayed = false, zeroPauses = 0, overrunAdded = false, - heldRequestCount = 0; + heldRequestCount = 0, + registeredRequests = 0, + expectedRequests = 0, + currentPlateau = null, + nextSlot = 0; const ordinaryRequests = new Set(), heldRequests = []; const writeGuard = observedAt => @@ -905,8 +910,14 @@ async function sustainedRun( const result = await runQualification({ ...c, baseUrl: server.baseUrl, - token: () => 'synthetic-secret', - comparisonToken: () => 'other', + token: async () => { + if (credentialDelayMs) await delay(credentialDelayMs); + return 'synthetic-secret'; + }, + comparisonToken: async () => { + if (credentialDelayMs) await delay(credentialDelayMs); + return 'other'; + }, mode: 'sustained', waitMs: 0, now: () => epoch + elapsed, @@ -914,6 +925,16 @@ async function sustainedRun( pause: async ms => { zeroPauses += ms === 0 ? 1 : 0; await delay(pauseMs); + const dispatchEnd = Date.now() + 5000; + while (registeredRequests < expectedRequests) { + const events = (await readFile(c.receiptPath, 'utf8')) + .trim() + .split('\n') + .map(JSON.parse); + if (events.some(event => event.type === 'failure')) break; + assert.ok(Date.now() < dispatchEnd, 'fixture dispatch barrier'); + await delay(1); + } if (ordinaryRequests.size) { await Promise.allSettled(ordinaryRequests); await new Promise(resolve => setImmediate(resolve)); @@ -926,6 +947,17 @@ async function sustainedRun( const missing = missingBoundary && elapsed >= 480000 && elapsed <= 490000; await writeGuard(epoch + (missing ? 479999 : elapsed)); + const starts = (await readFile(c.receiptPath, 'utf8')) + .trim() + .split('\n') + .map(JSON.parse) + .filter(event => event.type === 'sustained_start'); + const stage = starts.at(-1); + if (stage && stage.plateau !== currentPlateau) { + currentPlateau = stage.plateau; + nextSlot = 0; + } + if (stage && nextSlot++ < 48) expectedRequests += stage.concurrency; if (heldRequests.length && heldRequestCount >= holdSlots) setImmediate(() => { for (const release of heldRequests.splice(0)) release(); @@ -939,6 +971,7 @@ async function sustainedRun( const label = input?.code && /nonce="[^"]*:(\d+):(\d+):([ab]):\d+"/.exec(input.code); + if (label) registeredRequests++; const perform = async () => { await writeGuard(epoch + elapsed); const response = await exchange(...args); @@ -1182,9 +1215,10 @@ test('late, blocked, missing-boundary and overrun intervals stop without catch-u } }); -test('ordinary transport latency settles before the virtual sustained schedule advances', async t => { +test('ordinary signing and transport latency settle before the virtual sustained schedule advances', async t => { const { result, events } = await sustainedRun(t, { serverOptions: { sustainedDelay: { ms: 40 } }, + credentialDelayMs: 40, }); assert.equal(result.status, 'passed'); assert.equal(events.filter(x => x.type === 'sustained_phase').length, 3);