From ef8b4b2df8e54c53e7832839a8cf1ba52a949092 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 15:40:11 +0200 Subject: [PATCH 1/9] docs(codeapi): plan independent client and bounded performance qualification --- ...26-10-24-codeapi-capacity-contract-plan.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 docs/project/2026-10-24-codeapi-capacity-contract-plan.md diff --git a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md new file mode 100644 index 00000000..30209600 --- /dev/null +++ b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md @@ -0,0 +1,79 @@ +# CodeAPI independent-client and bounded performance qualification + +## Approval summary + +Approval mode: executable batch. The user accepted the next qualification package and requested a deeper capability assessment, a persisted plan and goal-driven execution. This package makes direct CodeAPI use repeatable without LibreChat and measures the existing staging configuration before recommending capacity changes. + +Add a dependency-free operator client derived from the public OpenAPI contract. Verify runtime selection, input/output files, exact downloaded bytes, metadata, ownership denial, profile mismatch, guest timeout and recovery. Reuse that client for a finite sequential Python baseline, Python burst and mixed-language burst. Distinguish normal rate-limit rejection from execution failure. Capture end-to-end latency, achieved throughput, aggregate processing/queue metrics and resource/scaling observations privately. Source-supported optional features receive a capability assessment with clear activation prerequisites and evidence limits. + +Authority covers synthetic STG API requests within the bounds below, existing-profile in-memory signing, existing-context read-only observation, task-owned Service port-forwards, owned test-file cleanup, source checks, reviews, commits, ordinary task-branch push and draft PR delivery. Existing production deployment authority is not extended. Capacity/concurrency changes, new trust identities, ingress exposure, diagnostics, fault injection, release publication and production traffic remain separate decisions. Upstream patch applicability is investigated alongside this package; importing or releasing a new runtime is not required to measure the current deployment. + +Done means the client and its failure controls pass offline tests, one approved bounded STG experiment produces honest acceptance or a precise blocker, owned artifacts/forwards are accounted for, and the reviewed source package is delivered as a draft. A successful Service-path test does not qualify ingress, a second application's issuer or an availability SLA. Missing telemetry reduces the claims rather than being filled by estimates. + +## Execution details + +### Working context and ownership + +Target: fork `main`. Baseline: `4105b53ecfeef1f9eea7a246a36601e181de11cf`. Branch: `rs/codeapi-capacity-contract`; worktree: `trees/storage-reliability-investigation/trees/codeapi-capacity-contract`. The existing repository artifact root remains `docs/project/`. History: [reliability roadmap](2026-10-23-codeapi-reliability-roadmap.md). All real-environment observations, identifiers, measurements and detailed review receipts stay in ignored `docs/project/_local/reviews/2026-10-24-capacity-contract/`. Tracked artifacts contain source-derived facts, generic procedures and progress only. + +Execution mode: standard. Main owns authority, authentication, live execution, integration and final proof. The read-only capability explorer owns optional-feature mapping. A planner challenges this frozen plan. One bounded implementation worker owns only the client and its offline tests; main owns integration, CI and documentation. Native trusted review covers private operational contracts; externally routed roles receive source-only scope. No separate chat is created. + +No product primitive or production contract changes. ADR gate: not met; this package adds qualification tools and documentation. New deployment ownership, trust or public capability activation would reopen design and authority review. No UI change; screenshots do not apply. + +### Evidence and research + +The public source supports a default HTTP execution profile, an issuer-bound JWT trust model, an independent file lifecycle, several runtimes and optional workspace/bridge/extraction backends. Current use and historical acceptance are evaluated privately. Inspect source gates before describing optional features as usable. `@librechat/code` is a remote-worker protocol/CLI; it is not assumed to be a general REST SDK. + +Main compares stable upstream tags and release metadata, recording a source-only applicability assessment against `v1.10.13`. Use official Node HTTP documentation for streaming, deadlines and abort behavior. Existing probes inform compatibility only; the new client imports no LibreChat code and does not monkey-patch global fetch. Preserve ordinary source CI without cluster or secret access. + +### Finite STG experiment and stop conditions + +Use one unique synthetic principal for the entire invocation; do not rotate identities to avoid limits. Fresh short-lived JWTs are signed in memory with only the existing approved profile mapping. A comparison principal may issue one owned-object denial check and performs no execution. Accept a caller-provided bearer token for reuse by a separately onboarded client; never persist or print either credential. + +Hard bounds: 900 seconds including cleanup, at most 160 HTTP requests, at most 40 executions, at most four in flight, at most 16 owned files, at most 1 MiB per request/response and at most 120 guest-seconds requested. Default profile only. Synthetic files are small; normal executions cap guest runtime at three seconds. Timeout testing uses a shorter guest cap and one bounded sleep. No retries or automatic replay. Stop ordinary admission at 720 seconds. Ordinary requests have a sixty-second whole-response deadline; already admitted requests settle within that deadline before cleanup. Reserve forty HTTP requests and the final 120 seconds for cleanup, which uses four-request concurrency and fifteen-second whole-response deadlines. Each owned object receives one DELETE and one absence check. The fixed workload can create at most four files; sixteen is the hard safety ceiling. No new ordinary work after SIGINT/SIGTERM or guard failure. Unresolved execution completion, missing returned references, failed deletion, expired cleanup credentials or any unknown object ownership makes the run failed with cleanup uncertainty; never claim cleanup or replay an execution. + +Contract qualification precedes load. Exercise Python, Node, Bun JavaScript, Bun TypeScript and bash with deterministic output, then verify Python input/output reuse, list/metadata, deletion absence, invalid language, invalid authentication, profile mismatch, one guest timeout and subsequent recovery. Expected negative cases must return their documented status/shape; HTTP 200 alone is insufficient. Track every returned owned output before validating it so failure still reaches cleanup. A response that cannot establish ownership leaves an explicit cleanup concern and blocks further workload admission. + +Load phases are eight short Python jobs sequentially, eight Python jobs in a bounded four-request burst, then eight alternating Python/Node jobs at four-request concurrency. Each load job sleeps two seconds and returns an exact marker plus guest-reported elapsed time; this measures dispatch/backlog and mixed-queue behavior, not CPU saturation. Enforce ordinary principal limits with at least one full observed execution-rate window between phases; never raise rate limits. Report offered/admitted/successful counts, phase duration, end-to-end percentiles and guest timing separately. Aggregate processing duration is not an individual queue-wait measurement. A short finite experiment is not a soak or maximum-capacity result. + +Stop admission on the first unexpected response, incorrect output, cleanup uncertainty, transport failure, new workload restart/unready state, memory pressure, sustained queue growth beyond the finite offered jobs, or an ordinary 429. Classify and preserve the 429 and Retry-After; do not retry it. Main runs a bounded observer every five seconds. Its mandatory private guard is bound to this invocation UUID and expires after twenty seconds. Admission checks it before every request. It must prove original workload readiness and unchanged restart counts, no node MemoryPressure, and observable waiting/active queues. More than eight waiting jobs for twenty seconds, any waiting count above twenty-four, or failed/stale observation stops admission. Required safety observations missing at preflight block live work. Optional CPU/memory and processing-time metrics may be absent only with that measurement limitation recorded. Do not infer stability from missing metrics. A failure holds subsequent phases, preserves receipts and runs only owned cleanup. No node drain, cache purge, worker restart or storage/Redis fault. + +The fixed ledger is: five runtime successes, two Python file executions, one guest-timeout case, one recovery, one invalid-language request and two pre-admission negative requests: at most twelve qualification execution-route calls. Add twenty-four load executions: at most thirty-six total, at most 107 requested guest-seconds, and fewer than eighty HTTP requests including negative cases and cleanup. Profile mismatch and authentication denial request no guest runtime. Negative execution requests still count toward the ordinary execution limiter. Read the effective rate window/limit privately before admission; wait a complete window after qualification and between phases. Abort before any workload if required waits cannot fit the finite envelope. No executable phase may exceed the lesser of its ledger and the hard budget. + +### Client and transport contract + +New paths: `scripts/qualification/codeapi.mjs`, `scripts/qualification/codeapi.test.mjs`, and `docs/project/2026-10-24-codeapi-capabilities.md`. Extend the existing deployment-config CI job with the offline tests. One implementation module contains the REST transport, synthetic cases, bounded scheduler and command entrypoint; avoid a new SDK package or framework. + +Require an exact base URL and explicit live mode; no default remote endpoint. Permit HTTPS or HTTP on loopback only, reject URL credentials/query/fragment, redirects and cross-origin requests. Require an explicit private receipt path for live use. The transport uses Node HTTP/HTTPS with whole-response byte bounds, finite deadlines, abort handling and no retries. Record safe machine outcomes only; response bodies, JWTs, private keys and server error messages never enter receipts. The signing JWK is deleted from the environment after parsing. Generated IDs and detailed timing remain in the caller's private receipt. + +Use existing cluster connectivity only. Service forwarding is an operator transport; record it distinctly from ingress. An absent consumer-facing route is an explicit unqualified transport, not permission to expose a new endpoint. Stop only task-owned forwards. Delete only synthetic object references returned during this invocation and verify absence, including after a test fails. Preserve unrelated resources and historical evidence. + +### Test portfolio + +| Consequential risk | Obligation and seam | Acceptance | +| -------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| Credential or response leakage / wrong destination | Add transport tests with an ephemeral local HTTP fixture | Redirect, oversized/truncated response and deadline fail closed; no auth in safe receipts; rejected destination receives no request | +| Incorrect outputs or false positive HTTP success | Add synthetic protocol-fixture tests for full client flow | Exact bytes and machine outcomes checked; malformed success fails; all returned owned refs still cleaned | +| Runaway work or incomplete owned cleanup | Add new scheduler/guard/failure tests in the same offline suite | Budget, concurrency, admission-stop and cleanup reserve enforced; failure blocks later phases; cleanup failure remains terminal | +| Runtime and independent-client compatibility | Live contract suite | All required cases pass or precise unsupported behavior is reported without changing the server | +| Python dispatch versus actual capacity | Bounded live phases plus read-only telemetry | Useful throughput/latency and actual useful work measured; pod count alone never passes acceptance | + +### Delegation map + +| Slice | Route and exclusive owner | Dependency and acceptance | +| --- | --- | --- | +| 1: plan and capability assessment | Main plus read-only source explorer; explorer returns mapping, main writes documents | Frozen source and planner approval; exact source references | +| 2: client and offline tests | Bounded executor owns the two named script paths; main owns CI and integration | Approved plan; offline protocol and failure controls pass | +| 3: STG experiment | Main; secrets and live effects retained for authority/privacy | Qualified tool commit and safety guard; one finite result with owned cleanup | +| 4: review and draft delivery | Main plus read-only final reviewer | Exact committed range, checks and private acceptance; reviewed pushed draft | + +### Slices and delivery + +1. Persist this reviewed plan and source-backed capability assessment. Main writes the plan; capability explorer maps optional features. Verify every material capability against immutable source and official documentation. Commit the plan before implementation. +2. Implement the independent client and offline consequential tests. Route the settled implementation to one worker; main integrates the authentication/ownership seams. Run checks in a disposable existing-toolchain container; host Git remains outside it. Add `node --test scripts/qualification/codeapi.test.mjs` to `.github/workflows/ci.yml` in the existing deployment-config job. Commit, simplify and review applicable risk once. +3. Main runs the live STG preflight, contract and bounded load phases once. Bind evidence privately to actual source/images/configuration and test-tool commit. Collect telemetry through existing read-only paths. Clean only owned files and local forwards, record missing transport/metrics and hold failed dependent phases. No configuration tuning is automatic; prepare an evidence-backed recommendation for a later named window. +4. Complete integrated review over the exact committed package, verify corrections, inspect staged content for real-environment data and secrets, push the task branch and create/update one draft PR. No merge, publication, production rollout or new automation follows from draft delivery. + +### Progress + +Status: planner-approved. The native planner requested executable workload budgets, cleanup deadlines, a mandatory observer guard and explicit ownership. All four corrections were accepted; its same-draft correction pass returned APPROVED. An optional external challenge was unavailable; it does not replace the passed native review. Active slice: capability/source inventory and frozen plan review. Required delivery: reviewed pushed draft plus private bounded-STG result or concrete blocker. Source baseline refreshed; no live experiment has run under this package. No PR exists yet. Next: planner pass, plan commit, then independent-client implementation. From 940b7e36448448e1f9d2043ddf27ffa6bf07ec00 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 16:00:38 +0200 Subject: [PATCH 2/9] test(codeapi): add independent client and bounded load qualification --- .github/workflows/ci.yml | 3 + .../2026-10-24-codeapi-capabilities.md | 107 +++ ...26-10-24-codeapi-capacity-contract-plan.md | 32 +- scripts/qualification/codeapi.mjs | 802 ++++++++++++++++++ scripts/qualification/codeapi.test.mjs | 395 +++++++++ 5 files changed, 1323 insertions(+), 16 deletions(-) create mode 100644 docs/project/2026-10-24-codeapi-capabilities.md create mode 100644 scripts/qualification/codeapi.mjs create mode 100644 scripts/qualification/codeapi.test.mjs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6e4ff853..4e508023 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -53,6 +53,9 @@ jobs: - name: OpenShell evaluation probe (no gateway) run: node --test scripts/openshell/probe.test.mjs + - name: Independent API qualification controls (offline) + run: node --test scripts/qualification/codeapi.test.mjs + - name: Release versioning run: tests/release-versioning.sh diff --git a/docs/project/2026-10-24-codeapi-capabilities.md b/docs/project/2026-10-24-codeapi-capabilities.md new file mode 100644 index 00000000..3e870376 --- /dev/null +++ b/docs/project/2026-10-24-codeapi-capabilities.md @@ -0,0 +1,107 @@ +# CodeAPI capabilities and adoption guide + +This assessment describes source capabilities, prerequisites and acceptance work. It does not assert that an optional service is deployed or qualified. Detailed environment observations and experiment results belong in the private qualification receipts. The source baseline is fork `4105b53ecfeef1f9eea7a246a36601e181de11cf`; the patch comparison is upstream `v1.10.13` at `2fa9f0d43e8e38b7d6240bc76e2483c6a47c64c7`. + +## Most useful next steps + +| Capability | Useful outcome | Adoption decision | +| -------------------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | +| Independent REST execution and storage files | Applications submit calculations and retrieve owned results without a chat UI | Qualify first with the bounded client package | +| Additional stateless runtimes | Reuse JavaScript/TypeScript libraries or run short shell transformations | Qualify deterministic examples before recommending a language to consumers | +| Replay programmatic tool calling | Code orchestrates several agent tools without holding a sandbox between calls | Adopt for a concrete agent workflow; measure continuation and cancellation behavior | +| Attached remote workspaces | Work against an existing repository or dataset on an operator-owned machine | Separate design and operational package with pairing, lifecycle and storage ownership | +| Hosted apps, extraction and alternative sandbox backends | Preview applications, parse supported documents, or evaluate another isolation platform | Separate opt-in packages; source presence alone is insufficient | + +## Independent REST execution + +`POST /v1/exec` accepts code, a language alias, optional file references and a runtime cap. It returns execution output and generated-file references. Clients can upload a file, execute against that owned input, download the generated result and reuse it in a later execution. These operations require no LibreChat UI or LibreChat client import. The public wire contract is [service/openapi.yml](../../service/openapi.yml), especially `ExecuteRequest`, `ExecuteResponse` and the `/exec` route. + +The caller must distinguish three outcomes: a rejected HTTP request, a completed execution whose guest failed, and an execution whose output delivery failed. HTTP 200 does not prove successful code or intact artifacts. Validate exit code, signal/status, artifact-delivery fields, expected outputs and actual downloaded bytes. The request's `timeout` caps guest runtime; it does not cap queueing or the HTTP transport. Clients need their own whole-request deadline. Queue time also consumes the service job deadline; a longer client timeout cannot fix an overloaded queue. See [router](../../service/src/service/router.ts), [worker](../../service/src/workers.ts) and [deadline calculation](../../service/src/config.ts). + +Authentication is a server trust decision. The JWT implementation binds issuer, audience, key, algorithm and allowed principal sources. A second application needs an approved issuer/trust integration or an existing approved token broker. Copying a signing key into a frontend is not an integration. The legacy `user_id` field does not choose the authenticated owner. Send `X-CodeAPI-Expected-Profile` so a routing mistake produces a profile-mismatch response before work is enqueued. See [JWT trust implementation](../../service/src/auth/librechat-jwt.ts) and [profile fencing](../../service/src/execution-profile.ts). + +Request-rate limits, queue-consumer concurrency/start limits and sandbox resources govern different parts of the path. A 429 with `Retry-After` is admission control, not a guest failure. File operations have independent buckets. Per-principal isolation and application fairness still need qualification under a realistic workload; a direct synthetic-token test does not establish a new application's identity lifecycle. + +The package [@librechat/code](../../packages/code/README.md) implements attached-worker protocols and a CLI. Its existence does not provide a supported general REST client SDK. The qualification client deliberately uses the published HTTP contract and standard Node modules. + +## Stateless languages and curated packages + +Source maps Python, Node.js, Bun JavaScript, Bun TypeScript and bash aliases to runtime identities. The baseline maps Python 3.14.4, Node 24.22.1, Bun 1.4.2 and bash 5.2.0; these are source mappings, not a promise about an arbitrary deployed image. See [language mapping](../../service/src/config.ts) and [sandbox runtimes](../../api/README.md#supported-runtimes). + +Python's documented image includes scientific, plotting, statistical, embedded analytic and geospatial packages. Node/Bun use curated offline packages. The upstream patch candidate moves Bun from 1.4.2 to 1.4.4 in its runtime/build mapping; qualification must bind to the selected version. This supports CSV transformations, charts, numerical work and short data-processing scripts. It is not a general package-install service. Check the exact candidate image before promising a particular package or native dependency. + +Stateless execution gives each request a fresh sandbox. Files survive between requests through explicitly owned storage references, rather than arbitrary process memory or a shared shell. Outbound access is restricted by the selected isolation/backend and egress configuration. Short scripts should not assume Internet access, arbitrary package downloads, an interactive terminal or host filesystem access. The NsJail path documents a separate network namespace and bounded CPU, memory, PID, file and runtime resources in [api/README.md](../../api/README.md). + +Acceptance: execute deterministic examples in each required language; verify effective runtime limits and timeout behavior; check the actual packages a consumer needs; test exact artifact bytes. Performance measurements must separate Python dispatch from the other-language queue. + +## Storage files and metadata + +The stable HTTP surface supports single and bounded batch uploads, listing, metadata, download and deletion. Session kinds distinguish user, skill and agent resources. A reference includes storage session and object identity; a filename alone is insufficient. Generated outputs can be reused as later inputs while retaining ownership checks. See the `/upload`, `/upload/batch`, `/files`, `/download` and `/sessions/.../objects` routes in [OpenAPI](../../service/openapi.yml). + +This object-storage lifecycle is distinct from a persistent worker workspace. Deleting a visible object and receiving a follow-up 404 proves the public object contract, not forensic erasure, backup deletion or a backend TTL policy. A test must retain ownership references before validating a response and clean only its own returned objects. Partial upload or output-delivery failure requires explicit handling; ignoring it creates false success and potential orphaned test data. + +Acceptance: upload/download exact bytes, list and inspect the same owned object, reuse generated input, reject another principal's access, delete every returned owned object and verify absence. Qualify partial failures separately from the happy path. + +## Programmatic tool calling + +Programmatic tool calling lets sandbox code request agent tools, receive their results and continue. Replay mode is the source default: the sandbox can exit between rounds, tool results persist in Redis, and a continuation starts a fresh sandbox with recorded history. Blocking mode keeps a process alive and is a legacy opt-in. Tool-history entries and aggregate history have configured bounds. See [PTC configuration](../../service/src/config.ts), [programmatic router](../../service/src/service/programmatic-router.ts) and [replay state](../../service/src/service/replay-state.ts). + +Replay reduces the need to keep a sandbox occupied while an external tool runs. It also changes the programming model: code before the next tool boundary can be replayed. A workflow must respect the protocol's side-effect/commit semantics and handle history bounds, competing continuations, expiration and cancellation. A running tool-call server does not establish that a particular agent/client implements this protocol correctly. + +Remote bridge and Lambda backends reject blocking PTC. Attached bash replay additionally requires its negotiated worker capability, Bash 5.2+, `jq`, native sandbox commands and the relay. The attached-workspace implementation uses a disposable copy-on-write probe before the final commit pass; do not generalize that protection to an arbitrary client implementation. See [remote bridge](../../docs/remote-bridge/README.md) and [worker CLI](../../packages/code/README.md). + +Acceptance: use one concrete agent workflow with successful continuation, tool error, duplicate/competing continuation and Stop. Measure round-trip overhead and confirm cancellation/cleanup. These routes extend beyond the stable OpenAPI path set, so bind the client to an exact source protocol. + +## Attached workspaces and bring-your-own-machine workers + +Remote Code Bridge connects an operator-owned execution machine outbound to CodeAPI. CodeAPI remains the authentication, policy, queue and result boundary. The worker can keep an existing repository/workspace across turns and restarts. It can run on a suitable Azure VM or another supported machine; this design does not require AWS Lambda. See [bridge guide](../../docs/remote-bridge/README.md), [worker runbook](../../docs/remote-bridge/worker-runbook.md) and [CLI](../../packages/code/README.md). + +This requires the remote-bridge backend, a compatible stateful session configuration and authenticated worker selection. Paired mode binds workers to tenant/principal identities. Pairing and worker credentials have finite lifetimes and proof-of-possession requests bind method, path, body, time and nonce. Static authentication is documented as a development compatibility route, not a replacement for managed pairing. + +Persistence transfers responsibility: the worker owns the workspace and its local data; CodeAPI/Redis own authorization, assignment, fencing and durable-request state. Redis durability and eviction behavior become part of correctness. Lost or ambiguous mutation settlement may fence a workspace and require explicit reset/discard. A reconnect is not proof that an interrupted mutation is safe to replay. + +Native SRT needs the documented OS/runtime tools. Command execution and workspace writes are opt-in; egress requires an explicit policy. Default execution capacity is bounded, and operations on the same workspace root serialize. Negotiated independent-root concurrency is a different capability from parallel writes to one root. Docker/NsJail is an optional adapter with additional host/runtime requirements. + +Acceptance: pairing/revocation, owner isolation, reconnect, real workspace persistence, interrupted mutation/fencing, cleanup and durable state recovery. Adopt this for an identified repository/data workflow rather than as an assumed upgrade to stateless `/exec`. + +## Workspace tools and durable requests + +The workspace API exposes listing, text search, reading, writing, edit preview/application, commands and import. It confines paths to registered roots and bounds results. Read-only tooling is the default subset; writes and commands require their explicit worker policies. A normal HTTP backend cannot serve remote workspace operations. See [workspace router](../../service/src/workspace-tools/router.ts), [selection](../../service/src/bridge/selection.ts) and [OpenAPI workspace routes](../../service/openapi.yml). + +`import_file` bridges an owned uploaded object into a workspace using a bounded one-object read grant. It requires the remote bridge, writes and the egress gateway. The import size/free-space policy and root identity protections are consequential: an uploaded filename is not authorization to write any worker path. The upstream patch comparison includes corrections for missing parent directories and root replacement; qualify these before adopting import. + +Durable workspace requests can return acceptance before completion and expose later status/results. Their queued/admitted/completed/failed/cancelled states require a client that follows the lifecycle. An accepted durable mutation must not be silently retried through a synchronous fallback. Result-retention duration is not the same as workspace-data retention. + +Acceptance: negotiated capabilities, root confinement, writes disabled by default, import ownership and exact bytes, durable status/reconnect/cancellation, and explicit handling of ambiguous settlement. + +## Stateful Lambda sessions and hosted apps + +The optional AWS Lambda MicroVM backend offers warm-session reuse and checkpoint/restore. It requires its AWS region/service support, IAM, connector/image configuration, quotas and checkpoint storage. Stateless, affinity and strict modes make different reuse promises. Checkpoint bytes and transfer time are bounded; persistence is not unlimited memory or indefinite VM lifetime. See [Lambda guide](../../docs/lambda-microvm/README.md). + +Hosted apps are an experimental stateful Lambda capability. They run an immutable app revision from a workspace checkpoint using a dedicated app-host image and preview gateway. They require preview-origin/signing configuration, DNS/TLS and a separate exposure/lifecycle policy. Documented restrictions include finite VM lifetime, no WebSockets and incomplete automatic reclamation of retained revisions. Launch environment variables are not a secret-management service. + +Acceptance: checkpoint and restore exact contents, session expiry, quotas and cost, immutable revision binding, preview authorization, termination and retained-revision cleanup. This is a separate deployment and exposure decision; ordinary CodeAPI artifact success does not qualify it. + +## Document extraction + +Extraction is a separate package/service, not an implicit `/exec` feature. Its protocol extracts structured text from PDF or DOCX and returns an input digest, format and ordered segments. It deliberately rejects unsupported/ambiguous documents rather than silently recovering every format. It does not provide OCR, broad Office rendering, image understanding or a general document-processing pipeline. See [protocol](../../packages/extraction/src/protocol.ts), [server](../../packages/extraction/src/server.ts) and [isolated parser backend](../../packages/extraction/src/backend.ts). + +Baseline limits include bounded input, text, result size, segment count, deadline and concurrent requests. PDF font/encoding limitations and the restricted DOCX body/table model are advertised capabilities, not implementation details clients can ignore. Errors such as unsupported content, encrypted document, output/resource limit and unavailable backend need deliberate consumer handling. + +The baseline uses a separate isolated Unix-socket Compose service. Upstream `v1.10.13` changes its network/credential boundary: an internal network, provisioned credential and hardened endpoint contract replace that earlier local arrangement. Do not copy a baseline setup guide across that update without reviewing its isolation/container tests. Adoption needs a separate image, caller, credential/sync plan and representative-document acceptance. + +## OpenShell + +[OpenShell](../../docs/openshell/README.md) is an evaluation track. There is no enabled backend or accepted production adapter. The offline probe covers its own lifecycle, while runner compatibility, the signed execution/artifact contract and a live gateway remain unqualified. Root-workload restrictions conflict with existing runner requirements in the documented evaluation. This is a research option, not a feature available by changing a backend name. + +## Upstream patch disposition + +[Stable upstream v1.10.13](https://github.com/LibreChat-AI/code-interpreter/releases/tag/v1.10.13) contains extraction boundary changes, workspace root/import correctness and runtime/build alignment. Applicability depends on enabled capabilities. Optional extraction/import fixes do not by themselves establish a defect in every stateless deployment. Preserve the fork's existing trust, cancellation, telemetry and GitOps contracts during source reconciliation. + +The current package records applicability and measures an immutable deployment. Source integration, exact images, staged runtime acceptance and production release authorization remain distinct. Upgrading source is not a substitute for measuring the queue/worker/runner configuration. + +## Qualification and evidence + +The [execution plan](2026-10-24-codeapi-capacity-contract-plan.md) owns the independent-client and finite load experiment. Run `node --test scripts/qualification/codeapi.test.mjs` for offline controls. Live operator use requires an explicit endpoint, authenticated synthetic token supplier, private receipt path and current safety guard. The module deliberately has no default remote endpoint or automatic retry. + +Detailed measurements belong in ignored private receipts. A passing short Service-path run supports that exact client, workload, configuration and transport; it does not prove all packages, ingress, new issuer onboarding, sustained capacity, fairness, fault recovery or a service SLA. diff --git a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md index 30209600..ae623f0b 100644 --- a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md +++ b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md @@ -16,7 +16,7 @@ Done means the client and its failure controls pass offline tests, one approved Target: fork `main`. Baseline: `4105b53ecfeef1f9eea7a246a36601e181de11cf`. Branch: `rs/codeapi-capacity-contract`; worktree: `trees/storage-reliability-investigation/trees/codeapi-capacity-contract`. The existing repository artifact root remains `docs/project/`. History: [reliability roadmap](2026-10-23-codeapi-reliability-roadmap.md). All real-environment observations, identifiers, measurements and detailed review receipts stay in ignored `docs/project/_local/reviews/2026-10-24-capacity-contract/`. Tracked artifacts contain source-derived facts, generic procedures and progress only. -Execution mode: standard. Main owns authority, authentication, live execution, integration and final proof. The read-only capability explorer owns optional-feature mapping. A planner challenges this frozen plan. One bounded implementation worker owns only the client and its offline tests; main owns integration, CI and documentation. Native trusted review covers private operational contracts; externally routed roles receive source-only scope. No separate chat is created. +Execution mode: standard. Main owns authority, authentication, live execution, integration and final proof. The read-only capability explorer completed optional-feature mapping. A planner challenges this frozen plan. The implementation worker returned no patch and was explicitly stopped. Main now exclusively owns the client, tests, CI and documentation; independent review remains required. Native trusted review covers private operational contracts; externally routed roles receive source-only scope. No separate chat is created. No product primitive or production contract changes. ADR gate: not met; this package adds qualification tools and documentation. New deployment ownership, trust or public capability activation would reopen design and authority review. No UI change; screenshots do not apply. @@ -50,30 +50,30 @@ Use existing cluster connectivity only. Service forwarding is an operator transp ### Test portfolio -| Consequential risk | Obligation and seam | Acceptance | -| -------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -| Credential or response leakage / wrong destination | Add transport tests with an ephemeral local HTTP fixture | Redirect, oversized/truncated response and deadline fail closed; no auth in safe receipts; rejected destination receives no request | -| Incorrect outputs or false positive HTTP success | Add synthetic protocol-fixture tests for full client flow | Exact bytes and machine outcomes checked; malformed success fails; all returned owned refs still cleaned | -| Runaway work or incomplete owned cleanup | Add new scheduler/guard/failure tests in the same offline suite | Budget, concurrency, admission-stop and cleanup reserve enforced; failure blocks later phases; cleanup failure remains terminal | -| Runtime and independent-client compatibility | Live contract suite | All required cases pass or precise unsupported behavior is reported without changing the server | -| Python dispatch versus actual capacity | Bounded live phases plus read-only telemetry | Useful throughput/latency and actual useful work measured; pod count alone never passes acceptance | +| Consequential risk | Obligation and seam | Acceptance | +| -------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | +| Credential or response leakage / wrong destination | `add new`: transport tests with an ephemeral local HTTP fixture | Redirect, oversized/truncated response and deadline fail closed; no auth in safe receipts; rejected destination receives no request | +| Incorrect outputs or false positive HTTP success | `add new`: synthetic protocol-fixture tests for full client flow | Exact bytes and machine outcomes checked; malformed success fails; all returned owned refs still cleaned | +| Runaway work or incomplete owned cleanup | `add new`: scheduler/guard/failure tests in the same offline suite | Budget, concurrency, admission-stop and cleanup reserve enforced; failure blocks later phases; cleanup failure remains terminal | +| Runtime and independent-client compatibility | `none`: live contract suite | All required cases pass or precise unsupported behavior is reported without changing the server | +| Python dispatch versus actual capacity | `none`: bounded live phases plus read-only telemetry | Useful throughput/latency and actual useful work measured; pod count alone never passes acceptance | ### Delegation map -| Slice | Route and exclusive owner | Dependency and acceptance | -| --- | --- | --- | -| 1: plan and capability assessment | Main plus read-only source explorer; explorer returns mapping, main writes documents | Frozen source and planner approval; exact source references | -| 2: client and offline tests | Bounded executor owns the two named script paths; main owns CI and integration | Approved plan; offline protocol and failure controls pass | -| 3: STG experiment | Main; secrets and live effects retained for authority/privacy | Qualified tool commit and safety guard; one finite result with owned cleanup | -| 4: review and draft delivery | Main plus read-only final reviewer | Exact committed range, checks and private acceptance; reviewed pushed draft | +| Slice | Route and exclusive owner | Dependency and acceptance | +| --------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | +| 1: plan and capability assessment | Main plus read-only source explorer; explorer returns mapping, main writes documents | Frozen source and planner approval; exact source references | +| 2: client and offline tests | Main owns the two script paths after explicit worker handback; main owns CI and integration | Approved plan; offline protocol and failure controls pass | +| 3: STG experiment | Main; secrets and live effects retained for authority/privacy | Qualified tool commit and safety guard; one finite result with owned cleanup | +| 4: review and draft delivery | Main plus read-only final reviewer | Exact committed range, checks and private acceptance; reviewed pushed draft | ### Slices and delivery 1. Persist this reviewed plan and source-backed capability assessment. Main writes the plan; capability explorer maps optional features. Verify every material capability against immutable source and official documentation. Commit the plan before implementation. -2. Implement the independent client and offline consequential tests. Route the settled implementation to one worker; main integrates the authentication/ownership seams. Run checks in a disposable existing-toolchain container; host Git remains outside it. Add `node --test scripts/qualification/codeapi.test.mjs` to `.github/workflows/ci.yml` in the existing deployment-config job. Commit, simplify and review applicable risk once. +2. Implement the independent client and offline consequential tests. The original worker was stopped without a patch; main implements the settled client and integrates the authentication/ownership seams. Run checks in a disposable existing-toolchain container; host Git remains outside it. Add `node --test scripts/qualification/codeapi.test.mjs` to `.github/workflows/ci.yml` in the existing deployment-config job. Commit, simplify and review applicable risk once. 3. Main runs the live STG preflight, contract and bounded load phases once. Bind evidence privately to actual source/images/configuration and test-tool commit. Collect telemetry through existing read-only paths. Clean only owned files and local forwards, record missing transport/metrics and hold failed dependent phases. No configuration tuning is automatic; prepare an evidence-backed recommendation for a later named window. 4. Complete integrated review over the exact committed package, verify corrections, inspect staged content for real-environment data and secrets, push the task branch and create/update one draft PR. No merge, publication, production rollout or new automation follows from draft delivery. ### Progress -Status: planner-approved. The native planner requested executable workload budgets, cleanup deadlines, a mandatory observer guard and explicit ownership. All four corrections were accepted; its same-draft correction pass returned APPROVED. An optional external challenge was unavailable; it does not replace the passed native review. Active slice: capability/source inventory and frozen plan review. Required delivery: reviewed pushed draft plus private bounded-STG result or concrete blocker. Source baseline refreshed; no live experiment has run under this package. No PR exists yet. Next: planner pass, plan commit, then independent-client implementation. +Status: planner-approved. The native planner requested executable workload budgets, cleanup deadlines, a mandatory observer guard and explicit ownership. All four corrections were accepted; its same-draft correction pass returned APPROVED. An optional external challenge was unavailable; it does not replace the passed native review. Active slice: capability/source inventory and frozen plan review. Required delivery: reviewed pushed draft plus private bounded-STG result or concrete blocker. Plan was committed before implementation; capability mapping and offline client controls are prepared. No live experiment has run under this package. No PR exists yet. Next: committed source risk review, bounded STG acceptance, then integrated draft delivery. diff --git a/scripts/qualification/codeapi.mjs b/scripts/qualification/codeapi.mjs new file mode 100644 index 00000000..df0828b5 --- /dev/null +++ b/scripts/qualification/codeapi.mjs @@ -0,0 +1,802 @@ +import { request as httpRequest } from 'node:http'; +import { request as httpsRequest } from 'node:https'; +import { randomUUID } from 'node:crypto'; +import { open, readFile } from 'node:fs/promises'; +import { pathToFileURL } from 'node:url'; +import { setTimeout as delay } from 'node:timers/promises'; + +const MAX_BYTES = 1024 * 1024; +const ID = /^[A-Za-z0-9_-]{21}$/; +const UUID = /^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i; +const CAPS = Object.freeze({ + requests: 160, + executions: 40, + files: 16, + guestMs: 120000, +}); + +export class QualificationError extends Error { + constructor(reason, status) { + super(reason); + this.reason = reason; + this.status = status; + } +} +const check = (ok, reason) => { + if (!ok) throw new QualificationError(reason); +}; + +export function apiBase(raw) { + let url; + try { + url = new URL(raw); + } catch { + throw new QualificationError('invalid_destination'); + } + check( + !url.username && !url.password && !url.search && !url.hash, + 'invalid_destination' + ); + check( + url.pathname === '/v1' || url.pathname === '/v1/', + 'invalid_destination' + ); + check( + url.protocol === 'https:' || + (url.protocol === 'http:' && + ['127.0.0.1', '[::1]'].includes(url.hostname)), + 'invalid_destination' + ); + url.pathname = '/v1/'; + return url; +} + +// The timer covers headers and the complete body; an idle socket timer alone +// does not bound a server that slowly continues sending bytes. +export function exchange( + base, + path, + { method = 'GET', token, body, headers = {}, timeoutMs = 60000 } = {} +) { + const root = apiBase(String(base)); + check( + typeof path === 'string' && + path.startsWith('/') && + !path.startsWith('//'), + 'invalid_path' + ); + const url = new URL(root.pathname.slice(0, -1) + path, root); + check( + url.origin === root.origin && + url.pathname.startsWith('/v1/') && + !url.hash, + 'invalid_path' + ); + const bytes = body === undefined ? undefined : Buffer.from(body); + check(!bytes || bytes.length <= MAX_BYTES, 'request_too_large'); + check( + Number.isFinite(timeoutMs) && timeoutMs > 0 && timeoutMs <= 60000, + 'invalid_deadline' + ); + return new Promise((resolve, reject) => { + let settled = false; + let res; + let timer; + const finish = (error, result) => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (error) { + res?.destroy(); + req.destroy(); + reject(error); + } else resolve(result); + }; + const req = (url.protocol === 'https:' ? httpsRequest : httpRequest)( + url, + { + method, + agent: false, + headers: { + ...headers, + ...(token ? { Authorization: `Bearer ${token}` } : {}), + ...(bytes ? { 'Content-Length': bytes.length } : {}), + }, + }, + response => { + res = response; + if (res.statusCode >= 300 && res.statusCode < 400) + return finish(new QualificationError('redirect_rejected')); + let size = 0; + const chunks = []; + res.on('data', chunk => { + size += chunk.length; + if (size > MAX_BYTES) + finish(new QualificationError('response_too_large')); + else chunks.push(chunk); + }); + res.on('aborted', () => + finish(new QualificationError('response_truncated')) + ); + res.on('error', () => + finish(new QualificationError('response_error')) + ); + res.on('close', () => { + if (!res.complete) + finish(new QualificationError('response_truncated')); + }); + res.on('end', () => + finish(null, { + status: res.statusCode, + headers: res.headers, + bytes: Buffer.concat(chunks), + }) + ); + } + ); + req.on('error', () => + finish(new QualificationError('transport_error')) + ); + timer = setTimeout( + () => finish(new QualificationError('request_deadline')), + timeoutMs + ); + req.end(bytes); + }); +} + +function json(response) { + try { + return JSON.parse(response.bytes.toString('utf8')); + } catch { + throw new QualificationError('invalid_json'); + } +} +function status(response, expected) { + if (response.status !== expected) + throw new QualificationError( + response.status === 429 ? 'rate_limited' : 'unexpected_http', + response.status + ); +} +function multipart(runId, name, bytes) { + const boundary = `qualification-${randomUUID()}`; + const fields = [ + ['kind', 'user'], + ['id', runId], + ].map( + ([k, v]) => + `--${boundary}\r\nContent-Disposition: form-data; name="${k}"\r\n\r\n${v}\r\n` + ); + return { + body: Buffer.concat([ + Buffer.from( + fields.join('') + + `--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="${name}"\r\nContent-Type: text/plain\r\n\r\n` + ), + bytes, + Buffer.from(`\r\n--${boundary}--\r\n`), + ]), + headers: { + 'Content-Type': `multipart/form-data; boundary=${boundary}`, + }, + }; +} + +export async function guardCheck(path, runId, now = Date.now()) { + let guard; + try { + guard = JSON.parse(await readFile(path, 'utf8')); + } catch { + throw new QualificationError('guard_unavailable'); + } + check( + guard.runId === runId && + guard.ok === true && + Number.isFinite(guard.observedAt) && + now >= guard.observedAt && + now - guard.observedAt <= 20000, + 'guard_rejected' + ); +} + +export function createClient({ + baseUrl, + token, + runId, + guardPath, + receipt = () => {}, + limits = {}, + now = Date.now, + request = exchange, +}) { + const base = apiBase(baseUrl); + check(UUID.test(runId), 'invalid_run_id'); + check(typeof token === 'function', 'token_supplier_required'); + const cap = Object.fromEntries( + Object.entries(CAPS).map(([k, v]) => [k, Math.min(v, limits[k] ?? v)]) + ); + check( + Object.values(cap).every(v => Number.isSafeInteger(v) && v >= 0), + 'invalid_budget' + ); + const start = now(); + const state = { + requests: 0, + executions: 0, + guestMs: 0, + owned: [], + failures: [], + uncertain: false, + stopped: false, + peak: 0, + inFlight: 0, + }; + const fail = error => { + state.stopped = true; + const result = + error instanceof QualificationError + ? { + reason: error.reason, + ...(error.status ? { httpStatus: error.status } : {}), + } + : { reason: 'unexpected_failure' }; + state.failures.push(result); + receipt({ type: 'failure', ...result }); + }; + async function call( + path, + { cleanup = false, guestMs = 0, auth = token, ...options } = {} + ) { + check(!state.stopped || cleanup, 'admission_stopped'); + check(now() - start < (cleanup ? 900000 : 720000), 'run_deadline'); + if (!cleanup) await guardCheck(guardPath, runId, now()); + check(!state.stopped || cleanup, 'admission_stopped'); + check( + state.requests < + (cleanup ? cap.requests : Math.max(0, cap.requests - 40)), + 'request_budget' + ); + + if (path === '/exec') { + check( + state.executions < cap.executions && + state.guestMs + guestMs <= cap.guestMs, + 'execution_budget' + ); + state.executions++; + state.guestMs += guestMs; + } + state.requests++; + const began = now(); + const credential = auth === null ? undefined : await auth(); + check( + auth === null || + (typeof credential === 'string' && credential.length > 0), + 'credential_unavailable' + ); + check(state.inFlight < 4, 'concurrency_budget'); + state.inFlight++; + state.peak = Math.max(state.peak, state.inFlight); + try { + const response = await request(base, path, { + ...options, + token: credential, + timeoutMs: Math.min( + cleanup ? 15000 : 60000, + 900000 - (now() - start) + ), + headers: { + 'X-CodeAPI-Expected-Profile': 'default', + ...options.headers, + }, + }); + receipt({ + type: 'request', + method: options.method ?? 'GET', + operation: cleanup + ? 'cleanup' + : path === '/exec' + ? 'exec' + : 'file_or_contract', + status: response.status, + durationMs: now() - began, + }); + if (response.status === 429) { + const retry = Number(response.headers['retry-after']); + receipt({ + type: 'rate_limit', + retryAfterSeconds: Number.isFinite(retry) ? retry : null, + }); + } + return response; + } catch (error) { + if (path === '/exec' || path === '/upload') state.uncertain = true; + throw error; + } finally { + state.inFlight--; + } + } + const own = (session, id, name) => { + if ( + !ID.test(session ?? '') || + !ID.test(id ?? '') || + typeof name !== 'string' + ) { + state.uncertain = true; + throw new QualificationError('unknown_output_reference'); + } + const existing = state.owned.find( + x => x.id === id && x.storage_session_id === session + ); + if (existing) return existing; + const ref = { + id, + storage_session_id: session, + name, + kind: 'user', + resource_id: runId, + }; + state.owned.push(ref); + receipt({ type: 'owned_object', ...ref }); + check(state.owned.length <= cap.files, 'file_budget'); + return ref; + }; + async function execute( + code, + lang, + { files = [], timeout = 3000, expectTimeout = false } = {} + ) { + const began = now(); + const response = await call('/exec', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ code, lang, timeout, files }), + guestMs: timeout, + }); + if (response.status !== 200 && response.status !== 429) + state.uncertain = true; + status(response, 200); + let value; + try { + value = json(response); + } catch (error) { + state.uncertain = true; + throw error; + } + if (!Array.isArray(value.files)) { + state.uncertain = true; + throw new QualificationError('unknown_output_reference'); + } + const created = []; + let badReference = false; + for (const file of value.files) { + if (file?.inherited === true) continue; + try { + created.push( + own( + file?.storage_session_id ?? value.session_id, + file?.id, + file?.name + ) + ); + } catch { + badReference = true; + } + } + if (badReference) + throw new QualificationError('unknown_output_reference'); + if (value.artifact_delivery) state.uncertain = true; + check(ID.test(value.session_id ?? ''), 'invalid_session'); + if (expectTimeout) + check( + value.code !== 0 && + (value.status === 'TO' || value.signal === 'SIGKILL'), + 'guest_timeout_missing' + ); + else + check( + value.code === 0 && + !value.signal && + !value.status && + !value.artifact_delivery && + value.stderr === '', + 'execution_failed' + ); + check(typeof value.stdout === 'string', 'invalid_stdout'); + return { value, created, durationMs: now() - began }; + } + async function parallel(items, concurrency, fn) { + check( + Number.isInteger(concurrency) && + concurrency >= 1 && + concurrency <= 4, + 'invalid_concurrency' + ); + let next = 0; + await Promise.all( + Array.from({ length: concurrency }, async () => { + while (next < items.length && !state.stopped) { + const item = items[next++]; + try { + await fn(item); + } catch (error) { + fail(error); + } + } + }) + ); + } + async function cleanup() { + const objects = [...state.owned]; + let next = 0; + await Promise.all( + Array.from({ length: Math.min(4, objects.length) }, async () => { + while (next < objects.length) { + const object = objects[next++]; + try { + const path = `/${object.storage_session_id}/${object.id}?kind=user`; + const removed = await call('/files' + path, { + method: 'DELETE', + cleanup: true, + }); + check( + removed.status === 200 || removed.status === 404, + 'delete_failed' + ); + status( + await call('/download' + path, { cleanup: true }), + 404 + ); + receipt({ + type: 'deleted_object', + id: object.id, + storage_session_id: object.storage_session_id, + }); + } catch (error) { + state.uncertain = true; + fail(error); + } + } + }) + ); + } + return { + state, + runId, + call, + own, + execute, + parallel, + cleanup, + fail, + receipt, + now, + stop: () => { + state.stopped = true; + }, + }; +} + +const percentile = (values, p) => + [...values].sort((a, b) => a - b)[ + Math.max(0, Math.ceil(values.length * p) - 1) + ] ?? null; + +async function contract(client, comparisonToken) { + const { call, execute, own, receipt } = client; + status(await call('/health'), 200); + const empty = JSON.stringify({ + code: 'print(1)', + lang: 'python', + timeout: 3000, + }); + status( + await call('/exec', { + method: 'POST', + body: empty, + headers: { 'Content-Type': 'application/json' }, + auth: null, + }), + 401 + ); + const mismatch = await call('/exec', { + method: 'POST', + body: empty, + headers: { + 'Content-Type': 'application/json', + 'X-CodeAPI-Expected-Profile': 'stateful', + }, + }); + status(mismatch, 409); + check( + json(mismatch).error === 'execution_profile_mismatch', + 'profile_mismatch_shape' + ); + status( + await call('/exec', { + method: 'POST', + body: JSON.stringify({ + code: '', + lang: 'unsupported-qualification-language', + timeout: 3000, + }), + headers: { 'Content-Type': 'application/json' }, + }), + 400 + ); + const marker = 'qualification-ok'; + for (const [lang, code] of [ + ['python', `print('${marker}')`], + ['node', `console.log('${marker}')`], + ['js', `console.log('${marker}')`], + ['ts', `const value: string = '${marker}'; console.log(value)`], + ['bash', `printf '${marker}\\n'`], + ]) { + const result = await execute(code, lang); + check( + result.value.stdout === marker + '\n' && + result.created.length === 0, + 'runtime_output_mismatch' + ); + receipt({ + type: 'case', + name: 'runtime', + language: lang, + status: 'passed', + }); + } + const bytes = Buffer.from('key,value\nalpha,2\nbeta,4\n'); + const response = await call('/upload', { + method: 'POST', + ...multipart(client.runId, 'input.csv', bytes), + }); + if (response.status !== 200) client.state.uncertain = true; + status(response, 200); + let uploaded; + try { + uploaded = json(response); + } catch (error) { + client.state.uncertain = true; + throw error; + } + if (!Array.isArray(uploaded.files) || !uploaded.files.length) { + client.state.uncertain = true; + throw new QualificationError('upload_reference_missing'); + } + const refs = uploaded.files.map(f => + own(uploaded.storage_session_id, f.fileId, f.filename) + ); + check(refs.length === 1, 'upload_count_mismatch'); + const input = refs[0]; + const inputDownload = await call( + `/download/${input.storage_session_id}/${input.id}?kind=user` + ); + status(inputDownload, 200); + check(inputDownload.bytes.equals(bytes), 'input_bytes_mismatch'); + const listResponse = await call( + `/files/${input.storage_session_id}?kind=user&detail=normalized` + ); + status(listResponse, 200); + const listing = json(listResponse); + check( + Array.isArray(listing) && + listing.some(f => f.id === input.id || f.fileId === input.id), + 'list_missing_input' + ); + const metadata = await call( + `/sessions/${input.storage_session_id}/objects/${input.id}?kind=user` + ); + status(metadata, 200); + check(json(metadata)?.size === bytes.length, 'invalid_metadata'); + if (comparisonToken) + status( + await call( + `/download/${input.storage_session_id}/${input.id}?kind=user`, + { auth: comparisonToken } + ), + 403 + ); + else + receipt({ + type: 'case', + name: 'ownership_denial', + status: 'not_tested', + }); + let previous = input; + for (const name of ['copy.csv', 'reused.csv']) { + const code = `from pathlib import Path\np=Path('/mnt/data')\np.joinpath('${name}').write_bytes(p.joinpath('${previous.name}').read_bytes())\nprint('copied')`; + const output = await execute(code, 'python', { files: [previous] }); + check( + output.value.stdout === 'copied\n' && + output.created.length === 1 && + output.created[0].name === name, + 'artifact_missing' + ); + previous = output.created[0]; + const download = await call( + `/download/${previous.storage_session_id}/${previous.id}?kind=user` + ); + status(download, 200); + check(download.bytes.equals(bytes), 'artifact_bytes_mismatch'); + } + await execute('import time\ntime.sleep(30)', 'python', { + timeout: 500, + expectTimeout: true, + }); + const recovered = await execute("print('recovered')", 'python'); + check( + recovered.value.stdout === 'recovered\n' && + recovered.created.length === 0, + 'recovery_failed' + ); + receipt({ type: 'case', name: 'contract', status: 'passed' }); +} + +async function load(client, waitMs) { + for (const [name, concurrency, mixed] of [ + ['python_sequential', 1, false], + ['python_burst', 4, false], + ['mixed_burst', 4, true], + ]) { + await delay(waitMs); + const began = client.now(); + const samples = []; + await client.parallel( + Array.from({ length: 8 }, (_, i) => i), + concurrency, + async i => { + const node = mixed && i % 2 === 1; + const marker = 'qualification-load'; + const code = node + ? `const t=performance.now(); await new Promise(r=>setTimeout(r,2000)); console.log('${marker} '+Math.round(performance.now()-t))` + : `import time\nt=time.monotonic()\ntime.sleep(2)\nprint('${marker} '+str(round((time.monotonic()-t)*1000)))`; + const result = await client.execute( + code, + node ? 'node' : 'python' + ); + const match = /^qualification-load (\d+)\n$/.exec( + result.value.stdout + ); + check( + match && + Number(match[1]) >= 1500 && + Number(match[1]) <= 3000 && + result.created.length === 0, + 'load_output_mismatch' + ); + samples.push({ + durationMs: result.durationMs, + guestMs: Number(match[1]), + language: node ? 'node' : 'python', + }); + } + ); + const durationMs = client.now() - began; + client.receipt({ + type: 'phase', + name, + offered: 8, + successful: samples.length, + concurrency, + durationMs, + throughputPerSecond: samples.length / (durationMs / 1000), + p50Ms: percentile( + samples.map(x => x.durationMs), + 0.5 + ), + p95Ms: percentile( + samples.map(x => x.durationMs), + 0.95 + ), + p99Ms: percentile( + samples.map(x => x.durationMs), + 0.99 + ), + samples, + }); + check(!client.state.stopped, 'phase_failed'); + } +} + +export async function runQualification({ + baseUrl, + token, + comparisonToken, + runId = randomUUID(), + guardPath, + receiptPath, + mode = 'contract', + waitMs = 31000, + ...testing +} = {}) { + check(['contract', 'all'].includes(mode), 'invalid_mode'); + check(waitMs === 31000 || testing.request, 'invalid_rate_window'); + check( + typeof receiptPath === 'string' && receiptPath.length > 0, + 'receipt_required' + ); + const file = await open(receiptPath, 'wx', 0o600); + const lines = []; + const receipt = value => lines.push(JSON.stringify(value) + '\n'); + let client; + const stopped = () => { + if (client) { + client.fail(new QualificationError('operator_stopped')); + } + }; + process.on('SIGINT', stopped); + process.on('SIGTERM', stopped); + try { + client = createClient({ + baseUrl, + token, + runId, + guardPath, + receipt, + ...testing, + }); + receipt({ type: 'start', runId, mode }); + try { + await contract(client, comparisonToken); + if (mode === 'all') await load(client, waitMs); + } catch (error) { + client.fail(error); + } + await client.cleanup(); + const summary = { + type: 'summary', + status: + !client.state.failures.length && !client.state.uncertain + ? 'passed' + : 'failed', + requests: client.state.requests, + executions: client.state.executions, + requestedGuestMs: client.state.guestMs, + ownedObjects: client.state.owned.length, + peakRequests: client.state.peak, + cleanupUncertain: client.state.uncertain, + failures: client.state.failures, + }; + receipt(summary); + return summary; + } finally { + process.removeListener('SIGINT', stopped); + process.removeListener('SIGTERM', stopped); + await file.writeFile(lines.join('')); + await file.close(); + } +} + +if ( + process.argv[1] && + import.meta.url === pathToFileURL(process.argv[1]).href +) { + const credential = process.env.CODEAPI_BEARER_TOKEN; + delete process.env.CODEAPI_BEARER_TOKEN; + if (!process.argv.includes('--live') || process.env.CI || !credential) { + process.stderr.write( + 'Explicit operator live mode and scoped authentication are required.\n' + ); + process.exitCode = 2; + } else { + runQualification({ + baseUrl: process.env.CODEAPI_BASE_URL, + token: () => credential, + runId: process.env.CODEAPI_RUN_ID, + guardPath: process.env.CODEAPI_GUARD, + receiptPath: process.env.CODEAPI_RECEIPT, + mode: process.argv.includes('--load') ? 'all' : 'contract', + }) + .then(result => { + process.stdout.write( + JSON.stringify({ status: result.status }) + '\n' + ); + process.exitCode = result.status === 'passed' ? 0 : 1; + }) + .catch(() => { + process.stderr.write('Qualification setup failed.\n'); + process.exitCode = 2; + }); + } +} diff --git a/scripts/qualification/codeapi.test.mjs b/scripts/qualification/codeapi.test.mjs new file mode 100644 index 00000000..dd46c84b --- /dev/null +++ b/scripts/qualification/codeapi.test.mjs @@ -0,0 +1,395 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { createServer } from 'node:http'; +import { mkdtemp, writeFile, readFile, rm, stat } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { setTimeout as delay } from 'node:timers/promises'; +import { + apiBase, + exchange, + guardCheck, + createClient, + runQualification, +} from './codeapi.mjs'; + +async function local(t, handler) { + const server = createServer(handler); + await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)); + t.after(() => { + server.closeAllConnections(); + return new Promise(resolve => server.close(resolve)); + }); + return `http://127.0.0.1:${server.address().port}/v1`; +} +async function files(t) { + const dir = await mkdtemp(join(tmpdir(), 'codeapi-qualification-')); + t.after(() => rm(dir, { recursive: true, force: true })); + const runId = randomUUID(), + guardPath = join(dir, 'guard.json'); + await writeFile( + guardPath, + JSON.stringify({ runId, observedAt: Date.now(), ok: true }) + ); + return { runId, guardPath, receiptPath: join(dir, 'receipt.jsonl') }; +} +const reply = (res, status, body) => { + res.writeHead(status, { 'Content-Type': 'application/json' }); + res.end(JSON.stringify(body)); +}; +const success = (stdout = 'qualification-ok\n', extra = {}) => ({ + session_id: 's'.repeat(21), + code: 0, + signal: null, + status: null, + stdout, + stderr: '', + files: [], + ...extra, +}); + +test('destination confinement rejects unsafe endpoints before any request', async t => { + let requests = 0; + await local(t, (_, res) => { + requests++; + res.end(); + }); + for (const url of [ + 'http://example.org/v1', + 'https://user:password@example.org/v1', + 'https://example.org/v1?x=1', + 'https://example.org/v1#x', + 'https://example.org/other', + 'file:///v1', + ]) { + assert.throws(() => apiBase(url), { reason: 'invalid_destination' }); + } + assert.throws(() => exchange('http://127.0.0.1:1/v1', '//example.org/x'), { + reason: 'invalid_path', + }); + assert.throws(() => exchange('http://127.0.0.1:1/v1', '/../other'), { + reason: 'invalid_path', + }); + assert.equal(requests, 0); +}); + +test('transport rejects redirects, overflow, truncation and whole-response deadlines', async t => { + const base = await local(t, (req, res) => { + if (req.url.endsWith('/redirect')) { + res.writeHead(302, { Location: 'https://example.org/' }); + res.end(); + } + if (req.url.endsWith('/large')) res.end(Buffer.alloc(1024 * 1024 + 1)); + if (req.url.endsWith('/truncated')) { + res.writeHead(200, { 'Content-Length': '1000' }); + res.write('x'); + setImmediate(() => res.destroy()); + } + if (req.url.endsWith('/slow')) { + res.writeHead(200); + res.write('x'); + } + }); + for (const [path, reason] of [ + ['/redirect', 'redirect_rejected'], + ['/large', 'response_too_large'], + ['/truncated', 'response_truncated'], + ['/slow', 'request_deadline'], + ]) { + await assert.rejects(exchange(base, path, { timeoutMs: 100 }), { + reason, + }); + } +}); + +test('guard is invocation-bound, fresh and fail-closed', async t => { + const c = await files(t), + now = Date.now(); + for (const guard of [ + { runId: c.runId, ok: false, observedAt: now }, + { runId: randomUUID(), ok: true, observedAt: now }, + { runId: c.runId, ok: true, observedAt: now - 21000 }, + { runId: c.runId, ok: true, observedAt: now + 1 }, + ]) { + await writeFile(c.guardPath, JSON.stringify(guard)); + await assert.rejects(guardCheck(c.guardPath, c.runId, now), { + reason: 'guard_rejected', + }); + } +}); + +test('admission budgets and four-request concurrency apply before remote work', async t => { + const c = await files(t); + let calls = 0, + active = 0, + peak = 0; + const request = async () => { + calls++; + active++; + peak = Math.max(active, peak); + await delay(5); + active--; + return { + status: 200, + headers: {}, + bytes: Buffer.from(JSON.stringify(success())), + }; + }; + const client = createClient({ + ...c, + baseUrl: 'http://127.0.0.1:1/v1', + token: () => 'synthetic', + request, + limits: { executions: 5 }, + }); + await client.parallel( + Array.from({ length: 30 }, (_, i) => i), + 4, + () => client.execute('pass', 'python') + ); + assert.equal(calls, 5); + assert.ok(peak <= 4); + assert.equal(client.state.stopped, true); + const exhausted = createClient({ + ...c, + baseUrl: 'http://127.0.0.1:1/v1', + token: () => 'synthetic', + request, + limits: { requests: 40 }, + }); + await assert.rejects(exhausted.call('/health'), { + reason: 'request_budget', + }); + assert.equal(calls, 5); +}); + +test('unknown generated references retain every valid owned ref and mark uncertainty', async t => { + const c = await files(t); + const value = success('bad\n', { + files: [ + { + id: 'v'.repeat(21), + name: 'output', + storage_session_id: 's'.repeat(21), + }, + { id: 'invalid', name: 'bad' }, + ], + }); + const client = createClient({ + ...c, + baseUrl: 'http://127.0.0.1:1/v1', + token: () => 'synthetic', + request: async () => ({ + status: 200, + headers: {}, + bytes: Buffer.from(JSON.stringify(value)), + }), + }); + await assert.rejects(client.execute('pass', 'python'), { + reason: 'unknown_output_reference', + }); + assert.equal(client.state.owned.length, 1); + assert.equal(client.state.uncertain, true); +}); + +async function protocol( + t, + { wrongRuntime = false, deleteFailure = false, rateLimit = false } = {} +) { + const objects = new Map(); + let counter = 0, + executions = 0, + peak = 0, + active = 0; + const put = (name, bytes) => { + const id = String(++counter).padStart(21, 'a'); + const ref = { id, name, storage_session_id: 's'.repeat(21) }; + objects.set(id, { ref, bytes }); + return ref; + }; + const baseUrl = await local(t, async (req, res) => { + const chunks = []; + for await (const chunk of req) chunks.push(chunk); + const body = Buffer.concat(chunks), + path = new URL(req.url, 'http://local').pathname; + if (path === '/v1/health') return reply(res, 200, {}); + if (!req.headers.authorization) + return reply(res, 401, { error: 'unauthorized' }); + if (req.headers['x-codeapi-expected-profile'] === 'stateful') + return reply(res, 409, { error: 'execution_profile_mismatch' }); + if (path === '/v1/exec') { + const input = JSON.parse(body); + if (input.lang === 'unsupported-qualification-language') + return reply(res, 400, {}); + if (rateLimit) { + res.setHeader('Retry-After', '30'); + return reply(res, 429, { error: 'rate_limited' }); + } + executions++; + active++; + peak = Math.max(peak, active); + await delay(2); + active--; + if (wrongRuntime) { + const ref = put('unexpected.csv', Buffer.from('owned')); + return reply(res, 200, success('wrong\n', { files: [ref] })); + } + if (input.code.includes('write_bytes')) { + const name = /joinpath\('([^']+)'\)\.write_bytes/.exec( + input.code + )[1]; + const ref = put(name, objects.get(input.files[0].id).bytes); + return reply(res, 200, success('copied\n', { files: [ref] })); + } + if (input.code.includes('sleep(30)')) + return reply( + res, + 200, + success('', { code: 137, signal: 'SIGKILL', status: 'TO' }) + ); + if (input.code.includes('recovered')) + return reply(res, 200, success('recovered\n')); + if (input.code.includes('qualification-load')) + return reply(res, 200, success('qualification-load 2000\n')); + return reply(res, 200, success()); + } + if (path === '/v1/upload') { + const text = body.toString(), + name = /filename="([^"]+)"/.exec(text)[1]; + const bytes = Buffer.from( + text + .split('Content-Type: text/plain\r\n\r\n')[1] + .split('\r\n--')[0] + ); + const ref = put(name, bytes); + return reply(res, 200, { + storage_session_id: ref.storage_session_id, + files: [{ fileId: ref.id, filename: name }], + }); + } + if (req.headers.authorization === 'Bearer other') + return reply(res, 403, {}); + if (req.method === 'DELETE') { + if (deleteFailure) + return reply(res, 503, { + error: 'synthetic server text must not escape', + }); + objects.delete(path.split('/').at(-1)); + return reply(res, 200, {}); + } + if (path.startsWith('/v1/files/')) + return reply( + res, + 200, + [...objects.values()].map(x => x.ref) + ); + const object = objects.get(path.split('/').at(-1)); + if (!object) return reply(res, 404, {}); + if (path.startsWith('/v1/sessions/')) + return reply(res, 200, { size: object.bytes.length }); + res.writeHead(200); + res.end(object.bytes); + }); + return { + baseUrl, + objects, + get executions() { + return executions; + }, + get peak() { + return peak; + }, + }; +} + +test('independent contract, bounded phases and exact artifacts complete with owned cleanup', async t => { + const c = await files(t), + server = await protocol(t); + const result = await runQualification({ + ...c, + baseUrl: server.baseUrl, + token: () => 'synthetic-secret', + comparisonToken: () => 'other', + mode: 'all', + waitMs: 0, + request: exchange, + }); + assert.equal(result.status, 'passed'); + assert.equal(server.objects.size, 0); + assert.ok( + result.executions <= 40 && + result.requests <= 160 && + result.requestedGuestMs <= 120000 + ); + assert.ok(server.peak <= 4); + const receipt = await readFile(c.receiptPath, 'utf8'); + assert.equal(receipt.includes('synthetic-secret'), false); + assert.equal(receipt.includes('qualification-ok'), false); + assert.equal((await stat(c.receiptPath)).mode & 0o777, 0o600); + const phases = receipt + .trim() + .split('\n') + .map(JSON.parse) + .filter(x => x.type === 'phase'); + assert.equal(phases.length, 3); + assert.ok(phases.every(x => x.successful === x.offered)); +}); + +test('malformed HTTP success stops load and still deletes returned owned output', async t => { + const c = await files(t), + server = await protocol(t, { wrongRuntime: true }); + const result = await runQualification({ + ...c, + baseUrl: server.baseUrl, + token: () => 'synthetic', + mode: 'all', + waitMs: 0, + request: exchange, + }); + assert.equal(result.status, 'failed'); + assert.equal(server.executions, 1); + assert.equal(server.objects.size, 0); + assert.equal(result.cleanupUncertain, false); +}); + +test('failed cleanup and ordinary rate limit stay terminal without retries', async t => { + for (const options of [{ deleteFailure: true }, { rateLimit: true }]) { + const c = await files(t), + server = await protocol(t, options); + const result = await runQualification({ + ...c, + baseUrl: server.baseUrl, + token: () => 'synthetic', + mode: 'all', + waitMs: 0, + request: exchange, + }); + assert.equal(result.status, 'failed'); + if (options.deleteFailure) assert.equal(result.cleanupUncertain, true); + else { + assert.equal(server.executions, 0); + assert.ok(result.failures.some(x => x.reason === 'rate_limited')); + } + const receipt = await readFile(c.receiptPath, 'utf8'); + assert.equal( + receipt.includes('synthetic server text must not escape'), + false + ); + } +}); + +test('stale safety guard prevents all traffic but does not prevent owned cleanup', async t => { + const c = await files(t), + server = await protocol(t); + const client = createClient({ + ...c, + baseUrl: server.baseUrl, + token: () => 'synthetic', + }); + client.own('s'.repeat(21), 'a'.repeat(21), 'synthetic'); + await writeFile(c.guardPath, '{}'); + await assert.rejects(client.call('/health'), { reason: 'guard_rejected' }); + await client.cleanup(); + assert.equal(client.state.requests, 2); +}); From 432187451e1a6cd444c72759323a1b6a1466f6d9 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 16:11:18 +0200 Subject: [PATCH 3/9] test(codeapi): simplify fixture and document operator prerequisites --- docs/project/2026-10-24-codeapi-capabilities.md | 4 +++- scripts/qualification/codeapi.test.mjs | 8 +------- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/project/2026-10-24-codeapi-capabilities.md b/docs/project/2026-10-24-codeapi-capabilities.md index 3e870376..a334037b 100644 --- a/docs/project/2026-10-24-codeapi-capabilities.md +++ b/docs/project/2026-10-24-codeapi-capabilities.md @@ -102,6 +102,8 @@ The current package records applicability and measures an immutable deployment. ## Qualification and evidence -The [execution plan](2026-10-24-codeapi-capacity-contract-plan.md) owns the independent-client and finite load experiment. Run `node --test scripts/qualification/codeapi.test.mjs` for offline controls. Live operator use requires an explicit endpoint, authenticated synthetic token supplier, private receipt path and current safety guard. The module deliberately has no default remote endpoint or automatic retry. +The [execution plan](2026-10-24-codeapi-capacity-contract-plan.md) owns the independent-client and finite load experiment. Run `node --test scripts/qualification/codeapi.test.mjs` for offline controls. Live operator use requires an explicit endpoint, authenticated synthetic token supplier, private receipt path and current safety guard. The module deliberately has no default remote endpoint or automatic retry. Its operator CLI is `node scripts/qualification/codeapi.mjs --live` (add `--load` for the finite phases). Provide `CODEAPI_BASE_URL`, `CODEAPI_BEARER_TOKEN`, `CODEAPI_RUN_ID`, `CODEAPI_GUARD` and `CODEAPI_RECEIPT` through the operator environment. CI runs only the offline suite. A static bearer must remain valid for the run; a trusted operator can instead import `runQualification` and supply a fresh in-memory token function. + +The caller-owned observer writes an atomic private guard containing `runId`, `observedAt` (epoch milliseconds) and `ok`. The UUID must match this invocation and the observation must be no older than twenty seconds. It must come from the live health/queue checks in the plan; writing a fixed passing fixture is not a live qualification. The included offline tests create only synthetic local fixtures. Receipt creation is exclusive and private; a pre-existing receipt path is rejected. The CLI never creates a cluster connection or changes a deployment. Detailed measurements belong in ignored private receipts. A passing short Service-path run supports that exact client, workload, configuration and transport; it does not prove all packages, ingress, new issuer onboarding, sustained capacity, fairness, fault recovery or a service SLA. diff --git a/scripts/qualification/codeapi.test.mjs b/scripts/qualification/codeapi.test.mjs index dd46c84b..81ea8f3f 100644 --- a/scripts/qualification/codeapi.test.mjs +++ b/scripts/qualification/codeapi.test.mjs @@ -49,12 +49,7 @@ const success = (stdout = 'qualification-ok\n', extra = {}) => ({ ...extra, }); -test('destination confinement rejects unsafe endpoints before any request', async t => { - let requests = 0; - await local(t, (_, res) => { - requests++; - res.end(); - }); +test('destination confinement rejects unsafe endpoints before any request', () => { for (const url of [ 'http://example.org/v1', 'https://user:password@example.org/v1', @@ -71,7 +66,6 @@ test('destination confinement rejects unsafe endpoints before any request', asyn assert.throws(() => exchange('http://127.0.0.1:1/v1', '/../other'), { reason: 'invalid_path', }); - assert.equal(requests, 0); }); test('transport rejects redirects, overflow, truncation and whole-response deadlines', async t => { From 9f4b2e1a1b73cfd36c416d1ba3812aa771f1431e Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 16:19:01 +0200 Subject: [PATCH 4/9] fix(qualification): preserve receipts and report incomplete coverage --- .../2026-10-24-codeapi-capabilities.md | 2 +- ...26-10-24-codeapi-capacity-contract-plan.md | 14 ++--- scripts/qualification/codeapi.mjs | 62 +++++++++++++++---- scripts/qualification/codeapi.test.mjs | 47 +++++++++++++- 4 files changed, 103 insertions(+), 22 deletions(-) diff --git a/docs/project/2026-10-24-codeapi-capabilities.md b/docs/project/2026-10-24-codeapi-capabilities.md index a334037b..aa478513 100644 --- a/docs/project/2026-10-24-codeapi-capabilities.md +++ b/docs/project/2026-10-24-codeapi-capabilities.md @@ -102,7 +102,7 @@ The current package records applicability and measures an immutable deployment. ## Qualification and evidence -The [execution plan](2026-10-24-codeapi-capacity-contract-plan.md) owns the independent-client and finite load experiment. Run `node --test scripts/qualification/codeapi.test.mjs` for offline controls. Live operator use requires an explicit endpoint, authenticated synthetic token supplier, private receipt path and current safety guard. The module deliberately has no default remote endpoint or automatic retry. Its operator CLI is `node scripts/qualification/codeapi.mjs --live` (add `--load` for the finite phases). Provide `CODEAPI_BASE_URL`, `CODEAPI_BEARER_TOKEN`, `CODEAPI_RUN_ID`, `CODEAPI_GUARD` and `CODEAPI_RECEIPT` through the operator environment. CI runs only the offline suite. A static bearer must remain valid for the run; a trusted operator can instead import `runQualification` and supply a fresh in-memory token function. +The [execution plan](2026-10-24-codeapi-capacity-contract-plan.md) owns the independent-client and finite load experiment. Run `node --test scripts/qualification/codeapi.test.mjs` for offline controls. Live operator use requires an explicit endpoint, authenticated synthetic token supplier, private receipt path and current safety guard. The module deliberately has no default remote endpoint or automatic retry. Its operator CLI is `node scripts/qualification/codeapi.mjs --live` (add `--load` for the finite phases). Provide `CODEAPI_BASE_URL`, `CODEAPI_BEARER_TOKEN`, `CODEAPI_RUN_ID`, `CODEAPI_GUARD` and `CODEAPI_RECEIPT` through the operator environment. Set `CODEAPI_COMPARISON_BEARER_TOKEN` for the owned-object denial case; without it the result is partial. CI runs only the offline suite. A static bearer must remain valid for the run; a trusted operator can instead import `runQualification` and supply a fresh in-memory token function. The caller-owned observer writes an atomic private guard containing `runId`, `observedAt` (epoch milliseconds) and `ok`. The UUID must match this invocation and the observation must be no older than twenty seconds. It must come from the live health/queue checks in the plan; writing a fixed passing fixture is not a live qualification. The included offline tests create only synthetic local fixtures. Receipt creation is exclusive and private; a pre-existing receipt path is rejected. The CLI never creates a cluster connection or changes a deployment. diff --git a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md index ae623f0b..2309ac19 100644 --- a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md +++ b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md @@ -60,12 +60,12 @@ Use existing cluster connectivity only. Service forwarding is an operator transp ### Delegation map -| Slice | Route and exclusive owner | Dependency and acceptance | -| --------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | -| 1: plan and capability assessment | Main plus read-only source explorer; explorer returns mapping, main writes documents | Frozen source and planner approval; exact source references | -| 2: client and offline tests | Main owns the two script paths after explicit worker handback; main owns CI and integration | Approved plan; offline protocol and failure controls pass | -| 3: STG experiment | Main; secrets and live effects retained for authority/privacy | Qualified tool commit and safety guard; one finite result with owned cleanup | -| 4: review and draft delivery | Main plus read-only final reviewer | Exact committed range, checks and private acceptance; reviewed pushed draft | +| Slice | Route and exclusive owner | Dependency and acceptance | +| --------------------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| 1: plan and capability assessment | Main plus read-only source explorer; explorer returns mapping, main writes documents | Frozen source and planner approval; exact source references | +| 2: client and offline tests | Main owns the two script paths after explicit worker handback; main owns CI and integration | Approved plan; offline protocol and failure controls pass | +| 3: STG experiment | Main; secrets and live effects retained for authority/privacy | Qualified tool commit and safety guard; one finite result with owned cleanup | +| 4: review and draft delivery | Main plus read-only final reviewer | Exact committed range, checks and private acceptance; reviewed pushed draft | ### Slices and delivery @@ -76,4 +76,4 @@ Use existing cluster connectivity only. Service forwarding is an operator transp ### Progress -Status: planner-approved. The native planner requested executable workload budgets, cleanup deadlines, a mandatory observer guard and explicit ownership. All four corrections were accepted; its same-draft correction pass returned APPROVED. An optional external challenge was unavailable; it does not replace the passed native review. Active slice: capability/source inventory and frozen plan review. Required delivery: reviewed pushed draft plus private bounded-STG result or concrete blocker. Plan was committed before implementation; capability mapping and offline client controls are prepared. No live experiment has run under this package. No PR exists yet. Next: committed source risk review, bounded STG acceptance, then integrated draft delivery. +Status: planner-approved. The native planner requested executable workload budgets, cleanup deadlines, a mandatory observer guard and explicit ownership. All four corrections were accepted; its same-draft correction pass returned APPROVED. An optional external challenge was unavailable; it does not replace the passed native review. Active slice: client risk-review correction before the bounded operator experiment. Required delivery: reviewed pushed draft plus private bounded-STG result or concrete blocker. Plan was committed before implementation; capability mapping and offline client controls are prepared. No live experiment has run under this package. No PR exists yet. Next: committed source risk review, bounded STG acceptance, then integrated draft delivery. diff --git a/scripts/qualification/codeapi.mjs b/scripts/qualification/codeapi.mjs index df0828b5..90906ab4 100644 --- a/scripts/qualification/codeapi.mjs +++ b/scripts/qualification/codeapi.mjs @@ -2,6 +2,7 @@ import { request as httpRequest } from 'node:http'; import { request as httpsRequest } from 'node:https'; import { randomUUID } from 'node:crypto'; import { open, readFile } from 'node:fs/promises'; +import { writeSync } from 'node:fs'; import { pathToFileURL } from 'node:url'; import { setTimeout as delay } from 'node:timers/promises'; @@ -209,6 +210,7 @@ export function createClient({ limits = {}, now = Date.now, request = exchange, + receiptHealthy = () => true, }) { const base = apiBase(baseUrl); check(UUID.test(runId), 'invalid_run_id'); @@ -249,6 +251,7 @@ export function createClient({ { cleanup = false, guestMs = 0, auth = token, ...options } = {} ) { check(!state.stopped || cleanup, 'admission_stopped'); + check(cleanup || receiptHealthy(), 'receipt_unavailable'); check(now() - start < (cleanup ? 900000 : 720000), 'run_deadline'); if (!cleanup) await guardCheck(guardPath, runId, now()); check(!state.stopped || cleanup, 'admission_stopped'); @@ -318,27 +321,27 @@ export function createClient({ } } const own = (session, id, name) => { - if ( - !ID.test(session ?? '') || - !ID.test(id ?? '') || - typeof name !== 'string' - ) { + if (!ID.test(session ?? '') || !ID.test(id ?? '')) { state.uncertain = true; throw new QualificationError('unknown_output_reference'); } const existing = state.owned.find( x => x.id === id && x.storage_session_id === session ); - if (existing) return existing; + if (existing) { + check(typeof name === 'string', 'unusable_output_name'); + return existing; + } const ref = { id, storage_session_id: session, - name, + name: typeof name === 'string' ? name : '', kind: 'user', resource_id: runId, }; state.owned.push(ref); - receipt({ type: 'owned_object', ...ref }); + receipt({ type: 'owned_object', id, storage_session_id: session }); + check(typeof name === 'string', 'unusable_output_name'); check(state.owned.length <= cap.files, 'file_budget'); return ref; }; @@ -717,8 +720,26 @@ export async function runQualification({ 'receipt_required' ); const file = await open(receiptPath, 'wx', 0o600); - const lines = []; - const receipt = value => lines.push(JSON.stringify(value) + '\n'); + let receiptFailed = false; + const receipt = value => { + if (receiptFailed) return; + try { + const bytes = Buffer.from(JSON.stringify(value) + '\n'); + let offset = 0; + while (offset < bytes.length) { + const written = writeSync( + file.fd, + bytes, + offset, + bytes.length - offset + ); + if (written <= 0) throw new Error(); + offset += written; + } + } catch { + receiptFailed = true; + } + }; let client; const stopped = () => { if (client) { @@ -727,6 +748,7 @@ export async function runQualification({ }; process.on('SIGINT', stopped); process.on('SIGTERM', stopped); + process.on('SIGHUP', stopped); try { client = createClient({ baseUrl, @@ -734,6 +756,7 @@ export async function runQualification({ runId, guardPath, receipt, + receiptHealthy: () => !receiptFailed, ...testing, }); receipt({ type: 'start', runId, mode }); @@ -747,8 +770,12 @@ export async function runQualification({ const summary = { type: 'summary', status: - !client.state.failures.length && !client.state.uncertain - ? 'passed' + !client.state.failures.length && + !client.state.uncertain && + !receiptFailed + ? comparisonToken + ? 'passed' + : 'partial' : 'failed', requests: client.state.requests, executions: client.state.executions, @@ -756,14 +783,20 @@ export async function runQualification({ ownedObjects: client.state.owned.length, peakRequests: client.state.peak, cleanupUncertain: client.state.uncertain, + receiptFailed, + untested: comparisonToken ? [] : ['ownership_denial'], failures: client.state.failures, }; receipt(summary); + if (receiptFailed) { + summary.status = 'failed'; + summary.receiptFailed = true; + } return summary; } finally { process.removeListener('SIGINT', stopped); process.removeListener('SIGTERM', stopped); - await file.writeFile(lines.join('')); + process.removeListener('SIGHUP', stopped); await file.close(); } } @@ -773,6 +806,8 @@ if ( import.meta.url === pathToFileURL(process.argv[1]).href ) { const credential = process.env.CODEAPI_BEARER_TOKEN; + const comparison = process.env.CODEAPI_COMPARISON_BEARER_TOKEN; + delete process.env.CODEAPI_COMPARISON_BEARER_TOKEN; delete process.env.CODEAPI_BEARER_TOKEN; if (!process.argv.includes('--live') || process.env.CI || !credential) { process.stderr.write( @@ -783,6 +818,7 @@ if ( runQualification({ baseUrl: process.env.CODEAPI_BASE_URL, token: () => credential, + comparisonToken: comparison ? () => comparison : undefined, runId: process.env.CODEAPI_RUN_ID, guardPath: process.env.CODEAPI_GUARD, receiptPath: process.env.CODEAPI_RECEIPT, diff --git a/scripts/qualification/codeapi.test.mjs b/scripts/qualification/codeapi.test.mjs index 81ea8f3f..499d4e6a 100644 --- a/scripts/qualification/codeapi.test.mjs +++ b/scripts/qualification/codeapi.test.mjs @@ -185,6 +185,32 @@ test('unknown generated references retain every valid owned ref and mark uncerta }); assert.equal(client.state.owned.length, 1); assert.equal(client.state.uncertain, true); + const missingName = createClient({ + ...c, + baseUrl: 'http://127.0.0.1:1/v1', + token: () => 'synthetic', + request: async () => ({ + status: 200, + headers: {}, + bytes: Buffer.from( + JSON.stringify( + success('', { + files: [ + { + id: 'v'.repeat(21), + storage_session_id: 's'.repeat(21), + }, + ], + }) + ) + ), + }), + }); + await assert.rejects(missingName.execute('pass', 'python'), { + reason: 'unknown_output_reference', + }); + assert.equal(missingName.state.owned.length, 1); + assert.equal(missingName.state.uncertain, false); }); async function protocol( @@ -307,7 +333,13 @@ test('independent contract, bounded phases and exact artifacts complete with own comparisonToken: () => 'other', mode: 'all', waitMs: 0, - request: exchange, + request: async (...args) => { + const activeReceipt = await readFile(c.receiptPath, 'utf8'); + assert.ok(activeReceipt.includes('"type":"start"')); + if (args[1].startsWith('/download/')) + assert.ok(activeReceipt.includes('"type":"owned_object"')); + return exchange(...args); + }, }); assert.equal(result.status, 'passed'); assert.equal(server.objects.size, 0); @@ -330,6 +362,19 @@ test('independent contract, bounded phases and exact artifacts complete with own assert.ok(phases.every(x => x.successful === x.offered)); }); +test('missing comparison identity reports partial qualification after cleanup', async t => { + const c = await files(t), + server = await protocol(t); + const result = await runQualification({ + ...c, + baseUrl: server.baseUrl, + token: () => 'synthetic', + }); + assert.equal(result.status, 'partial'); + assert.deepEqual(result.untested, ['ownership_denial']); + assert.equal(server.objects.size, 0); +}); + test('malformed HTTP success stops load and still deletes returned owned output', async t => { const c = await files(t), server = await protocol(t, { wrongRuntime: true }); From 5b7d807db85b2c9d16d2cba595465d7624b187f6 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 16:32:41 +0200 Subject: [PATCH 5/9] docs(project): finalize qualification source delivery checkpoint --- docs/project/2026-10-24-codeapi-capacity-contract-plan.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md index 2309ac19..2b957505 100644 --- a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md +++ b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md @@ -76,4 +76,8 @@ Use existing cluster connectivity only. Service forwarding is an operator transp ### Progress -Status: planner-approved. The native planner requested executable workload budgets, cleanup deadlines, a mandatory observer guard and explicit ownership. All four corrections were accepted; its same-draft correction pass returned APPROVED. An optional external challenge was unavailable; it does not replace the passed native review. Active slice: client risk-review correction before the bounded operator experiment. Required delivery: reviewed pushed draft plus private bounded-STG result or concrete blocker. Plan was committed before implementation; capability mapping and offline client controls are prepared. No live experiment has run under this package. No PR exists yet. Next: committed source risk review, bounded STG acceptance, then integrated draft delivery. +Status: source package implemented, simplified and reviewed. The plan preceded implementation. All source review findings are dispositioned, and the independent final correction passed. The offline protocol/failure controls and focused source checks pass. Substantive scope: the qualification client, its tests and one existing CI step; no deployed service code changes. + +Delivery: [draft PR #46](https://github.com/uzh-bf/code-interpreter/pull/46), branch `rs/codeapi-capacity-contract`. Source-qualified head: `9f4b2e1a1b73cfd36c416d1ba3812aa771f1431e`. Documentation-only progress updates preserve that source evidence. The operational continuation, acceptance results, cleanup accounting and any next-window recommendation belong exclusively in the existing ignored review directory. Consult those private receipts before any release or tuning decision. + +The source package ends at reviewed draft delivery. Merge, runtime publication, capacity/concurrency changes, optional-feature activation and production effects retain their named gates. No additional source slice is active in this package. From 680e3d2b9a7eb91acc3afe3b0113f57270140186 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 19:26:01 +0200 Subject: [PATCH 6/9] docs(project): plan bounded resource and stability qualification --- ...26-10-24-codeapi-capacity-contract-plan.md | 31 ++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md index 2b957505..1bd22154 100644 --- a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md +++ b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md @@ -80,4 +80,33 @@ Status: source package implemented, simplified and reviewed. The plan preceded i Delivery: [draft PR #46](https://github.com/uzh-bf/code-interpreter/pull/46), branch `rs/codeapi-capacity-contract`. Source-qualified head: `9f4b2e1a1b73cfd36c416d1ba3812aa771f1431e`. Documentation-only progress updates preserve that source evidence. The operational continuation, acceptance results, cleanup accounting and any next-window recommendation belong exclusively in the existing ignored review directory. Consult those private receipts before any release or tuning decision. -The source package ends at reviewed draft delivery. Merge, runtime publication, capacity/concurrency changes, optional-feature activation and production effects retain their named gates. No additional source slice is active in this package. +The source package ends at reviewed draft delivery. Merge, runtime publication, capacity/concurrency changes, optional-feature activation and production effects retain their named gates. The approved resource-and-stability extension below is now active in the same draft package. + + +## Approved resource-and-stability extension + +### Approval and terminal condition + +The user approved the proposed CPU-heavy and longer-duration STG tests, requested Claude discussion, and asked for goal-driven execution. Extend the existing draft [PR #46](https://github.com/uzh-bf/code-interpreter/pull/46); do not create a stack or another test framework. Only the existing qualification module, its tests and this plan change. Main owns private observations, credentials and final proof; the source executor owns the two qualification files. A native planner challenges this extension, and Claude supplies source-only advice and integrated review. Execution mode remains standard. + +Authority covers one bounded synthetic STG resource/stability invocation after offline checks and source review, existing-context observation, existing-profile signing, task-owned Service forwards and owned test-file cleanup. Preparation that admits no CodeAPI workload does not consume the invocation. No deployed setting, image, replica/node capacity, trust scope or production behavior changes. A failed or incomplete invocation preserves evidence and ends without automatic load replay. Done means the reviewed source is delivered in the existing draft and the finite test either passes its complete coverage and safety checks or records a precise blocker, with all owned cleanup accounted for. + +### Workload and accounting + +Add an explicit `resources` mode; existing contract/all modes and transport bounds remain unchanged. Run the existing full contract first. After a full ordinary rate window, execute six deterministic Python CPU jobs at two in flight. Each computes a fixed one-million-step SHA-256 chain with a per-call nonce equal to UTF-8 `runId:phase:index`. Start from those nonce bytes; each of one million iterations replaces the current bytes with SHA-256(current bytes). Compare the final lowercase hex digest against Node crypto computed independently. All soak CPU calls use this same recurrence. Record guest CPU time, guest wall time and client elapsed time separately; require a positive CPU observation and exact output, without inferring configured CPU capacity. After another full rate window, execute four memory jobs at two in flight. Each touches every page of a 32 MiB allocation with a nonce-derived one-byte pattern and SHA-256 of the entire allocation after touching each page. Node soak memory jobs allocate the same 32 MiB, fill it with the same pattern and hash the entire Buffer. Verify the byte count, nonce and full-buffer hash independently. Both programs report process CPU and monotonic wall deltas. Peak test allocation may include two 32 MiB copies; require the effective configured guest limit at least 128 MiB, leaving at least 64 MiB for interpreter/other memory. Stop before workload admission on a lower/unknown effective limit; do not infer the runtime limit from a source default alone. No intentional OOM or extra timeout test. + +Then run exactly sixteen paced slots spanning ten minutes, one every 37.5 seconds. Alternate Python CPU work and Node memory/integrity work. Memory-to-soak overlap is at most five calls in any thirty-second window (four memory plus one soak); live admission requires an effective execution window no longer than thirty seconds and allowance at least twelve, covering the earlier contract. No third full-window wait is needed. The first slot starts at the soak anchor; the last at anchor plus 562.5 seconds. The observation interval is `[anchor, anchor + 600 seconds]`; after the final result wait only the remainder of that interval. If the final response extends beyond that interval, mark coverage incomplete, settle it within the existing sixty-second request deadline, and perform cleanup within the whole-run bound. Require the soak anchor within 100 seconds of invocation start, so the planned window ends within 700 seconds and preserves the existing 720-second admission cutoff and 900-second whole-run/cleanup limit. If preparation or an earlier call makes this impossible, stop before soak admission and report incomplete coverage. Use monotonic scheduling, fail coverage on a slot delayed by over ten seconds, and never compress missed slots or issue catch-up requests. The final two slots also prove ordinary execution after the earlier resource phases. Do not adapt the passing sample count or calibrate/replay against the live deployment. + +The exact maximum ledger is 12 contract execution-route calls plus 6 CPU, 4 memory and 16 soak calls: 38 of 40. Requested guest runtime is 24.5 plus 18 plus 12 plus 48 seconds: 102.5 of 120. The unchanged contract costs 26 HTTP requests including six owned cleanup calls; adding 26 resource executions gives an expected-path total of 52 HTTP requests, leaving the existing forty-call cleanup reserve and total ceiling of 160 intact. There is one synchronous HTTP exec response per guest, no polling. Guard reads are local. Token signing is per request in memory with the same principal, so a short JWT lifetime does not expire the long invocation. No new output files are expected from resource phases; track and clean any returned reference before rejecting it. Three contract files remain owned until ordinary cleanup. + +### Safety, observations and interpretation + +Keep the fresh UUID-bound guard and ordinary stop conditions. Stable API/worker/file/egress/tool control pods retain their UID/readiness/restart bindings. Record runner UIDs and replica changes under the existing KEDA owner/bounds. For this finite window, keep the conservative original-runner health guard: disappearance or restart of an original runner stops admission as an inconclusive health change, even if healthy autoscaling could explain it. Do not infer a crash from missing historical state, relax guards during the run, or alter replica ownership. Optional new runner pods may be created by ordinary KEDA; require a ready floor and no node MemoryPressure. + +Use a bounded observer at the existing five-second cadence, at most 180 cycles under the 900-second envelope. Kubernetes/Prometheus read-only observations are separate from the CodeAPI HTTP ledger. Queue freshness/readiness remain mandatory for safety. Use fixed five-second soak bins: 120 expected bins, with a missed bin counted missing. Require CPU and working-set coverage for each initial worker/runner container in at least 108 bins, and nonempty fresh observations during both resource phases. Record new runner metrics without substituting their series for a missing original. Coverage for the bound worker/runner containers must meet these counts, using a disclosed sixty-second freshness ceiling; otherwise resource acceptance is incomplete while exact functional evidence stands. Record sampled CPU, memory, queue and ready-runner series, plus early/late ranges and raw receipt pointers privately. Missing data is not zero. After the absolute observation end, require one new successful observer cycle and check its UUID-bound fresh guard before ordinary completion; a failure in the final idle interval fails the run. Bound final queue-drain verification to sixty seconds without admitting more work, and require both existing storage CronJobs’ latest owner-bound completed Jobs/logs no older than twenty minutes. No per-job queue-delay estimate or maximum-capacity, leak-free, percentile/SLO, cold-zero-pool, fairness or exactly-once claim follows from this periodic bounded workload. + +Pass requires the entire contract, six CPU jobs, four memory jobs and sixteen on-time soak slots with exact output/no unexpected artifacts, no unexpected HTTP or guest failures, no guard/receipt uncertainty, the required metric coverage, final drained queues, fresh owner-bound storage canaries and verified owned cleanup. An ordinary 429 is recorded without retry and makes coverage incomplete; it is not proof of a service defect. A configured-memory headroom mismatch holds admission; do not intentionally discover the limit through failure. Source/live drift or missing existing connectivity holds live work. No rollback is needed because the package changes no deployed setting. + +### Extension delivery + +Before live admission: persist and commit this reviewed plan, implement the two named source files, run offline contract/scheduler/failure checks in the existing Node 24 container toolchain, inspect the exact diff and obtain source review. Tests cover deterministic output validation, total budgets, monotonic pacing/deadline refusal and failure-stop/cleanup behavior with a fake clock/transport; no ten-minute local sleep is needed. Reuse ordinary CI without secrets or Kubernetes access. Resource test risks extend the existing qualification boundary; capacity claims remain absent beyond bounded evidence. Add a bounded real execution check of generated Python and Node fixtures with known-answer outputs in existing local/container runtimes, plus corrupt-result rejection. No live tuning or replays follow from a local speed observation. The source package ends at reviewed draft delivery; source merge and production rollout remain separate gates. Keep all measurements and operational continuation in ignored `docs/project/_local/reviews/2026-10-24-resource-soak/`. From 9d640c42726ad52d88993940a2518b0b4450b7bd Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 19:42:51 +0200 Subject: [PATCH 7/9] test(codeapi): add bounded CPU memory and stability phases --- ...26-10-24-codeapi-capacity-contract-plan.md | 4 +- scripts/qualification/codeapi.mjs | 199 +++++++++++++++++- scripts/qualification/codeapi.test.mjs | 175 ++++++++++++++- 3 files changed, 372 insertions(+), 6 deletions(-) diff --git a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md index 1bd22154..3103f5eb 100644 --- a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md +++ b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md @@ -87,13 +87,13 @@ The source package ends at reviewed draft delivery. Merge, runtime publication, ### Approval and terminal condition -The user approved the proposed CPU-heavy and longer-duration STG tests, requested Claude discussion, and asked for goal-driven execution. Extend the existing draft [PR #46](https://github.com/uzh-bf/code-interpreter/pull/46); do not create a stack or another test framework. Only the existing qualification module, its tests and this plan change. Main owns private observations, credentials and final proof; the source executor owns the two qualification files. A native planner challenges this extension, and Claude supplies source-only advice and integrated review. Execution mode remains standard. +The user approved the proposed CPU-heavy and longer-duration STG tests, requested Claude discussion, and asked for goal-driven execution. Extend the existing draft [PR #46](https://github.com/uzh-bf/code-interpreter/pull/46); do not create a stack or another test framework. Only the existing qualification module, its tests and this plan change. Main owns private observations, credentials and final proof; Main owns the two qualification files after the source executor returned its discovery checkpoint without a patch. A native planner challenges this extension, and Claude supplies source-only advice and integrated review. Execution mode remains standard. Authority covers one bounded synthetic STG resource/stability invocation after offline checks and source review, existing-context observation, existing-profile signing, task-owned Service forwards and owned test-file cleanup. Preparation that admits no CodeAPI workload does not consume the invocation. No deployed setting, image, replica/node capacity, trust scope or production behavior changes. A failed or incomplete invocation preserves evidence and ends without automatic load replay. Done means the reviewed source is delivered in the existing draft and the finite test either passes its complete coverage and safety checks or records a precise blocker, with all owned cleanup accounted for. ### Workload and accounting -Add an explicit `resources` mode; existing contract/all modes and transport bounds remain unchanged. Run the existing full contract first. After a full ordinary rate window, execute six deterministic Python CPU jobs at two in flight. Each computes a fixed one-million-step SHA-256 chain with a per-call nonce equal to UTF-8 `runId:phase:index`. Start from those nonce bytes; each of one million iterations replaces the current bytes with SHA-256(current bytes). Compare the final lowercase hex digest against Node crypto computed independently. All soak CPU calls use this same recurrence. Record guest CPU time, guest wall time and client elapsed time separately; require a positive CPU observation and exact output, without inferring configured CPU capacity. After another full rate window, execute four memory jobs at two in flight. Each touches every page of a 32 MiB allocation with a nonce-derived one-byte pattern and SHA-256 of the entire allocation after touching each page. Node soak memory jobs allocate the same 32 MiB, fill it with the same pattern and hash the entire Buffer. Verify the byte count, nonce and full-buffer hash independently. Both programs report process CPU and monotonic wall deltas. Peak test allocation may include two 32 MiB copies; require the effective configured guest limit at least 128 MiB, leaving at least 64 MiB for interpreter/other memory. Stop before workload admission on a lower/unknown effective limit; do not infer the runtime limit from a source default alone. No intentional OOM or extra timeout test. +Add an explicit `resources` mode; existing contract/all modes and transport bounds remain unchanged. Run the existing full contract first. After a full ordinary rate window, execute six deterministic Python CPU jobs at two in flight. Each computes a fixed one-million-step SHA-256 chain with a per-call nonce equal to UTF-8 `runId:phase:index`. Start from those nonce bytes; each of one million iterations replaces the current bytes with SHA-256(current bytes). Compare the final lowercase hex digest against Node crypto computed independently. All soak CPU calls use this same recurrence. Record guest CPU time, guest wall time and client elapsed time separately; require a positive CPU observation and exact output, without inferring configured CPU capacity. Each finite resource phase includes a five-second observation hold after its calls. After another full rate window, execute four memory jobs at two in flight. Each touches every page of a 32 MiB allocation with a nonce-derived one-byte pattern and SHA-256 of the entire allocation after touching each page. Node soak memory jobs allocate the same 32 MiB, fill it with the same pattern and hash the entire Buffer. Verify the byte count, nonce and full-buffer hash independently. Both programs report process CPU and monotonic wall deltas. Peak test allocation may include two 32 MiB copies; require the effective configured guest limit at least 128 MiB, leaving at least 64 MiB for interpreter/other memory. Stop before workload admission on a lower/unknown effective limit; do not infer the runtime limit from a source default alone. No intentional OOM or extra timeout test. Then run exactly sixteen paced slots spanning ten minutes, one every 37.5 seconds. Alternate Python CPU work and Node memory/integrity work. Memory-to-soak overlap is at most five calls in any thirty-second window (four memory plus one soak); live admission requires an effective execution window no longer than thirty seconds and allowance at least twelve, covering the earlier contract. No third full-window wait is needed. The first slot starts at the soak anchor; the last at anchor plus 562.5 seconds. The observation interval is `[anchor, anchor + 600 seconds]`; after the final result wait only the remainder of that interval. If the final response extends beyond that interval, mark coverage incomplete, settle it within the existing sixty-second request deadline, and perform cleanup within the whole-run bound. Require the soak anchor within 100 seconds of invocation start, so the planned window ends within 700 seconds and preserves the existing 720-second admission cutoff and 900-second whole-run/cleanup limit. If preparation or an earlier call makes this impossible, stop before soak admission and report incomplete coverage. Use monotonic scheduling, fail coverage on a slot delayed by over ten seconds, and never compress missed slots or issue catch-up requests. The final two slots also prove ordinary execution after the earlier resource phases. Do not adapt the passing sample count or calibrate/replay against the live deployment. diff --git a/scripts/qualification/codeapi.mjs b/scripts/qualification/codeapi.mjs index 90906ab4..541d25ab 100644 --- a/scripts/qualification/codeapi.mjs +++ b/scripts/qualification/codeapi.mjs @@ -1,6 +1,6 @@ import { request as httpRequest } from 'node:http'; import { request as httpsRequest } from 'node:https'; -import { randomUUID } from 'node:crypto'; +import { randomUUID, createHash } from 'node:crypto'; import { open, readFile } from 'node:fs/promises'; import { writeSync } from 'node:fs'; import { pathToFileURL } from 'node:url'; @@ -199,6 +199,7 @@ export async function guardCheck(path, runId, now = Date.now()) { now - guard.observedAt <= 20000, 'guard_rejected' ); + return guard; } export function createClient({ @@ -702,6 +703,179 @@ async function load(client, waitMs) { } } +const RESOURCE_BYTES = 32 * 1024 * 1024; +const HASH_STEPS = 1000000; + +export function resourceWorkload(kind, nonce) { + check( + ['cpu', 'memory', 'node_memory'].includes(kind), + 'invalid_resource_kind' + ); + check( + typeof nonce === 'string' && /^[a-zA-Z0-9:_-]{1,160}$/.test(nonce), + 'invalid_resource_nonce' + ); + const memory = kind !== 'cpu'; + const pattern = createHash('sha256').update(nonce).digest()[0]; + let value = Buffer.from(nonce); + if (memory) value = Buffer.alloc(RESOURCE_BYTES, pattern); + else + for (let i = 0; i < HASH_STEPS; i++) + value = createHash('sha256').update(value).digest(); + const digest = memory + ? createHash('sha256').update(value).digest('hex') + : value.toString('hex'); + const literal = JSON.stringify(nonce); + const code = + kind === 'node_memory' + ? `// qualification-resource:node_memory +import {createHash} from 'node:crypto'; +const nonce=${literal}; +const cpu=process.cpuUsage(), wall=performance.now(); +const data=Buffer.alloc(${RESOURCE_BYTES},${pattern}); +for(let i=0;i 0 && + Number.isFinite(output.wallMs) && + output.wallMs > 0 && + output.wallMs <= 3000 && + result.created.length === 0, + 'resource_output_mismatch' + ); + const item = { + phase, + index, + language: work.lang, + kind, + bytes: work.bytes, + cpuMs: output.cpuMs, + guestWallMs: output.wallMs, + durationMs: result.durationMs, + }; + client.receipt({ type: 'resource_sample', ...item }); + return item; + } + for (const [name, count, kind] of [ + ['cpu', 6, 'cpu'], + ['memory', 4, 'memory'], + ]) { + await pause(waitMs); + const startedAt = client.now(), + phaseBegan = monotonic(); + const samples = []; + await client.parallel( + Array.from({ length: count }, (_, i) => i), + 2, + async i => samples.push(await sample(kind, name, i)) + ); + check( + !client.state.stopped && samples.length === count, + 'resource_phase_incomplete' + ); + await pause(5000); + client.receipt({ + type: 'resource_phase', + name, + offered: count, + successful: samples.length, + concurrency: 2, + startedAt, + endedAt: client.now(), + durationMs: monotonic() - phaseBegan, + }); + } + const anchor = monotonic(), + wallAnchor = client.now(); + check(anchor - began <= 100000, 'soak_anchor_deadline'); + const end = anchor + 600000; + client.receipt({ + type: 'soak_start', + startedAt: wallAnchor, + durationMs: 600000, + offered: 16, + spacingMs: 37500, + }); + for (let i = 0; i < 16; i++) { + const due = anchor + i * 37500; + await pause(Math.max(0, due - monotonic())); + check(monotonic() - due <= 10000, 'soak_slot_missed'); + check(!client.state.stopped, 'admission_stopped'); + await sample(i % 2 ? 'node_memory' : 'cpu', 'soak', i); + check(monotonic() <= end, 'soak_response_overrun'); + } + await pause(Math.max(0, end - monotonic())); + // Admission checks do not prove health during the final observation-only interval. + const finalBound = monotonic() + 10000; + let finalGuard; + while (monotonic() < finalBound) { + finalGuard = await guardCheck(guardPath, client.runId, client.now()); + if (finalGuard.observedAt >= wallAnchor + 600000) break; + await pause(1000); + } + check( + finalGuard.observedAt >= wallAnchor + 600000 && !client.state.stopped, + 'final_observation_missing' + ); + client.receipt({ + type: 'resource_phase', + name: 'soak', + offered: 16, + successful: 16, + concurrency: 1, + startedAt: wallAnchor, + endedAt: client.now(), + durationMs: monotonic() - anchor, + }); +} + export async function runQualification({ baseUrl, token, @@ -713,7 +887,14 @@ export async function runQualification({ waitMs = 31000, ...testing } = {}) { - check(['contract', 'all'].includes(mode), 'invalid_mode'); + check(['contract', 'all', 'resources'].includes(mode), 'invalid_mode'); + check( + (!testing.monotonic && !testing.pause) || testing.request, + 'test_clock_requires_transport' + ); + const monotonic = testing.monotonic ?? (() => performance.now()); + const pause = testing.pause ?? delay; + const began = monotonic(); check(waitMs === 31000 || testing.request, 'invalid_rate_window'); check( typeof receiptPath === 'string' && receiptPath.length > 0, @@ -763,6 +944,14 @@ export async function runQualification({ try { await contract(client, comparisonToken); if (mode === 'all') await load(client, waitMs); + if (mode === 'resources') + await resources(client, { + waitMs, + began, + monotonic, + pause, + guardPath, + }); } catch (error) { client.fail(error); } @@ -822,7 +1011,11 @@ if ( runId: process.env.CODEAPI_RUN_ID, guardPath: process.env.CODEAPI_GUARD, receiptPath: process.env.CODEAPI_RECEIPT, - mode: process.argv.includes('--load') ? 'all' : 'contract', + mode: process.argv.includes('--resources') + ? 'resources' + : process.argv.includes('--load') + ? 'all' + : 'contract', }) .then(result => { process.stdout.write( diff --git a/scripts/qualification/codeapi.test.mjs b/scripts/qualification/codeapi.test.mjs index 499d4e6a..e3163c9e 100644 --- a/scripts/qualification/codeapi.test.mjs +++ b/scripts/qualification/codeapi.test.mjs @@ -12,6 +12,7 @@ import { guardCheck, createClient, runQualification, + resourceWorkload, } from './codeapi.mjs'; async function local(t, handler) { @@ -215,7 +216,12 @@ test('unknown generated references retain every valid owned ref and mark uncerta async function protocol( t, - { wrongRuntime = false, deleteFailure = false, rateLimit = false } = {} + { + wrongRuntime = false, + deleteFailure = false, + rateLimit = false, + corruptResource = false, + } = {} ) { const objects = new Map(); let counter = 0, @@ -255,6 +261,29 @@ async function protocol( const ref = put('unexpected.csv', Buffer.from('owned')); return reply(res, 200, success('wrong\n', { files: [ref] })); } + const resource = + /qualification-resource:(cpu|memory|node_memory)/.exec( + input.code + ); + if (resource) { + const nonce = JSON.parse( + /nonce=("[^"\n]+")/.exec(input.code)[1] + ); + const work = resourceWorkload(resource[1], nonce); + return reply( + res, + 200, + success( + JSON.stringify({ + nonce, + digest: corruptResource ? 'invalid' : work.digest, + bytes: work.bytes, + cpuMs: 25, + wallMs: 30, + }) + '\n' + ) + ); + } if (input.code.includes('write_bytes')) { const name = /joinpath\('([^']+)'\)\.write_bytes/.exec( input.code @@ -432,3 +461,147 @@ test('stale safety guard prevents all traffic but does not prevent owned cleanup await client.cleanup(); assert.equal(client.state.requests, 2); }); + +test('resource workload oracle has independent known answers and bounded allocations', () => { + const cpu = resourceWorkload('cpu', 'synthetic-fixture'); + assert.equal( + cpu.digest, + '6a03d93e5d75c8e8d3b5e5f8e54f5db6710e3cf7c8bc8128b8506c827cb7b6f1' + ); + assert.equal(cpu.bytes, 0); + for (const kind of ['memory', 'node_memory']) { + const work = resourceWorkload(kind, 'synthetic-fixture'); + assert.equal(work.bytes, 32 * 1024 * 1024); + assert.equal(work.digest.length, 64); + } + assert.throws(() => resourceWorkload('cpu', 'invalid\nnonce')); +}); + +async function resourceRun( + t, + { + lateAnchor = false, + lateSlot = false, + finalFailure = false, + corruptResource = false, + } = {} +) { + const c = await files(t), + server = await protocol(t, { corruptResource }); + const epoch = Date.now(); + let elapsed = 0, + delayed = false; + const result = await runQualification({ + ...c, + baseUrl: server.baseUrl, + token: () => 'synthetic-secret', + comparisonToken: () => 'other', + mode: 'resources', + waitMs: 31000, + now: () => epoch + elapsed, + monotonic: () => elapsed, + pause: async ms => { + elapsed += ms; + if (lateSlot && !delayed && ms === 37500) { + elapsed += 11000; + delayed = true; + } + await writeFile( + c.guardPath, + JSON.stringify({ + runId: c.runId, + observedAt: epoch + elapsed, + ok: !(finalFailure && elapsed >= 672000), + }) + ); + }, + request: async (...args) => { + if (lateAnchor && args[1] === '/health') elapsed += 35000; + // This fixture observes local fake time only, while production uses the external observer. + await writeFile( + c.guardPath, + JSON.stringify({ + runId: c.runId, + observedAt: epoch + elapsed, + ok: true, + }) + ); + return exchange(...args); + }, + }); + const events = (await readFile(c.receiptPath, 'utf8')) + .trim() + .split('\n') + .map(JSON.parse); + assert.equal(server.objects.size, 0); + assert.ok( + result.requests <= 160 && + result.executions <= 40 && + result.requestedGuestMs <= 120000 + ); + assert.equal( + (await readFile(c.receiptPath, 'utf8')).includes('synthetic-secret'), + false + ); + return { result, events, elapsed }; +} + +test('resource phases and all paced slots fit one bounded invocation with owned cleanup', async t => { + const { result, events, elapsed } = await resourceRun(t); + assert.equal(result.status, 'passed'); + assert.equal(result.executions, 38); + assert.equal(result.requests, 52); + assert.equal(result.requestedGuestMs, 102500); + assert.equal(result.ownedObjects, 3); + const phases = events.filter(x => x.type === 'resource_phase'); + assert.deepEqual( + phases.map(x => [x.name, x.offered, x.successful]), + [ + ['cpu', 6, 6], + ['memory', 4, 4], + ['soak', 16, 16], + ] + ); + assert.equal(phases[2].durationMs, 600000); + assert.equal(elapsed, 672000); +}); + +test('late anchor or missed slot cannot silently shrink or bunch soak coverage', async t => { + for (const options of [{ lateAnchor: true }, { lateSlot: true }]) { + const { result, events } = await resourceRun(t, options); + assert.equal(result.status, 'failed'); + assert.ok( + result.failures.some( + x => + x.reason === + (options.lateAnchor + ? 'soak_anchor_deadline' + : 'soak_slot_missed') + ) + ); + assert.equal( + events.some(x => x.type === 'resource_phase' && x.name === 'soak'), + false + ); + } +}); + +test('corrupt resource results stop admission and final-idle guard failure cannot pass', async t => { + for (const options of [{ corruptResource: true }, { finalFailure: true }]) { + const { result, events } = await resourceRun(t, options); + assert.equal(result.status, 'failed'); + assert.ok( + result.failures.some( + x => + x.reason === + (options.corruptResource + ? 'resource_output_mismatch' + : 'guard_rejected') + ) + ); + assert.equal( + events.some(x => x.type === 'resource_phase' && x.name === 'soak'), + false + ); + } +}); From 48d0bd917fde0370794744bbeb986083011e5f9f Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 19:54:17 +0200 Subject: [PATCH 8/9] fix(qualification): preserve soak margin and phase timing --- ...26-10-24-codeapi-capacity-contract-plan.md | 2 +- scripts/qualification/codeapi.mjs | 26 +++++++++++++------ scripts/qualification/codeapi.test.mjs | 2 +- 3 files changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md index 3103f5eb..46ba13f8 100644 --- a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md +++ b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md @@ -93,7 +93,7 @@ Authority covers one bounded synthetic STG resource/stability invocation after o ### Workload and accounting -Add an explicit `resources` mode; existing contract/all modes and transport bounds remain unchanged. Run the existing full contract first. After a full ordinary rate window, execute six deterministic Python CPU jobs at two in flight. Each computes a fixed one-million-step SHA-256 chain with a per-call nonce equal to UTF-8 `runId:phase:index`. Start from those nonce bytes; each of one million iterations replaces the current bytes with SHA-256(current bytes). Compare the final lowercase hex digest against Node crypto computed independently. All soak CPU calls use this same recurrence. Record guest CPU time, guest wall time and client elapsed time separately; require a positive CPU observation and exact output, without inferring configured CPU capacity. Each finite resource phase includes a five-second observation hold after its calls. After another full rate window, execute four memory jobs at two in flight. Each touches every page of a 32 MiB allocation with a nonce-derived one-byte pattern and SHA-256 of the entire allocation after touching each page. Node soak memory jobs allocate the same 32 MiB, fill it with the same pattern and hash the entire Buffer. Verify the byte count, nonce and full-buffer hash independently. Both programs report process CPU and monotonic wall deltas. Peak test allocation may include two 32 MiB copies; require the effective configured guest limit at least 128 MiB, leaving at least 64 MiB for interpreter/other memory. Stop before workload admission on a lower/unknown effective limit; do not infer the runtime limit from a source default alone. No intentional OOM or extra timeout test. +Add an explicit `resources` mode (`node scripts/qualification/codeapi.mjs --live --resources`; this takes precedence over `--load`); existing contract/all modes and transport bounds remain unchanged. Run the existing full contract first. After the contract, refuse resource admission unless its elapsed time plus the two rate waits, two observation holds and a conservative twenty-second finite-phase allowance fit the one-hundred-second soak anchor. Compute phase oracles during the preceding rate wait, before timing or dispatching the concurrent phase; preserve the full wait by subtracting only actual preparation time. After a full ordinary rate window, execute six deterministic Python CPU jobs at two in flight. Each computes a fixed one-million-step SHA-256 chain with a per-call nonce equal to UTF-8 `runId:phase:index`. Start from those nonce bytes; each of one million iterations replaces the current bytes with SHA-256(current bytes). Compare the final lowercase hex digest against Node crypto computed independently. All soak CPU calls use this same recurrence. Record guest CPU time, guest wall time and client elapsed time separately; require a positive CPU observation and exact output, without inferring configured CPU capacity. Each finite resource phase includes a five-second observation hold after its calls. After another full rate window, execute four memory jobs at two in flight. Each touches every page of a 32 MiB allocation with a nonce-derived one-byte pattern and SHA-256 of the entire allocation after touching each page. Node soak memory jobs allocate the same 32 MiB, fill it with the same pattern and hash the entire Buffer. Verify the byte count, nonce and full-buffer hash independently. Both programs report process CPU and monotonic wall deltas. Peak test allocation may include two 32 MiB copies; require the effective configured guest limit at least 128 MiB, leaving at least 64 MiB for interpreter/other memory. Stop before workload admission on a lower/unknown effective limit; do not infer the runtime limit from a source default alone. No intentional OOM or extra timeout test. Then run exactly sixteen paced slots spanning ten minutes, one every 37.5 seconds. Alternate Python CPU work and Node memory/integrity work. Memory-to-soak overlap is at most five calls in any thirty-second window (four memory plus one soak); live admission requires an effective execution window no longer than thirty seconds and allowance at least twelve, covering the earlier contract. No third full-window wait is needed. The first slot starts at the soak anchor; the last at anchor plus 562.5 seconds. The observation interval is `[anchor, anchor + 600 seconds]`; after the final result wait only the remainder of that interval. If the final response extends beyond that interval, mark coverage incomplete, settle it within the existing sixty-second request deadline, and perform cleanup within the whole-run bound. Require the soak anchor within 100 seconds of invocation start, so the planned window ends within 700 seconds and preserves the existing 720-second admission cutoff and 900-second whole-run/cleanup limit. If preparation or an earlier call makes this impossible, stop before soak admission and report incomplete coverage. Use monotonic scheduling, fail coverage on a slot delayed by over ten seconds, and never compress missed slots or issue catch-up requests. The final two slots also prove ordinary execution after the earlier resource phases. Do not adapt the passing sample count or calibrate/replay against the live deployment. diff --git a/scripts/qualification/codeapi.mjs b/scripts/qualification/codeapi.mjs index 541d25ab..ee860ea3 100644 --- a/scripts/qualification/codeapi.mjs +++ b/scripts/qualification/codeapi.mjs @@ -766,11 +766,14 @@ async function resources( client, { waitMs, began, monotonic, pause, guardPath } ) { - async function sample(kind, phase, index) { - const work = resourceWorkload( - kind, - `${client.runId}:${phase}:${index}` - ); + check( + monotonic() - began + 2 * waitMs + 10000 + 20000 <= 100000, + 'soak_preflight_deadline' + ); + async function sample(kind, phase, index, prepared) { + const work = + prepared ?? + resourceWorkload(kind, `${client.runId}:${phase}:${index}`); const result = await client.execute(work.code, work.lang); let output; try { @@ -807,14 +810,18 @@ async function resources( ['cpu', 6, 'cpu'], ['memory', 4, 'memory'], ]) { - await pause(waitMs); + const preparationBegan = monotonic(); + const prepared = Array.from({ length: count }, (_, i) => + resourceWorkload(kind, `${client.runId}:${name}:${i}`) + ); + await pause(Math.max(0, waitMs - (monotonic() - preparationBegan))); const startedAt = client.now(), phaseBegan = monotonic(); const samples = []; await client.parallel( Array.from({ length: count }, (_, i) => i), 2, - async i => samples.push(await sample(kind, name, i)) + async i => samples.push(await sample(kind, name, i, prepared[i])) ); check( !client.state.stopped && samples.length === count, @@ -848,7 +855,10 @@ async function resources( await pause(Math.max(0, due - monotonic())); check(monotonic() - due <= 10000, 'soak_slot_missed'); check(!client.state.stopped, 'admission_stopped'); - await sample(i % 2 ? 'node_memory' : 'cpu', 'soak', i); + const kind = i % 2 ? 'node_memory' : 'cpu'; + const prepared = resourceWorkload(kind, `${client.runId}:soak:${i}`); + check(monotonic() - due <= 10000, 'soak_slot_missed'); + await sample(kind, 'soak', i, prepared); check(monotonic() <= end, 'soak_response_overrun'); } await pause(Math.max(0, end - monotonic())); diff --git a/scripts/qualification/codeapi.test.mjs b/scripts/qualification/codeapi.test.mjs index e3163c9e..59fce646 100644 --- a/scripts/qualification/codeapi.test.mjs +++ b/scripts/qualification/codeapi.test.mjs @@ -575,7 +575,7 @@ test('late anchor or missed slot cannot silently shrink or bunch soak coverage', x => x.reason === (options.lateAnchor - ? 'soak_anchor_deadline' + ? 'soak_preflight_deadline' : 'soak_slot_missed') ) ); From 70f9da32bf2b60b46b58d0f32786b15ee6255b70 Mon Sep 17 00:00:00 2001 From: Roland Schlaefli Date: Sat, 24 Oct 2026 20:54:57 +0200 Subject: [PATCH 9/9] docs(project): checkpoint reviewed resource qualification source --- .../project/2026-10-24-codeapi-capacity-contract-plan.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md index 46ba13f8..0f96ef29 100644 --- a/docs/project/2026-10-24-codeapi-capacity-contract-plan.md +++ b/docs/project/2026-10-24-codeapi-capacity-contract-plan.md @@ -78,9 +78,9 @@ Use existing cluster connectivity only. Service forwarding is an operator transp Status: source package implemented, simplified and reviewed. The plan preceded implementation. All source review findings are dispositioned, and the independent final correction passed. The offline protocol/failure controls and focused source checks pass. Substantive scope: the qualification client, its tests and one existing CI step; no deployed service code changes. -Delivery: [draft PR #46](https://github.com/uzh-bf/code-interpreter/pull/46), branch `rs/codeapi-capacity-contract`. Source-qualified head: `9f4b2e1a1b73cfd36c416d1ba3812aa771f1431e`. Documentation-only progress updates preserve that source evidence. The operational continuation, acceptance results, cleanup accounting and any next-window recommendation belong exclusively in the existing ignored review directory. Consult those private receipts before any release or tuning decision. +Delivery: [draft PR #46](https://github.com/uzh-bf/code-interpreter/pull/46), branch `rs/codeapi-capacity-contract`. Source-qualified head: `48d0bd917fde0370794744bbeb986083011e5f9f`. Documentation-only progress updates preserve that source evidence. The operational continuation, acceptance results, cleanup accounting and any next-window recommendation belong exclusively in the existing ignored review directory. Consult those private receipts before any release or tuning decision. -The source package ends at reviewed draft delivery. Merge, runtime publication, capacity/concurrency changes, optional-feature activation and production effects retain their named gates. The approved resource-and-stability extension below is now active in the same draft package. +The source package ends at reviewed draft delivery. Merge, runtime publication, capacity/concurrency changes, optional-feature activation and production effects retain their named gates. The resource-and-stability source extension below is implemented and reviewed in the same draft package. Environment acceptance remains in the private continuation. ## Approved resource-and-stability extension @@ -110,3 +110,8 @@ Pass requires the entire contract, six CPU jobs, four memory jobs and sixteen on ### Extension delivery Before live admission: persist and commit this reviewed plan, implement the two named source files, run offline contract/scheduler/failure checks in the existing Node 24 container toolchain, inspect the exact diff and obtain source review. Tests cover deterministic output validation, total budgets, monotonic pacing/deadline refusal and failure-stop/cleanup behavior with a fake clock/transport; no ten-minute local sleep is needed. Reuse ordinary CI without secrets or Kubernetes access. Resource test risks extend the existing qualification boundary; capacity claims remain absent beyond bounded evidence. Add a bounded real execution check of generated Python and Node fixtures with known-answer outputs in existing local/container runtimes, plus corrupt-result rejection. No live tuning or replays follow from a local speed observation. The source package ends at reviewed draft delivery; source merge and production rollout remain separate gates. Keep all measurements and operational continuation in ignored `docs/project/_local/reviews/2026-10-24-resource-soak/`. + + +### Extension source progress + +The opt-in resource/stability mode and its failure controls are implemented. Fourteen offline cases pass in the pinned Node container, and the generated Python and Node programs pass bounded real-runtime checks. Integrated source review and its focused correction pass are complete. [Hosted source CI](https://github.com/uzh-bf/code-interpreter/actions/runs/40118267705) passed at the source-qualified head above. This checkpoint changes documentation only and preserves that source verification. Optional scoped scanner rule retrieval was unavailable; no passing scan is claimed. The source remains in [draft PR #46](https://github.com/uzh-bf/code-interpreter/pull/46); runtime evidence and the operational continuation are maintained privately.