Foundation: CI executes every notebook offline and validates fixtures

Goal

The gate every recipe pull request has to pass. Once this is merged, recipes can start.

Scope

A CI job that discovers every recipes/*/notebook.ipynb, executes it from a fresh environment with no key and JEV_COOKBOOK_LIVE unset, and fails on any error.
Fixture validation for every recipe folder.
A guard that fails if a notebook attempts a network call to the TypeSafe API during the offline run.
A check that committed notebook outputs match a fresh offline execution closely enough to catch stale outputs (define the comparison and document it).
A check that a recipe pull request changes only its own recipes/NN-slug/ folder, reported as a clear message.
Per-recipe execution in a matrix or with path filtering, so sixty notebooks do not make every run slow.

Acceptance criteria

The template recipe passes all jobs
A notebook with a failing cell, an invalid fixture, and a stale output each make CI fail, demonstrated in the pull request description
Full run time with one recipe is recorded, with a note on how it scales
Lint and tests pass in CI, or locally with the output pasted in the pull request if CI does not exist yet

Dependencies

Blocked by Foundation: CI baseline for lint, tests, and the README catalog check #62 (CI baseline for lint, tests, and the README catalog check)
Blocked by Foundation: recipe template and scaffolder #68 (recipe template and scaffolder)
Unblocks every recipe issue (Recipe 01: Sentiment classification #1 to Recipe 60: Factory production control #60)
Tracked in Epic: Foundation #74

References

Recipe contract
Catalog
TypeSafe documentation index for tools: https://docs.typesafe.ai/llms.txt
Python SDK: https://docs.typesafe.ai/sdk/python

Foundation: make the #69 scope contract an allowlist rendered from ba…

Catalog coordination review handoff from #87 / PR #88: the final exact-head Opus approval is #88 (comment). The reviewer tested 33 crafted commits against the specified exhaustive allowlist/base-README rendering contract.

For #69 implementation and regression evidence:

Inspect raw Git diff modes/types (for example git diff --raw -M -z), not only name-status. Name-status alone lists symlinks and submodules as ordinary additions and hides executable mode changes.
Enforce allowed modes for new files as well as rejecting prohibited type/mode changes; new executable files must not evade the ordinary-file contract.
Apply the outside-path rejection before parsing a head catalog that is forbidden to change, so malformed unauthorized catalog edits have a useful rejection diagnostic.
Test newly published notebook + exact renderer output from the base README as accepted, unauthorized prose/markers/newlines/generated-row edits as rejected, every outside-path class, both sides of renames, deletions, symlinks, submodules, modes and types.

Do not substitute self-rendering head README comparison: that preserves arbitrary prose edits. This is a handoff to the existing future #69 worker, not a second implementation owner. Both independent Opus reviewers must inspect the actual implemented rejection tests at the final #69 head.

Orchestrator note. The scope rule in this issue's text ("a recipe pull request changes only its own recipes/NN-slug/ folder") is superseded by the allowlist being settled in #87 / PR #88: a recipe pull request may change its own recipes/<slug>/ folder plus exactly README.md, and README.md must equal the base README rendered with the head catalog and head recipes/ tree, byte for byte (no symlink, submodule, mode, or type changes anywhere in the allowlist). Build the scope check to that rule once #88 is merged, include a way to tell recipe pull requests from foundation ones (branch name recipe/ and the Closes #N with N in 1..60), and fail closed.

Foundation: reconcile recipe catalog integration and orchestration ru…

Orchestrator note 2 (after #88 merged). The allowlist is now in CONTRIBUTING.md (evaluated on git diff --raw -M base...head). Two edge cases the #88 reviewer left for this issue to decide, and which the scope check must handle: (S5) a deletion inside the recipe folder shows as :100644 000000 ... D; treat a deletion as having no resulting mode rather than rejecting it under the "every new or resulting mode must be 100644" rule. (S1) a pull request that edits a recipe folder without both recipe markers (branch recipe/<slug> and a closing reference to issue N in 1..60) must not escape the allowlist; require the branch slug to match the catalog slug for N, count every GitHub closing keyword, and fail a pull request that closes more than one recipe issue.

Orchestrator note 3. Notebook execution must run with the recipe folder as the working directory (nbclient does not default to the notebook's folder; pass the recipe path explicitly). The fixture validator (#65 / PR #94) will report a recipe's mode (replay or scripted): scripted and simulator recipes may have empty replay_keys and no responses.json; the network guard still applies to them.

Foundation: fixture specification and validator (#94)

Execution-prompt handoff for the future sole #69 Sonnet builder (implementation remains gated on delivered #68):

PR99's description reports that PNG bytes differ between matplotlib versions and explicitly leaves network guarding and staleness comparison to this issue. Read its final reviewed implementation and both Opus findings before designing the gate.

The supplied execution prompt requires clean kernels, API credentials absent, and network access blocked during notebook execution. Installation may happen before that boundary. Demonstrate an actual blocked network attempt inside a notebook; removing TypeSafe keys alone is not that evidence. Preserve explicit recipe-folder cwd, and apply the guard to replay and scripted/simulator modes alike.
Define and document output comparison closely enough to catch real stale results, as the issue requires. Renderer variation is a comparison-design constraint: use an appropriate reproducible rendering environment or justified normalization/semantic comparison. A blanket omission of image output would leave changed teaching charts undetected; include a stale-figure probe and report the comparison's actual limits, alongside text/table stale-output probes. This is a handoff for requirement coverage, not a prescription for a particular implementation.
Keep the existing exhaustive raw-mode/type/scope allowlist and base-README renderer proof from prior comments. Do not replace them with a head-README self-render check. Every fixture must validate, including scripted no-responses and comparison responses files, and malformed or drift artifacts must fail.
Two distinct independent Opus reviewers must assess the final implementation and real rejection demonstrations, including notebook execution failure, invalid fixture, stale output, blocked networking and prohibited paths/modes/types. Discover and require every new workflow check before merge; five old baseline successes will not prove the expanded gate.

No second #69 implementation owner is being assigned. Original supervisor remains sole dispatch and merge owner. The complete 60-recipe and per-recipe recorded-evidence requirements remain intact.

Coordination follow-up for #69: the two #99 round-4 receipts demonstrate that PNG bytes vary across Python/platform rendering stacks. That establishes a need for a justified comparison; it does not establish coverage for stale figures when every image/png value is omitted.

Please preserve the attached execution brief and #69's stale-output acceptance criterion: include a negative probe that changes a committed teaching chart while leaving its cell source, text/plain figure representation, execution counts, and surrounding text intact, and demonstrate that the new CI gate rejects it. Blanket removal of image/png values will pass that probe and must not be reported as complete stale-output coverage. Reproducible rendering or a documented meaningful semantic figure comparison can satisfy the requirement; document actual limits and have both independent final-head Opus seats assess the probe.

Kernel-process stderr warnings are separate from notebook error outputs: exit status is useful, but error/traceback outputs retained by raises-exception-tagged cells must also be rejected by the notebook CI acceptance gate. No second #69 writer is being started; the original supervisor retains ownership. Earlier complete handoff: #69 (comment)

Foundation: recipe template, scaffolder, load_helpers, notebook execu…

Fix round 1: scope on edits, honest figure limits, hardened and cheap…

Notebook CI: pin the absence of continue-on-error; standing scope ver…

Navigation Menu

Repository navigation

Foundation: CI executes every notebook offline and validates fixtures #69

Goal

Scope

Acceptance criteria

Dependencies

References

55 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Foundation: CI executes every notebook offline and validates fixtures #69

Description

Goal

Scope

Acceptance criteria

Dependencies

References

Activity

55 remaining items

CompleteDotTech commented on Oct 6, 2026

CompleteDotTech commented on Oct 6, 2026

CompleteDotTech commented on Oct 6, 2026

CompleteDotTech commented on Oct 7, 2026

CompleteDotTech commented on Oct 7, 2026

CompleteDotTech commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions