Cross-harness parity: goose vs Claude Code vs Codex #3

Activity

1. daveh-beep

   added

   setup-experienceField report from running the setup protocolField report from running the setup protocol

   on Jul 21, 2026
2. daveh-beep

   added a commit that references this issue on Jul 21, 2026

   Conformance harness: transcript checks + fixtures from real sessions

   ...

   dcdcc0a
3. daveh-beep commented on Jul 30, 2026

   

   daveh-beep

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   🤖 Parity run completed 2026-07-30 (maintainer's agent)

   Method — and how it differs from this issue's original ask

   This issue asks for the setup protocol run in three harnesses. What ran first is the cheaper, higher-signal half: the consult path — same completed writing system (the Larkspur fixture from #2, pinned @ baecf3d), same rewrite task verbatim, in three harnesses plus a control. Setup-protocol parity (interview behavior per harness) remains open; consult parity gates it, because if harnesses can't even read the same system consistently, interview drift is moot.

   • goose 1.43.0 (reads AGENTS.md natively)
   • Codex CLI 0.145.0 (reads AGENTS.md natively)
   • Claude Code 2.1.217 — run twice: unbridged (control Dry-run: Meridian setup end-to-end #2) and with a one-line CLAUDE.md bridge
   • Control: Claude Code in the same repo, no bridge → never saw the writing system

   Caveat: this conflates harness and model (each ran its default model). Convergence across both variables is the strong version of the finding; per-variable isolation wasn't done.

   Result: PASS on the core claim

   All three consulting runs converged on the approved calibration direction from one pinned writing system:

   	Subject line	House terms	Banned patterns	Invented facts
   Claude + bridge	"Your Impact Driver loan was due back June 2"	✓	none	0
   Codex	same subject, fresh body	✓	none	0
   goose	same subject, fresh body	✓	none	0
   Control (no references)	"…is due back at Larkspur"	✗ "borrow," "the shop" in copy	guilt-adjacent warmth	4+ (extension mechanics, waiting neighbor, "pause for a bit," open-hours framing)

   References, not model choice, dominated the output. The with/without contrast is the whole thesis in one table.

   Two failures → fixes on PR #15

   1. Claude Code doesn't read AGENTS.md. Unbridged, the writing system was silently invisible — worst failure mode, because the user believes it's on. One committed line of CLAUDE.md fixed it completely (the bridged run matched the calibration pair nearly verbatim and flagged the "06/02" ambiguity per the new Phase 4 rule). Template now ships the bridge; the pin docs now say put pins where the consuming repo's agents actually read.
   2. Fact strength drifted. 2 of 3 consulting runs softened "suspension of borrowing privileges" → "may need to pause." Not invented — weakened. Rule 9, Phase 4, and calibration rules now state: preserving a fact includes its strength.

   Also observed

   • Few-shot pull varies by harness: bridged Claude reproduced the approved pair nearly verbatim; codex/goose wrote fresh convergent copy. Both fine; teams should know.
   • The control put "the shop" (an everyday phrase from vocabulary.md) inside copy — live evidence for the deliverable fence, triggered exactly when the system wasn't consulted.

   Still open on this issue

   • Setup-protocol parity (the original ask): interview order, phase fidelity, frontmatter fidelity per harness. The Larkspur persona sheet from Dry-run: Meridian setup end-to-end #2 is reusable for it.
   • Per-variable isolation (same model across harnesses, or same harness across models).
   • Codex/goose bridging conventions beyond CLAUDE.md (e.g., harnesses that read neither file).

   Artifacts (outputs, clones, fixture provenance) preserved in the maintainer's local run archive.
4. daveh-beep commented on Jul 30, 2026

   

   daveh-beep

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   🤖 Follow-up: candidate fix rerun completed

   I reran the exact Larkspur task against the candidate contract in PR #15.

   Important honesty note: the first rerun was invalid — it cloned the upstream empty template instead of applying the candidate contract changes to the pinned Larkspur system. That result was discarded; it tested the wrong artifact. The corrected run used three fresh clones of Larkspur @ baecf3d, added the candidate CLAUDE.md bridge and fact-strength rule, and held the task fixed.

   Corrected result:

   • Claude Code read the system through the committed bridge and preserved the certain consequence as certain.
   • Codex changed its prior "may need to pause" to "we have to suspend your borrowing."
   • goose changed its prior "may need to pause" to "to avoid suspension of your borrowing privileges."
   • all three kept house terms and invented no facts.

   This closes the consult-path learning cycle: observed routing + modality failures → candidate contract changes → reproduction → corrected behavior. Setup-protocol parity (the original broader issue) stays open.