As vibe coding becomes standard practice, many organisations are running into serious token shortages. This repository manages hypotheses about token usage and verifies the actual savings through controlled experiments.
- Common task: implementing the RealWorld App backend — a fixed benchmark task so conditions can be compared. See
tasks/realworld-backend/for the spec. - Fixed model: every experiment runs on a single Claude Opus model (removing model differences as a confound).
- Measurement tools: existing tools — tokenhabit (
habit_scan.py), ccusage.scripts/holds only minimal wrappers for extracting experiment windows and aggregating comparisons. - Primary tool under test: Claude Code. See
ROADMAP.mdfor comparisons with other tools.
- Workflow strategy (S axis): token differences by the strategy used for the same task
- Ralph loop: set a goal, then let an autonomous loop run
- Plan-then-execute: plan → split into tasks → implement tasks in parallel
- Token habits (H axis): token differences before/after correcting the tokenhabit H1–H8 habit patterns
- Language (L axis): token differences by the language (Korean/English) of prompts and deliverables
The full hypothesis list and experiment status live in hypotheses/catalog.md.
The table and per-experiment summaries below are generated by
scripts/update_readme_results.pyfrom each experiment'sreport.md(translations for the English, Japanese and Chinese READMEs come fromscripts/readme_i18n.json). A pre-commit hook runs it whenever areport.mdis committed (manual run:python3 scripts/update_readme_results.py).
📊 Live dashboard: Ralph loop completion by model — latest experiment included: EXP-023 (2026-09-16)
| Experiment | Hypothesis | Verdict |
|---|---|---|
| EXP-001 Ralph loop vs Plan-then-execute | S-01: Plan-then-execute uses fewer tokens than a Ralph loop on the same task | Rejected (refuted) |
| EXP-002 Korean vs English pipeline token comparison | L-01: Running the whole pipeline in English cuts billable tokens meaningfully versus Korean | Pending |
| EXP-003 PTE + skill-style progressive disclosure | S-02: Structuring context per the official skill guidance (documents under 200 lines, load only what is needed via skills) cuts billable by 30%+ versus the EXP-001 PTE | Verified |
| EXP-004 Ralph loop + skill structure | S-03: Giving a single-session Ralph loop domain-contract skills reduces billable | Pending (effectively no effect) |
| EXP-005 Claude Code × Upstage Solar Pro 3 backend | M-01: Swapping the Claude Code backend to Solar Pro 3 lets it finish the same task (RealWorld backend) without intervention, and the total cost on completion is meaningfully lower than Opus. | Pending |
| EXP-006 Claude Code × Upstage Solar Open 2 backend | M-02: Swapping the Claude Code backend to Solar Open 2 lets it finish the same task (RealWorld backend) without intervention, and the total cost on completion is meaningfully lower than Opus. | Pending |
| EXP-007 Autopsy of the Solar Open 2 non-completion | M-03: The EXP-006 (Solar Open 2) non-completion is not a single convergence-speed bottleneck but an overlap of several failure factors (model behaviour defects · experiment environment contamination · metering distortion). | Verified |
| EXP-008 Solar Open 2 uncontaminated clean run — completion check | M-04: With contamination removed (isolated config), no disturbance and a 30-iteration cap, solar-open2 can finish the RealWorld backend (Hurl 154/154) in a Ralph loop without intervention (cost excluded; completion is the single verdict). | Verified |
| EXP-009 Opus 5 Ralph loop (EXP-002 en condition, n=3) | M-05: Opus 5 reproduces single-session completion in the EXP-002 en Ralph loop and shows efficiency equal to or better than the Opus 4.x baseline (en 6–7 min · 38–54 API calls). | Partially verified (n=3) |
| EXP-010 Opus 4.8 vs Opus 5 pure A/B (same day, n=3 each) | M-06: Under fully identical conditions, Opus 5's expanded-output profile (output · commits ↑) reproduces against Opus 4.8 and both models keep single-session completion. | Verified |
| EXP-011 Codex CLI × gpt-5.6-sol RealWorld backend completion check | M-07: In the Codex CLI (codex exec) harness, gpt-5.6-sol (effort medium) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 30-iteration cap in an isolated, undisturbed Ralph loop without intervention. |
Verified |
| EXP-012 Claude Code × gpt-5.6-sol backend (ccr) RealWorld backend completion check | M-08: Connecting the Claude Code backend to gpt-5.6-sol through ccr (reasoning effort medium) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 30-iteration cap in an isolated, undisturbed Ralph loop without intervention. | Verified |
| EXP-013 Claude Code × qwen3.8-max direct (ANTHROPIC_BASE_URL) RealWorld backend completion check | M-09: Connecting Claude Code directly to qwen3.8-max through DashScope's Anthropic-compatible endpoint (thinking default) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 30-iteration cap in an isolated, undisturbed Ralph loop without intervention. | Verified |
| EXP-014 Claude Code × kimi-k3 direct (ANTHROPIC_BASE_URL) RealWorld backend completion check | M-10: Connecting Claude Code directly to kimi-k3 through Moonshot's Anthropic-compatible endpoint (thinking default) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 30-iteration cap in an isolated Ralph loop without intervention. | Verified |
| EXP-015 Claude Code × solar-open2 direct (ANTHROPIC_BASE_URL) RealWorld backend completion check | M-11: Connecting Claude Code directly to solar-open2 through Upstage's Anthropic-compatible endpoint (thinking default) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 30-iteration cap in an isolated, undisturbed Ralph loop without intervention. | Verified |
| EXP-016 Replication of the three n=1 completion conditions (n=3 each) | M-12: The unattended completions of the three EXP-011/013/014 conditions (Codex CLI × gpt-5.6-sol, qwen3.8-max direct, kimi-k3 direct) reproduce: two additional runs per condition (n=3 total) all complete within the 30-iteration cap with gate and independent re-check agreeing. | Verified |
| EXP-017 qwen3.8-max · kimi-k3 Korean-condition Ralph loop completion check (n=3 each) | L-02: qwen3.8-max and kimi-k3 (direct, thinking default) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 30-iteration cap in an isolated, undisturbed Ralph loop without intervention even under the Korean canonical prompt (including the all-deliverables-in-Korean instruction). | Verified |
| EXP-018 solar-open2 direct replication — impossible after provider endpoint withdrawal | M-13: The EXP-015 solar-open2 direct unattended completion reproduces: two additional runs (n=3 total) both complete within 30 iterations with gate (measure v4) and re-check agreeing. | Pending |
| EXP-019 Native Opus 4.8 · Opus 5 · Codex×gpt-5.6-sol Korean-condition completion check (n=3 each) | L-03: Native Opus 4.8 and Opus 5 (Claude Code) and gpt-5.6-sol (Codex CLI) can finish the RealWorld backend (Hurl 13/13 · 154/154) within the iteration cap in an isolated, undisturbed Ralph loop without intervention even under the Korean canonical prompt (including the all-deliverables-in-Korean instruction). | Verified |
| EXP-020 Claude Code × solar-pro4 direct completion check (n=3) | M-14: Connecting Claude Code directly to solar-pro4 through Upstage's Anthropic-compatible endpoint (thinking default) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 30-iteration cap in an isolated, undisturbed Ralph loop without intervention (n=3, completion-rate verdict, cost excluded). | Verified |
| EXP-021 Codex CLI × gpt-6-astra Ralph loop completion check (n=3) | M-15: In the Codex CLI (codex exec) harness, gpt-6-astra (effort medium) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 30-iteration cap in an isolated, undisturbed Ralph loop without intervention (n=3, completion-rate verdict, cost excluded). |
Verified |
| EXP-023 Claude Code × Fable 5.1 native Ralph loop completion check (n=3) | M-17: In the native Claude Code harness, Fable 5.1 (claude-fable-5-1, thinking default) can finish the RealWorld backend (Hurl 13/13 · 154/154) within a 10-iteration cap in an EN-canonical Ralph loop without intervention (n=3, completion-rate verdict, cost excluded). |
Verified |
EXP-001 — Ralph loop vs Plan-then-execute (Rejected (refuted))
Plan-then-execute used about 8.7× more billable tokens → report
EXP-002 — Korean vs English pipeline token comparison (Pending)
The pre-registered rule (|KO mean − EN mean| > within-condition run-to-run spread) was not met. The language effect (29K mean difference) is buried under trajectory variance between runs (up to 138K) → report
EXP-003 — PTE + skill-style progressive disclosure (Verified)
39.3% reduction (2,839,815 → 1,723,575). Same workflow; only the context structure changed. → report
EXP-004 — Ralph loop + skill structure (Pending (effectively no effect))
A +3.5% mean difference (opposite to the hypothesis) is completely buried under within-condition variance (200K) → report
EXP-005 — Claude Code × Upstage Solar Pro 3 backend (Pending)
solar-1 did not finish (0 test runs, 0 commits, stopped early at iteration 6/15): the integration stack was validated, but in the headless autonomous loop the permission-wait and context-overflow failure modes repeated and it never reached a completion trajectory. → report
EXP-006 — Claude Code × Upstage Solar Open 2 backend (Pending)
0/2 completions, but only one run followed the full protocol (open2-1 self-terminated after 1 iteration with a false completion claim): open2-2 used all 15 iterations and reached only 3/13 files (94/154 requests) on independent verification, yet it established the autonomous TDD loop that Solar Pro 3 lacked and converged monotonically — the “behaviour layer” bottleneck moved from autonomy to convergence speed. → report
EXP-007 — Autopsy of the Solar Open 2 non-completion (Verified)
Three layers demonstrated: ① metering distortion (usage overstated 3.07× — actually 483 requests, 23.3M input, an estimated ~$3.8, lower than Opus at $6.41), ② environment contamination (superpowers hooks and the global CLAUDE.md injection ate at least 3 iterations), ③ model behaviour defects (declare-without-execute leaving 0 commits, 25 thinking-only truncations, 2 off-task hallucinations). → report
EXP-008 — Solar Open 2 uncontaminated clean run — completion check (Verified)
Completed at iteration 10/30: .ralph-done created → harness gate passed 13/13 files · 154/154 requests → two independent re-checks by the experimenter agree. About 2 h 53 min wall-clock, no intervention or interruption, 4 git commits (in Korean). Completion came inside EXP-006's cap (15), so the deciding variable was contamination removal and non-disturbance, not a higher cap. → report
EXP-009 — Opus 5 Ralph loop (EXP-002 en condition, n=3) (Partially verified (n=3))
Completion clause verified: all 3/3 runs completed in a single session at iteration 1 (9:03–12:22, gate 13/13 · 154/154 plus two independent re-checks each, 6–7 commits). Efficiency clause confirmed unmet: the time range (8.9–12.2 min) does not overlap 4.x (5.8–6.9 min) — but the cause is not serving speed; it is a behaviour-profile shift with consistently higher output (+62%). → report
EXP-010 — Opus 4.8 vs Opus 5 pure A/B (same day, n=3 each) (Verified)
6/6 completions (every run at iteration 1, gate 13/13 · 154/154). All pre-registered metrics met: non-overlapping output-token ranges (4.8: 29.6–37.1K vs 5: 41.1–47.9K, +37% mean) · non-overlapping git-commit ranges (1–2 vs 4–8), same direction as EXP-009 (5 > 4.8). The generational difference is a real profile, not a timing or metering artefact. → report
EXP-011 — Codex CLI × gpt-5.6-sol RealWorld backend completion check (Verified)
Completed at iteration 1 (gate 13/13 · 154/154 plus two matching independent re-checks, codex exec 5 min 46 s · 1 session · 3 commits, no intervention). → report
EXP-012 — Claude Code × gpt-5.6-sol backend (ccr) RealWorld backend completion check (Verified)
Completed at iteration 11/30 (gate 13/13 · 154/154 plus two matching independent re-checks, 58 min total · 11 commits). One harness-level intervention at iteration 2 for a stream stall (process killed to recover the iteration boundary; model output untouched) — see the protocol issues below. → report
EXP-013 — Claude Code × qwen3.8-max direct (ANTHROPIC_BASE_URL) RealWorld backend completion check (Verified)
Completed at iteration 1 (gate 13/13 · 154/154 plus two matching independent re-checks, 15 min 5 s · 4 commits · 0 interventions, zero direct-connection troubleshooting). → report
EXP-014 — Claude Code × kimi-k3 direct (ANTHROPIC_BASE_URL) RealWorld backend completion check (Verified)
Completed at iteration 1 (gate 13/13 · 154/154 plus two matching independent re-checks, 21 min 18 s · 4 commits · 0 interventions, zero direct-connection troubleshooting). → report
EXP-015 — Claude Code × solar-open2 direct (ANTHROPIC_BASE_URL) RealWorld backend completion check (Verified)
Completed at iteration 2 (two direct re-grades of the output both 13/13 · 154/154, 58 min 15 s · 2 commits. Footnote: one gate false rejection — the measure v3 port-detection flaw rejected a legitimate completion; the EXP-010 48-1 precedent was applied). → report
EXP-016 — Replication of the three n=1 completion conditions (n=3 each) (Verified)
All 6 additional runs completed (gate pass plus two independent re-checks each at 13/13 · 154/154, 0 interventions). Including the original runs, completion is 3/3 per condition, 9/9 combined. → report
EXP-017 — qwen3.8-max · kimi-k3 Korean-condition Ralph loop completion check (n=3 each) (Verified)
All 6/6 runs completed at iteration 1 (gate pass plus two independent re-checks each at 13/13 · 154/154, 0 interventions). Language compliance also holds: every run's commit messages and README are in Korean. → report
EXP-018 — solar-open2 direct replication — impossible after provider endpoint withdrawal (Pending)
External-to-model failure (pre-registered criterion): at the start of the experiment Upstage withdrew the Anthropic-compatible endpoint (/v1/messages) and the solar-open2 hosted API, so the runs themselves were impossible. The hypothesis is unverifiable, not rejected. → report
EXP-019 — Native Opus 4.8 · Opus 5 · Codex×gpt-5.6-sol Korean-condition completion check (n=3 each) (Verified)
All 9/9 runs completed at iteration 1 (gate pass plus two independent re-checks each at 13/13 · 154/154, 0 interventions). Language compliance holds across the board: all 9 runs have Korean commit messages and READMEs (including the English-first model gpt-5.6-sol). → report
EXP-020 — Claude Code × solar-pro4 direct completion check (n=3) (Verified)
3/3 completions (gate pass plus two independent re-checks each at 13/13 · 154/154, 0 model interventions). pro4-1's loop was extended by a gate false rejection (harness fault) — retroactive re-grading fixed its effective completion at iter 3. → report
EXP-021 — Codex CLI × gpt-6-astra Ralph loop completion check (n=3) (Verified)
All 3/3 runs completed at iteration 1 (gate pass plus two independent re-checks each at 13/13 · 154/154, 0 interventions, 7-minute sessions · 3–4 commits). → report
EXP-023 — Claude Code × Fable 5.1 native Ralph loop completion check (n=3) (Verified)
All 3/3 runs completed at iteration 1 (gate pass plus two independent re-checks each at 13/13 · 154/154, 0 interventions, 6–8-minute sessions · 2–4 commits). Secondary metrics moved down, with non-overlapping time and output ranges versus Opus 5 (EXP-009/010): 6.2–7.8 min vs 8.9–17.6 min, 28.7–35.8K vs 41.1–48.4K — Opus 5's expanded-output profile returns to the 4.8 level in Fable 5.1. → report
Conclusions running through the experiments (RealWorld backend, Opus fixed):
- The dominant variable in token cost is context (cache) reuse. A single-session Ralph loop (about 290K) reuses the context it built once, all the way to the end — see docs/context-reuse-mechanism.md for the technical background (prefix-based prompt caching, 0.1× cache reads, recomputed dollar conversion). The moment you split sessions, the fixed startup cost (about 20.6K per session) and the cost of rebuilding context accumulate, and the same task becomes 6–9× more expensive (EXP-001).
- Progressive disclosure (skills) is a prescription for multi-session work only. When a session needs only a subset of the full context (a PTE task session), it removes repeated reads and fix loops: −39.3% (EXP-003). A single session that needs everything preloads all skills and gains nothing (EXP-004).
- Agent trajectory variance is constant noise of ± hundreds of thousands of tokens. Runs under identical conditions diverge by up to 2× (EXP-002 EN 191K–329K, EXP-004 199K–400K). An effect of roughly 10% (e.g. language) cannot be resolved with n=2.
- Practical guidance: if the task fits in one session, run it in one session. If splitting is unavoidable, structure the context as skills to limit the loss. Document language (Korean/English) is a far smaller variable than either of those decisions.
- The dominant factor behind “the model cannot finish” was experiment-environment contamination (M axis, EXP-005–008). The Solar backend four-part story: EXP-005 (no completion, no autonomy) → EXP-006 (TDD established but 3/13, no completion) → EXP-007 (autopsy: a 3.07× usage overstatement illusion + superpowers hooks and global CLAUDE.md contamination eating 23% of iterations + model defects) → EXP-008 (a clean run with contamination removed finished 13/13 · 154/154 in just 10 iterations, with 4 commits). With the same model, the same PROMPT and the same env, isolation alone flipped the verdict, and because completion came inside EXP-006's cap, raising the cap contributed nothing. Three lessons: (a) isolating the experimenter's local environment (hooks, global settings) is a precondition for autonomous-loop experiments — otherwise “model capability” measurement becomes “contamination compliance” measurement. (b) Do not attribute a completion failure to the model without a log autopsy — it may be a translation-layer defect (the CCR multi-delta bug), a metering error (summing usage rows; message.id dedup is mandatory), environment contamination, or a false rejection by the grading gate itself (EXP-010 48-1 grading files not copied, EXP-015 iter 2 port-detection flaw — fixed in measure v4). (c) Intrinsic model defects (one permission-wait, 94% thinking, an unexecutable state at a turn boundary) can persist and still be absorbed by the Ralph loop's repetition. A cost-advantage verdict remains impossible while pricing is unpublished.
- The Ralph loop protocol ports across harnesses; the model decided completion and the harness decided the trajectory (EXP-011/012 pair). With the same PROMPT, gate and model (gpt-5.6-sol, effort medium) and only the harness swapped, both finished without intervention — Codex CLI at iteration 1 · 5 min 46 s · a single 436-line file · 3 commits; Claude Code (via ccr) at iteration 11 · 58 min · 11 modular files · 11 commits. Codex read “the single most important piece” as “until done”, Claude Code read it literally as one piece, and the Ralph loop absorbed the small iterations. The isolation principle (dedicated CODEX_HOME/CLAUDE_CONFIG_DIR) and the external per-iteration grading gate hold regardless of tool. Cross-tool token-efficiency comparison needs measurement standardisation first (rollout totals vs ccr tab vs ccusage). The Codex harness also ports across model generations — the first GPT-6 model, gpt-6-astra (Codex is the only consistent path since there is no Anthropic-compatible endpoint), connected by swapping the model ID alone and completed 3/3 at iter 1 (EXP-021, 7-minute sessions · 0 false rejections). Session time was +23–32% versus sol, so the official “1.9× faster” figure did not reproduce in this harness — the speed narrative is withheld.
- For third-party models, direct connection through an Anthropic-compatible endpoint is structurally superior to a translation layer (ccr), and the direct template is reusable across providers (EXP-013/014). Connecting qwen3.8-max (DashScope) and kimi-k3 (Moonshot) directly via
ANTHROPIC_BASE_URLeliminated every failure mode that recurred on the ccr stack (lost usage → separate tab, transformer-chain tuning, stream stalls) — both passed Phase 0 unadjusted, completed at iteration 1 (15 min 5 s / 21 min 18 s) and logged usage normally in session jsonl. EXP-014 ran the EXP-013 harness with only three env values swapped (endpoint/key/model ID) — the method is provider-independent. Direct connection also returns metering to the Claude standard path (jsonl + message.id dedup), partly resolving the standardisation prerequisite from item 6, though cache accounting differs by provider (Moonshot reports cache_create as 0). Comparison against ccr is confounded by different models — when a provider offers an Anthropic-compatible endpoint, make direct connection the default, but quantitative stack comparison needs same-model experiments. Reproducibility is confirmed at n=3 (EXP-016): Codex×gpt-5.6-sol, qwen direct and kimi direct each 3/3 (9/9 combined, 8/9 at iter 1) — but the non-completion metrics (time, commits, output) vary up to 3× under the same condition (kimi 21 min → 7 min), so the S-axis conclusion “trajectory variance is constant noise” holds on the M axis too. Only completion rate is a stable metric. The Korean condition does not hurt completion either (EXP-017/019): under the Korean canon (all deliverables instructed in Korean), the five conditions qwen · kimi · Opus 4.8 · Opus 5 · Codex×gpt-5.6-sol each went 3/3, 15/15 cumulative at iter 1, with all commits and READMEs in Korean and zero language drift — even the English-first model (gpt-5.6-sol) complied, so Korean compliance is a matter of instruction-following, not model family. Time, commits and output for all three EXP-019 conditions overlap the EN ranges — qwen's +48% time under ko (EXP-017) is the exception and is kept as a directional note only (L-01 principle). Model profiles (4.8 single commit vs 5 fine-grained) persist after the language flip. But direct-connection reproducibility depends on the provider (EXP-018): one day after the EXP-015 completion, Upstage withdrew the Anthropic-compatible endpoint and the solar-open2 hosted API without notice (end of the application-only beta), closing the replication window — for third-party benchmarks, the stated experiment date defines the limit of any reproducibility claim, and a provider smoke test before reusing a harness is mandatory. Two weeks later the endpoint returned as solar-pro4, and direct completion was confirmed 3/3 (EXP-020) — completion ability on par with the top direct-connection models, but the longest profile at 119–258 effective minutes (8–17× qwen), caused by the product of no caching (cache 0 on every call, ~80K tokens re-prefilled per call) · 37-second round-trips · 88% thinking. Caution: the solar-open2 and solar-pro4 endpoints were preview (application-only beta) APIs, not commercial GA, permanently subject to infrastructure constraints (no prompt caching, long round-trips, unannounced withdrawal), so the Solar family's time and usage profiles confound model ability with preview infrastructure — do not read them as production performance. A third gate false rejection (a global npm contamination shadowing the hurl binary) added the lesson to pin the grading binary to an absolute path. - The expanded-output profile is a model trait, not a generational one (EXP-023). Fable 5.1, the top-tier model of the Claude 5 generation, completed 3/3 at iter 1 on the same harness and EN canon with output 28.7–35.8K and 6.2–7.8 min — overlapping the Opus 4.8 range (29.6–37.1K · 7.8–8.7 min) and not the Opus 5 range (41.1–48.4K · 8.9–17.6 min). The output and commit expansion that EXP-010 established as a “generational trait” of Opus 5 does not reproduce in the higher model of the same generation, so it is reinterpreted as a profile specific to Opus 5 (n=3 · 7-week gap, directional note).
- Copy
templates/experiment-readme.mdtoexperiments/NNN-name/README.mdand write the design (hypothesis, conditions, measurement method, success criteria) - Run sessions per condition and store session logs and measurements under
runs/<condition>/ - Write the token-difference analysis and conclusion in
report.md— keep the header lines- 가설: [code](...) — ...and- **판정: ...** — <one-line summary>(the README generator parses these two lines) - Update the status in
hypotheses/catalog.md(untested → in progress → verified/rejected) - Add the new experiment's English, Japanese and Chinese translations (title · hypothesis · verdict · summary) to
scripts/readme_i18n.json— if missing, the Korean original is inserted into that language's README and the script warns - When
report.mdis committed, the pre-commit hook regenerates the results section of all four READMEs (ko/en/ja/zh-CN). On a fresh clone rungit config core.hooksPath hooksonce (manual refresh:python3 scripts/update_readme_results.py)
├── ideation.md # original ideation (kept as-is)
├── ROADMAP.md # phased roadmap
├── hypotheses/catalog.md # hypothesis catalog + experiment status table
├── experiments/ # one directory per experiment (NNN-name/)
│ └── 001-ralph-vs-plan-then-execute/
├── tasks/ # shared task specs (reused across conditions)
│ └── realworld-backend/
├── templates/ # experiment design / report templates
├── scripts/ # measurement / aggregation wrapper scripts
└── docs/specs/ # design documents