Skip to content

Iteration 3 stopped at validation (spec 007 T010–T013) - #53

Merged
IanMayo merged 1 commit into
mainfrom
iter3/validation-writeup
Oct 22, 2026
Merged

IanMayo merged 1 commit into
mainfrom
iter3/validation-writeup

Conversation

@IanMayo

@IanMayo IanMayo commented Oct 22, 2026

Copy link
Copy Markdown
Member

Summary

Iteration 3 (second zig, both turns across the line of sight) stops at validation under its pre-registered rule (spec 007 FR-006). Seed 204's test split is not scored and stays reserved.

σ = 1°, seed-204 validation Range, zig+300 s Course, zig+300 s Speed, zig+600 s
LSTM (iteration 3) 8.8% 6.19° 1.10 kt [1.06, 1.16]
BiLSTM (iteration 3, reported) 7.2% 4.36° 1.02 kt [0.98, 1.05]
2c models (out of distribution) 11.4% / 17.7% 6.53° / 7.72° 2.01 / 1.93 kt
classical 3.4% 1.43° 0.34 kt
naive 40.2% 89.7° 4.23 kt
  • Operative condition fails: the LSTM's speed interval needed a lower bound under 1 kt.
  • Sanity condition holds: iteration-3 LSTM / 2c LSTM, paired speed ratio 0.55 [0.52, 0.59]. This is a weak reference, as recorded before training.
  • Gap to classical widened: LSTM/classical speed ratio 3.29 [3.11, 3.54], against 2.4× in 2c. The forecast (1.1–1.3 kt) held.
  • BiLSTM: it finished by early stopping after seven chained jobs over three dispatches (35 runner-hours).

Contents

  • experiments/20261021-i3a-val/: CI run 39604573996 output (models, metrics, paired.json) and validation.md (the rule, tables, training, reading).
  • Site:
    • The iteration-3 results page and sidebar entry.
    • A blog post.
    • The viewer catalogue gains the iteration-3 pair. prebuild.mjs accepts a validation experiment without verdict.json.
  • Docs: AGENTS.md gate status, plus a note that an incomplete model needs a re-dispatch. Spec 007 status and tasks: T012 not run, T013 done.

No agent panel: this is the mechanical application of a rule committed before training, not a gate decision. The next Part B lever is the owner's choice.

Constitution check

  • Data types defined only in schema/tma.yaml (I): no schema change.
  • Deterministic; golden files unchanged (II).
  • Pure sim/features; one CLI per stage; Keras-equivalent layers (III): no code change outside the site prebuild.
  • Every model result sits beside naive and classical on the same rows (IV).
  • Respects the current gate (V): the test split was not scored, per FR-006.

Site pages

  • Glossary entry: no new term.
  • Data dictionary: regenerated automatically.
  • Algorithm note: no new algorithm.
  • Blog post: site/blog/2026-10-22-iteration-3.md.
  • Not applicable.

Checks

  • pnpm lint && pnpm typecheck && pnpm test && pnpm schema:check && pnpm site:build: all pass (325 tests).

Generated by Claude Code

- experiments/20261021-i3a-val: CI run 39604573996 (train/i3a, e3a8522),
  seed-204 validation with the 2c models beside; validation.md applies
  FR-006: LSTM speed 1.10 kt [1.06, 1.16] fails the operative condition,
  so the test split is not scored
- results page and blog post; AGENTS.md status; spec 007 status and tasks
- viewer: iteration-3 pair added (validation experiment, no verdict.json);
  seed-204 validation source no longer says untrained

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv4979qzsScV5ohT5RNUoG
@github-actions

Copy link
Copy Markdown
Contributor

Site preview: https://debrief.github.io/Gitma/pr/53/ (updates on every push; removed when the PR closes).

@IanMayo
IanMayo merged commit a73c8a0 into main Oct 22, 2026
2 checks passed
@IanMayo
IanMayo deleted the iter3/validation-writeup branch October 22, 2026 06:49
github-actions Bot added a commit that referenced this pull request Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants