Choose search's vector by experiment and add .latent format version 2

README.md

@@ -13,8 +13,11 @@ uv run jepa-scope clip.mp4
 
 The first run downloads the V-JEPA 2 ViT-L weights from Hugging Face (about 1.3 GB). After that, a
 10-second clip embeds in about 3.5 seconds on an M3 Max. Results are cached under `~/.cache/jepa-scope`, keyed by the
-video's hash, the settings, the device and dtype, and the jepa-scope version, so opening the same
-clip again is instant.
+video's hash, the settings, the device and dtype, and the jepa-scope output version, so opening the
+same clip again takes a few seconds (mostly loading libraries) instead of re-embedding. A cached
+file is reused only when its format version, output version and representation (described below)
+match what the current settings would produce; otherwise it is re-embedded in place, with a message
+saying why.
 
 ## Why this exists
 
@@ -87,10 +90,26 @@ tensors. The metadata holds one JSON header under the key `jepa_scope`, which re
 token grid, window settings, preprocessing recipe and hash, crop rectangle and source video hash.
 `jepa_scope.latent_file.load` checks that header before handing the tensors over.
 
+The header's `representation` block says exactly how a window vector was made: the model, an
+identity of its weights, the preprocessing hash, the token layer, the normalisation before pooling,
+the pooling, the window settings and the output version, with a hash over them. The weights are
+identified by content, not by a version label: for a downloaded model, by the content hashes
+Hugging Face stores for its files, so an upstream edit to the model card usually changes nothing
+(at most one clip is re-embedded while the new snapshot is filled in, and on systems where Hugging
+Face copies files instead of linking them, the copies' times stand in for the hashes); for a model
+in a local folder, by its config and each weight file's size and modification time. The
+snapshot's commit is recorded for information as `run.model_revision`. Two files with the same
+representation hash hold vectors made the same way. The device, dtype and `max_seconds` are
+recorded too (`run.device`, `run.dtype`, `source.max_seconds`); vectors from float16 and float32
+runs differ slightly, so the cache, which is keyed by them, reuses a file only when they match as
+well. The current format is version 2; a version-1 file is refused with a message to re-embed the
+video with `jepa-scope embed`.
+
 | Tensor | Shape | Meaning |
 | --- | --- | --- |
 | `patches` | `[windows, T, H, W, D]` float16 | Encoder output for every window, reshaped to its grid. Left out with `--no-patches` |
 | `pooled` | `[windows, D]` float32 | Mean over tokens, for search and simple classifiers |
+| `timeline_pooled` | `[tubelets, D]` float32 | Mean over each timeline tubelet's patches, from the same tokens as `pooled` |
 | `window_start_time` | `[windows]` | Start of each window in seconds |
 | `timeline_time` | `[tubelets]` | Start of each tubelet on the flattened timeline |
 | `timeline_window` | `[tubelets]` | Which window each timeline tubelet was taken from |

docs/decisions.md

@@ -41,6 +41,7 @@ anything the code already shows.
 
 ## Process & tooling · `docs/decisions/process.md`
 
+- 2026-11-06 — Milestone plans get a second-model adversarial review, then an outside review, before approval; public result claims get an outside review first
 - 2026-11-05 — Cloud GPUs within the spending cap of an owner-approved plan need no further approval: log the price, run a watchdog, delete the pod
 - 2026-11-05 — Milestones follow the roadmap's weekly order; Week 2's scope landed in Week 1, so Weeks 3 and 4 come next
 - 2026-11-05 — Adopted the workflow kit at the full level, with its process files, the roadmap and this record published in the public repo

docs/decisions/process.md

@@ -5,6 +5,34 @@ Full rationale for this theme's decisions, newest first. The lean, auto-loaded i
 
 ---
 
+## 2026-11-06 — Milestone plans get an adversarial review by a second model and then an outside review before approval; measured results get an outside review before they become public claims
+
+**Status:** live, indexed
+
+**Context.** Week 1's outside review found claims stated more confidently than the evidence
+supported, after the work was done. Weeks 3 and 4's plan then went through a fresh adversarial
+review before any code existed, and it found two planned tests that could not fail and one built
+on a false assumption about how windows and the timeline relate.
+
+**Decision.** Reviews scale with how hard a mistake is to undo.
+- Every milestone plan: a fresh adversarial review by a different model from the one that wrote
+  it, a revision, an outside review (a model from another provider) of the revised plan, then the
+  owner's approval, all before code changes.
+- Measured results that will become public claims (README numbers, launch material): an outside
+  review of the evidence first. For results inside a milestone, only when they are borderline or
+  surprising; for launch material, always.
+- Design calls mid-task, a pass bar set mid-milestone, and a root cause after two failed fixes: a
+  second-model review, without asking the owner first.
+- Routine code changes: the existing fresh-context `reviewer` pass on every change, nothing more.
+
+**Rejected.**
+- *An outside review of every pull request.* Each round costs the owner time to hand over and
+  bring back, and routine changes are covered by the fresh-context reviewer.
+- *Outside reviews only after a milestone ends.* A plan's mistakes are cheapest to fix before
+  code exists.
+
+---
+
 ## 2026-11-05 — Milestones follow the roadmap's weekly order; Week 2's scope landed in Week 1, so Weeks 3 and 4 come next
 
 **Status:** live, indexed

docs/plans/2026-11-06-weeks3-4-folder-search-trajectory-log.md

@@ -0,0 +1,108 @@
+# Execution log: jepa-scope Weeks 3 and 4
+
+The plan is `2026-11-06-weeks3-4-folder-search-trajectory.md`. Entries are newest last.
+
+- **2026-11-06 — Plan approved: every owner question as recommended.**
+  - **Evidence:** the owner answered "Approve as recommended" after the adversarial review, the
+    outside review and a consistency check.
+  - **Impact:** T0 starts. The gates that remain are the chosen default after T0, the Week 3
+    checkpoint after T7, and merging each pull request. Three.js is approved (question 7).
+  - **Resolution:** Status set to In progress.
+  - **Permanent decision record needed:** Yes, at T10, as the plan lists.
+
+- **2026-11-06 — T0 deviation: the localisation composite is built at 8 fps, not 24.**
+  - **Evidence:** the first `val` run stopped while building the composite, before any table was
+    printed: `val/bowling/-WH-lxmGJVY_000005_000015.mp4`, the second bowling clip in sorted order
+    and one of the two distractor sources, is 15 fps, and `decode` refuses to sample it at 24.
+  - **Impact:** none on what is measured. The model samples every video at 8 fps, so a composite
+    decoded and written at 8 fps gives the same 30 s and 29 windows, with the same transition
+    windows (4, 9, 14, 19 and 24 s).
+  - **Resolution:** `COMPOSITE_FPS = 8` in `tests/retrieval_eval.py`; the sources and order are
+    unchanged. No result had been seen when this was changed.
+  - **Permanent decision record needed:** No.
+
+- **2026-11-06 — T0 result: the default is (a), the stored mean of all final tokens, with plain
+  cosine. It awaits the owner's approval at the gate after T0.**
+  - **Evidence (selection set, 50 `val` clips, chance 0.18, none skipped):**
+
+    | Vector | Clip top-1 | Window top-1 | Null mean / 95th | Single-frame control |
+    |---|---|---|---|---|
+    | (a) mean of final tokens, plain cosine | 42/50 (0.84) | 0.74 | 0.19 / 0.28 | 35/50 (0.70) |
+    | (b) the same, centred | 39/50 (0.78) | 0.76 | 0.19 / 0.30 | 38/50 (0.76) |
+    | (c) mean of re-normalised tokens | 42/50 (0.84) | 0.74 | 0.19 / 0.30 | 35/50 (0.70) |
+    | (d) mean of the predicted half | 42/50 (0.84) | 0.74 | 0.19 / 0.30 | 35/50 (0.70) |
+
+    The colour baselines retrieved 10/50 (middle frame) and 15/50 (whole clip). Every candidate
+    passed localisation 16/16, and the clip-average mutation passed 0/16 for each. The parity of
+    (a) recomputed from patches was 7.2e-8. All four candidates were eligible; (a), (c) and (d) tie
+    at 42 and (b) is three clips behind, so the rule chose (a), the simplest. The tie is not an
+    artefact: (c) is almost the same vector as (a) (window cosine about 0.9997, because the
+    encoder's final layer norm already standardises tokens), and (d) differs more (about 0.96)
+    and picks a different nearest clip for 4 of 50, with the same count correct. Raw clip cosines
+    for (a) span 0.42 to 0.94 (median 0.70), so plain cosine is not compressed into a narrow band.
+  - **Evidence (confirmation set, 50 `train` clips, (a) only, nothing re-chosen):** 44/50 (0.88),
+    window top-1 0.79, null 0.18 / 0.28; colour baselines 15/50 and 12/50; single-frame control
+    35/50 (0.70). (a) meets both bars (0.60, and 0.58 above the better baseline).
+  - **What it means:** search on these clips is far better than colour, and full windows beat a
+    single repeated frame by 14 to 18 points, but the single frame already retrieves 70%. On these
+    five classes, which differ by scene, search mostly reflects what is in the scene, with motion
+    adding a measurable amount. The README should say so when the numbers go there (T10). The
+    misses concentrate in archery, flying_kite and bowling, often across classes shot outdoors.
+  - **Run:** downloads 95 MB (`val`) and about 68 MB (`train`); embedding the `train` set and its
+    controls took 196 s; the `val` rerun took 24 s from cache. Logs and JSON are in
+    `~/Workspace/jepa-scope-lab/retrieval/` (`t0_val_run2.log`, `t0_train_run1.log`,
+    `retrieval-val.json`, `retrieval-train.json`). The result cache now holds 3.8 GB, to be
+    deleted once T1 lands.
+  - **Impact:** T1's descriptor records `pooling: mean of final tokens` and
+    `similarity: cosine`; `pooled` is unchanged, so `OUTPUT_VERSION` stays. Centring stays an
+    explicit `--centre` option. The result is exploratory, as the plan says.
+  - **Resolution:** owner approval of (a) requested.
+  - **Permanent decision record needed:** Yes, at T10 (search as plain cosine over window means).
+
+- **2026-11-06 — Gate after T0: the owner approved (a) as the default.**
+  - **Evidence:** the owner answered "Approve (a)".
+  - **Impact:** T1 records `token_norm: none` and `pooling: mean_all`; `OUTPUT_VERSION` stays 2.
+  - **Resolution:** T1 starts.
+  - **Permanent decision record needed:** Yes, at T10, as already noted.
+
+- **2026-11-06 — T1 implemented: format version 2, `timeline_pooled`, the representation block,
+  `save` as a current-format writer.**
+  - **Evidence:** the fast suite passes (86). Breaking the code three ways made the tests fail:
+    taking `timeline_pooled` from the previous window (1 failure), dropping it (14), and restoring
+    `save`'s old merge order (1). On the real model, the version-1 cache file for
+    `val/archery/-Qz25rXdMjE…` printed the re-embed message, was re-embedded over the same path
+    (202 cache files before and after), and the second run reused it, so `Encoder.describe` (no
+    weights) agrees with what embedding writes. Its representation hash is `6cdbf94c50680ea2`
+    (model revision `b3c1679…`, `mean_all`). The server serves the version-2 file.
+  - **Impact:** every other version-1 cache file, including T0's, is now re-embedded on use.
+  - **Resolution:** the T0 cache files (3.8 GB) are deleted once this change is merged.
+  - **Permanent decision record needed:** Yes, at T10 (format version 2 and the descriptor).
+
+- **2026-11-06 — T1 deviation after review: weights are identified by content, not by commit.**
+  - **Evidence:** the fresh review found that for a model in a local folder the block held only
+    the path, so retraining into the same folder reused old vectors; and that the snapshot commit
+    changes when only a model card is edited upstream, which would force needless re-embedding.
+  - **Impact:** the block's `model_revision` field is replaced by `weights`, an identity built from
+    `config.json`'s bytes and each weight file's content name (the blob a Hugging Face snapshot
+    links to, or size and modification time for an ordinary folder). The commit moves to
+    `run.model_revision`, for information. A local model's `model_id` is its absolute path. The
+    hash's canonical form is now written down (sorted keys, no whitespace). `describe` is skipped
+    when no cache file exists, and the README states the reuse rule precisely. New fast tests use
+    a fake Hugging Face cache (same identity across commits with the same blob, a new one for new
+    weights) and a retrained local folder.
+  - **Resolution:** on the real model the earlier version-2 file was refused (its block no longer
+    matched), re-embedded over the same path, and reused on the next run; weights identity
+    `550d387c4f4a1862`, representation hash `e8eb61d31255e782`. The fast suite passes (88).
+  - **Permanent decision record needed:** Yes, at T10, with the format change.
+
+- **2026-11-06 — T1 second review: two identity gaps closed.**
+  - **Evidence:** a symlinked weight file in an ordinary folder was identified by its target's
+    file name, so retraining a "latest" checkpoint kept the identity; and right after an upstream
+    commit the new snapshot can hold only the config, so `describe` hashed the config alone.
+  - **Resolution:** a link's name is trusted only when it points at a Hugging Face blob (a `blobs`
+    folder and a 40- or 64-character hex name); anything else is described by the resolved path,
+    size and modification time. A folder with no weight file has weights `unknown`, and the CLI's
+    cache check then loads the model to get the identity. Two fast tests cover these cases (90
+    pass). On the real model the identity is unchanged (`550d387c4f4a1862`, hash
+    `e8eb61d31255e782`), so nothing re-embeds.
+  - **Permanent decision record needed:** No beyond the T10 entry.