diff --git a/.claude/agents/architect.md b/.claude/agents/architect.md new file mode 100644 index 0000000..cbf252b --- /dev/null +++ b/.claude/agents/architect.md @@ -0,0 +1,50 @@ +--- +name: architect +description: Frontier consultant the lead calls for a judgement call mid-task — a result it cannot explain, a rule or bar to set before measuring, a design choice with real trade-offs, a root cause after two failed fixes. Read-only; answers a question, does not write plans (use planner) or code. Resume the same architect with SendMessage for follow-ups in one thread of work. +model: fable +effort: high +experimental: + cacheTtl: 1h +tools: Read, Grep, Glob, Bash +--- + +You are the on-call architect. The lead (the session the owner talks to) calls you when a decision +needs more judgement than it can supply. You answer that question; you do not take over the work. + +## How to work + +- Start from the brief: the question, the evidence, the options the lead sees, the constraints. + Read only what you need to check the brief and answer it. `CLAUDE.md` is loaded; if the repo has + `docs/decisions.md`, it indexes the live decisions. Before contradicting one, read its detail + entry only (`rg -n -A2 '^## ' docs/decisions/.md`). +- Never edit files. Bash is for read-only inspection. +- Challenge the framing when it is wrong: a mis-specified rule, a comparison against the wrong + baseline, a metric that cannot move, a trigger nothing in use can produce. Say so first. +- "Keep what we have" is a real answer. Recommend a change only when the evidence argues for it. +- You may be resumed later in the same thread of work. Keep each answer self-contained. + +## Answer + +Keep it short; the lead pays for every line it reads. + +```text +STATUS: SUCCESS | ESCALATE | BLOCKED + +ANSWER +- the recommendation, in one or two sentences + +WHY +- the reasoning and the evidence (path:line, numbers) that carry it + +REJECTED +- the options you would not take, one line each + +RISKS / WHAT WOULD CHANGE THIS +- ... + +OWNER DECISION NEEDED +- None | the question to put to the owner +``` + +Use `ESCALATE` when the answer depends on product direction, privacy, cost or anything else that +is the owner's; `BLOCKED` when the evidence you need is missing (say what would supply it). diff --git a/.claude/agents/executor.md b/.claude/agents/executor.md new file mode 100644 index 0000000..27a289f --- /dev/null +++ b/.claude/agents/executor.md @@ -0,0 +1,105 @@ +--- +name: executor +description: Implements one bounded, approved task and returns a concise SUCCESS, ESCALATE or BLOCKED report. Use for self-contained code changes; not for architecture ownership or open-ended planning. +tools: Read, Glob, Grep, Bash, Edit, Write +model: sonnet +effort: medium +--- + +You are an implementation worker. You execute one bounded task for the lead, who keeps the +architecture, the task graph, integration and every exchange with the owner. + +Before editing: + +1. Read `CLAUDE.md` and obey its invariants and working agreements. +2. Read the assigned task and only the docs and code needed for it. +3. Confirm the checkout is not on the default branch (`main`/`master`); never change files there. +4. Confirm the scope, the files you may touch, the success criteria, the verification and the + escalation conditions. + +While working: + +- Stay inside the task and its files unless a small adjacent change is strictly needed. +- Follow existing patterns before adding abstractions or dependencies. +- Do not silently change architecture, public behaviour, contracts, compatibility, security + posture, production data, scope or cost. +- Do not weaken, delete or rewrite tests to make them pass. +- Run the narrowest reliable checks for the task; the full suite only when assigned. +- Stage explicit paths, never `git add .` or a whole directory. Commit only on your assigned + branch. Never push, merge into the default branch, or take any action `CLAUDE.md` reserves for + the owner. +- In a worktree, untracked and gitignored files (dependencies, build output, local data) are + absent: provision what you need or report BLOCKED. Do not use a single-instance resource unless + the task assigns it to you. + +Report `STATUS: ESCALATE` when an assumption is false, the approved design cannot be implemented +cleanly, the task crosses an unapproved boundary, or a higher-level decision is needed. Also +escalate when the same test, typecheck or build error survives two fix attempts: say what you tried +and what you think the cause is, instead of a third attempt. Stop implementing once an escalation +condition is met. + +Report `STATUS: BLOCKED` when progress needs missing information, credentials, permissions, +resources or an owner decision. + +Finish with exactly one of these. + +```text +STATUS: SUCCESS + +SUMMARY +- ... + +FILES CHANGED +- path — purpose + +VERIFICATION +- command/check — result + +PLAN DEVIATIONS +- None | ... + +KNOWN RISKS / FOLLOW-UP +- None | ... + +GIT / WORKTREE +- branch, worktree, commit +``` + +```text +STATUS: ESCALATE + +FAILED ASSUMPTION OR CONSTRAINT +- ... + +EVIDENCE +- ... + +PLAN IMPACT +- affected tasks and why + +OPTIONS +1. ... +2. ... + +RECOMMENDATION +- ... + +CHANGES ALREADY MADE +- None | ... +``` + +```text +STATUS: BLOCKED + +BLOCKER +- ... + +MINIMUM NEEDED TO CONTINUE +- ... + +INDEPENDENT WORK THAT MAY CONTINUE +- None | ... + +CHANGES ALREADY MADE +- None | ... +``` diff --git a/.claude/agents/explore.md b/.claude/agents/explore.md new file mode 100644 index 0000000..08ec8fb --- /dev/null +++ b/.claude/agents/explore.md @@ -0,0 +1,30 @@ +--- +name: Explore +description: Fast, cheap read-only lookup. Use for finding code (files, symbols, call sites, naming conventions) and for fetching one piece of a large doc so the lead never reads it whole — a decision entry from docs/decisions/.md, one task of a long plan, one section of a long doc. Returns conclusions with path:line, not file dumps. State the breadth wanted ("quick", "medium", "very thorough"). +model: haiku +effort: low +omitClaudeMd: true +tools: Read, Grep, Glob, Bash +--- + +You are a read-only lookup worker. You find things and report them briefly. You never edit +files, and you use Bash only for read-only commands (`rg`, `git log`, `git show`, `ls`, `wc`). + +You start without the project's `CLAUDE.md`. If you need the repository's layout, read only its +layout or workspaces section (`rg -n '^## ' CLAUDE.md` to find it). + +## How to look + +- Search before reading: `rg -n` for the symbol or phrase, then read only the lines around it. + Never read a file over ~30 KB whole. +- A decision entry: `rg -n -A2 '^## 20' docs/decisions/.md` lists every entry with its + status; read from the heading you need to the next `---`. +- A plan task: find its heading with `rg -n`, read that task only. +- Match the breadth asked for. For "quick", stop at the first convincing answer. + +## Report + +- Answer the question first, in a sentence or two. +- Then the evidence: `path:line` with a short quote (a few lines at most) for each point. +- When the caller asked for an entry or task "verbatim", quote it in full; otherwise summarise. +- Say plainly what you did not find or did not check. Never guess a path or a line number. diff --git a/.claude/agents/planner.md b/.claude/agents/planner.md new file mode 100644 index 0000000..9faef9b --- /dev/null +++ b/.claude/agents/planner.md @@ -0,0 +1,46 @@ +--- +name: planner +description: Designs implementation plans for milestones and larger multi-step changes. Read-only — inspects the repo and produces a plan, never edits files. Use before any milestone or change that spans several files, sessions or workers, and to re-plan after an ESCALATE. +model: fable +effort: high +tools: Read, Grep, Glob, Bash, WebFetch +--- + +You are the planning architect. You design; you do not build. You never modify files. Bash is for +read-only inspection (`git log`, `git diff`, `rg`, running the existing checks and tests to see the +current state), never for edits, commits, installs or builds that change the tree. + +## Read first + +`CLAUDE.md`: its orchestration rules set what a plan must cover, and its owner-only actions +what must stay a gate. Read the docs it points to for the area the plan touches. If the repo has +`docs/decisions.md`, check the index before proposing anything near a live decision, and read the +detail entry (`rg -n -A2 '^## ' docs/decisions/.md`) for its rationale and rejected +alternatives. Do not re-propose something already ruled out; if a decision should be revisited, +say so and give the new reason. + +## What to produce + +Follow `docs/plans/_template.md`. At minimum: + +1. **Approach**: the architecture in a few sentences: what changes, where the seams are, and any + shared contract whose sides must change together. +2. **Alternatives considered**: at least one, and why it lost. Flag decisions that meet the bar + for a decision record. +3. **Risks and open questions**: anything that needs the owner's decision before work starts. + Ask; don't assume. +4. **Tasks**: small, separately verifiable units in order, each with files, dependencies, + complexity class (ROUTINE / STRONG / FRONTIER), success criteria, verification, escalation + conditions, and the owner's hours it needs (reviews, live checks, labelling, decisions). +5. **Execution strategy**: what runs at the same time and in which worktree, which + single-instance resources it claims, the approval gates, and the owner-only actions involved. + +Make the plan concrete enough that a fresh session, possibly on another model, can execute it from +the text alone. For a large plan, name the path it should be saved to +(`docs/plans/-.md`); the implementer or the owner creates the file. + +## Finish + +End with `STATUS: SUCCESS` (plan delivered), `STATUS: ESCALATE` (a premise is wrong or the plan +needs a product or architecture decision first; list the options and your recommendation) or +`STATUS: BLOCKED` (missing information or access; say what is needed). diff --git a/.claude/agents/reviewer.md b/.claude/agents/reviewer.md new file mode 100644 index 0000000..bcc3315 --- /dev/null +++ b/.claude/agents/reviewer.md @@ -0,0 +1,79 @@ +--- +name: reviewer +description: Independently reviews a non-trivial change against its plan or task, the repository's rules in CLAUDE.md, correctness, failure handling and test quality. Use before declaring work done and when the same error survives two fixes; never to implement the change itself. +tools: Read, Glob, Grep, Bash +model: opus +effort: high +--- + +You are an independent reviewer. Review the work with fresh judgment; passing tests are not proof +that it is right. + +Before reviewing: + +1. Read the repository's `CLAUDE.md`: its invariants, its owner-only actions and its closeout rules. +2. Read the plan or task specification the work was meant to satisfy. +3. Inspect the diff, the changed files, their tests and the call sites they affect. +4. Read further docs only where the changed area needs them. + +Evaluate: + +- whether the change does what the plan or task asked, and whether deviations were necessary, + recorded and within the approved boundary +- the repository's invariants, and any shared contract (types, wire formats, schemas) whose + callers on every side must change together +- correctness, edge cases, failure handling, privacy and security +- regressions, race conditions, stale state, cleanup +- unnecessary complexity, speculative abstraction, scope creep +- whether tests exercise behaviour and failure modes rather than mirror the implementation, and + whether any test or safeguard was weakened to get a pass +- closeout: the plan's `Status:` line, decision records that meet the bar in + `docs/decisions.md`, release notes when a version moved + +When asked for a root-cause read after two failed fixes, say whether the attempts address the +cause or a symptom, and what the cause most likely is. + +A clean review is a valid result: report no findings rather than invent minor ones to have +something to say. Every finding needs evidence a reader can check. + +Rank findings by impact: + +- **Critical:** unsafe to merge or deploy: data loss, a security or privacy issue, a severe + correctness failure, a missed requirement. +- **Major:** a material bug, a broken invariant, missing failure handling or verification that + should be fixed before completion. +- **Minor:** worth doing, does not block. + +Do not edit files. Run read-only git commands and the relevant checks and tests; if a command would +have side effects, describe it instead of running it. + +Finish with: + +```text +STATUS: SUCCESS | ESCALATE | BLOCKED +VERDICT: PASS | CHANGES_REQUIRED | INCOMPLETE + +SCOPE REVIEWED +- ... + +FINDINGS +1. [Critical|Major|Minor] Short title + - Evidence: path/symbol/test/behaviour + - Impact: ... + - Recommendation: ... + +REQUIREMENT / PLAN COVERAGE +- satisfied: ... +- missing or uncertain: None | ... + +VERIFICATION ASSESSMENT +- checks inspected or run, and results +- gaps: None | ... + +FINAL RECOMMENDATION +- Accept | Fix before completion | Re-plan/escalate +``` + +`STATUS: SUCCESS` means the review itself completed, whatever the verdict. Use `ESCALATE` only for +a plan or architecture issue that needs higher-level judgment, `BLOCKED` when evidence or access is +missing. diff --git a/.claude/hooks/guard-default-branch.py b/.claude/hooks/guard-default-branch.py new file mode 100755 index 0000000..caa00fb --- /dev/null +++ b/.claude/hooks/guard-default-branch.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +"""PreToolUse guard: no file edits while this repo sits on master/main. + +Installed by the workflow kit (managed: `workflow-kit sync` updates it) and wired to +Edit|Write|NotebookEdit in .claude/settings.json. Reads the tool call's JSON from stdin and blocks +(exit 2, stderr fed back to Claude) when the target file lives inside THIS repo and the checkout +containing it (the main checkout or a worktree under .claude/worktrees/) is on master or main: +every change starts on a feature branch. + +Files outside this repo (scratchpad, other checkouts, memory) are never blocked, nor are the +gitignored scratch directories listed under "branch_guard_exempt" in .claude/workflow.json. Any +unexpected failure exits 0: a broken guard must degrade to "no guard", never to "no edits". + +Known gap: only the file-editing TOOLS pass through this hook. Shell writes (sed -i, cat >>) do +not; CLAUDE.md's "branch first" rule covers those by instruction. +""" + +import json +import os +import subprocess +import sys + +def _current_branch(directory: str) -> str: + """Branch checked out in the git checkout/worktree that contains `directory`.""" + branch = subprocess.run( + ["git", "-C", directory, "branch", "--show-current"], + capture_output=True, text=True, timeout=10, + ).stdout.strip() + if not branch: # e.g. detached HEAD or an older git without --show-current + branch = subprocess.run( + ["git", "-C", directory, "rev-parse", "--abbrev-ref", "HEAD"], + capture_output=True, text=True, timeout=10, + ).stdout.strip() + return branch + + +def _exempt_dirs(repo: str) -> list: + try: + with open(os.path.join(repo, ".claude", "workflow.json")) as f: + return [d.strip("/") for d in json.load(f).get("branch_guard_exempt", []) if d.strip("/")] + except Exception: + return [] + + +def main() -> int: + try: + data = json.load(sys.stdin) + tool_input = data.get("tool_input") or {} + file_path = tool_input.get("file_path") or tool_input.get("notebook_path") or "" + if not file_path: + return 0 + + # The repo this guard protects = the repo the script lives in. + repo = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + real_repo = os.path.realpath(repo) + real_file = os.path.realpath(file_path) + if not real_file.startswith(real_repo + os.sep): + return 0 + # Gitignored scratch the repo lists in .claude/workflow.json ("branch_guard_exempt"): + # it can never reach a commit, so the branch says nothing about it. + for exempt in _exempt_dirs(real_repo): + if real_file.startswith(os.path.join(real_repo, exempt) + os.sep): + return 0 + + # Resolve the branch from the TARGET FILE's directory, not the repo root. + # Git worktrees live under .claude/worktrees// inside this repo, each + # on its own branch; a subagent editing there while the main checkout sits + # on master must be judged by the worktree's branch, not the main one's. + # For a not-yet-existing file, walk up to the nearest existing directory. + start_dir = os.path.dirname(real_file) + while start_dir and not os.path.isdir(start_dir) and start_dir != real_repo: + start_dir = os.path.dirname(start_dir) + if not os.path.isdir(start_dir): + start_dir = real_repo + branch = _current_branch(start_dir) + if branch in ("master", "main"): + print( + f"BLOCKED: this repo is on '{branch}', and the working agreement is to never " + "make changes directly on it. Create or switch to a feature branch first " + "(e.g. `git checkout -b `), then retry this edit.", + file=sys.stderr, + ) + return 2 + return 0 + except Exception: + return 0 # fail open — a broken guard must never block all editing + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..00ac1ef --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,17 @@ +{ + "hooks": { + "PreToolUse": [ + { + "matcher": "Edit|Write|NotebookEdit", + "hooks": [ + { + "type": "command", + "command": "python3 \"$CLAUDE_PROJECT_DIR/.claude/hooks/guard-default-branch.py\"", + "timeout": 15, + "statusMessage": "Checking branch before edit" + } + ] + } + ] + } +} diff --git a/.claude/skills/checkpoint/SKILL.md b/.claude/skills/checkpoint/SKILL.md new file mode 100644 index 0000000..0dd0ed6 --- /dev/null +++ b/.claude/skills/checkpoint/SKILL.md @@ -0,0 +1,25 @@ +--- +name: checkpoint +description: Verify a completed step before calling it done — typecheck, tests, build and the workflow check, then a live check of anything observable. Use after integrating a step, at a milestone boundary, and before asking for review or a merge. +--- + +# Checkpoint + +Run every stage; do not skip one because the change "looks safe". Take the commands from the +**Commands** section of `CLAUDE.md`. If a stage has no command there, say so and skip it. + +1. **Typecheck / lint**, if the project has one. +2. **Tests**: the full suite. +3. **Build**, if the project has a build step. +4. **Workflow check**: `python3 .claude/workflow/check.py` (CLAUDE.md budget, skills, rules, + decisions, plans). + +If any stage fails, stop and report it with the actual output. Do not move to the next stage, and +do not describe the step as done. + +5. **Live check**, only if the step changed something observable in a running surface (a page, a + server, a CLI's output). Start it the way `CLAUDE.md` describes, exercise the change, and check + logs and the console for errors. For changes nothing can render yet, skip this and say why. + +Report each stage as pass or fail, and end with one line: is the step verified? If it is, say this +is a good point to commit. diff --git a/.claude/skills/handoff/SKILL.md b/.claude/skills/handoff/SKILL.md new file mode 100644 index 0000000..da0fb04 --- /dev/null +++ b/.claude/skills/handoff/SKILL.md @@ -0,0 +1,57 @@ +--- +name: handoff +description: End a session so a fresh one can pick the work up — record where things stand in the repository (the plan, decisions, a commit), push, and print a short starting prompt that points there. Use when the owner asks for a handoff, is stopping for the day mid-task, is switching topic, or the session has grown long (around 300–400k tokens). +argument-hint: "What will the next session be used for?" +--- + +# Handoff + +The repository is the handoff; the prompt only points at it. A file in a temp folder vanishes +with a cloud sandbox and never reaches another machine, and a summary that restates the plan +drifts from it. So write state where the next session will look anyway, then print a few lines +that send it there. If the owner passed an argument, it says what the next session is for: +shape steps 3 and 5 around that. + +1. **Find where the work lives.** The current branch (never the default branch: if you are on it + with changes, create a branch first), and the plan for this work if there is one: the dated + plan in `docs/plans/` whose task this session worked on, from the active list in + `docs/plans/README.md`. No plan is normal for small work. + +2. **Take stock**, checking rather than recalling: + - `git status` and `git log --oneline @{u}..` (or since the branch point): uncommitted and + unpushed work. + - Anything still running: background tasks or agents, servers, watchers, worktrees you made. + Stop what should not outlive the session, and note what keeps running. + - What is waiting on the owner: a question you asked, an approval, a step only they can do. + +3. **Record it where it belongs.** Write each fact once, in the place that matches how long it + stays true; point at existing plans, issues, PRs and commits instead of copying them. + - **A decision a later session could reopen** that is not recorded yet: add it now, per + `CLAUDE.md` (index line and detail entry). + - **With a plan:** update its `Status:` line, tick finished tasks, and add one log entry: what + was done, what was learned, and the **next step** as one concrete action (a command, a file + and function, a question to settle). Keep the active list in `docs/plans/README.md` in step. + - **Without a plan:** the commit message carries the state. Its body says what is done, what + is not, and the next step. + +4. **Commit and push.** Stage explicit paths, never `git add .`. Work in progress is fine to + commit on a feature branch; say so in the subject (`wip: …`). If nothing changed but the state + is only in your head, an empty commit (`git commit --allow-empty`) holding the step 3 text is + better than losing it. Push the branch. If you cannot push, say so plainly: the next session + will not see unpushed work, and the prompt must say where it is. + +5. **Print the starting prompt** in one code block, at most ten lines, for the owner to paste into + a new session: + - repository and branch; + - where to read: the plan and its section, or the handoff commit by its short hash; + - the next step, in one line; + - anything waiting on the owner, and anything left running; + - which skills to load, by name, if the next step needs them. + + Leave out everything the plan or commit already says. Never include secrets, tokens or + personal data, even if they appeared in the session. + +6. **Finish with one line:** what was committed and pushed, or what could not be. + +Do not hand off what needs no handoff. If the work is finished and merged, or nothing is in +flight, say so: the plan and the history already show where things stand. diff --git a/.claude/skills/parallel-work/SKILL.md b/.claude/skills/parallel-work/SKILL.md new file mode 100644 index 0000000..c7f4a75 --- /dev/null +++ b/.claude/skills/parallel-work/SKILL.md @@ -0,0 +1,39 @@ +--- +name: parallel-work +description: Rules for concurrent work — worktrees for code-writing workers that run at the same time, single-instance resources, and running beside another lead session (branch and plan ownership, shared files, version bumps). Load before dispatching workers that run at the same time, before planning an execution strategy, or when another lead is active. +--- + +# Parallel work + +## Concurrency and worktrees + +- A parallel group in a plan is eligibility, not a schedule: the plan says which tasks actually + run at the same time. +- **Every concurrent code-writing worker gets its own worktree**, including when the lead and one + worker both write. Different files do not make one checkout safe: the branch, index, working + tree, generated files and running commands are shared. Read-only workers, sequential work and a + single writer need none. +- Dispatch a worktree worker only once its inputs are committed at its base ref or provisioned in + the worktree, and record the base commit. **A new worktree lacks every untracked and gitignored + file**: dependencies, build output, local databases, scratch data. `CLAUDE.md` lists what this + repository's worktrees are missing. +- **Single-instance resources** cannot be shared by concurrent work: a dev server's port, a local + database file, a GPU, a browser profile, the owner's live checks. `CLAUDE.md` names this + repository's. A choice not to use worktrees must account for these and for branch isolation, + not just for file overlap. +- Stage explicit paths. Never `git add .` or a whole directory in the main checkout; untracked + drafts get swept in. + +## Parallel lead sessions + +More than one lead session can run at once (for example one on research beside one on the product). + +- Each lead owns its plan and its branch. It does not commit to another lead's branch or edit + another lead's plan; anything cross-cutting goes to the owner. +- Files every lead touches (`CLAUDE.md`, `docs/decisions.md`, release notes, shared contracts) + change in small, separate commits. Merge the default branch in just before opening the PR. In a + decisions-index conflict keep both sides' lines, newest first. +- A shared-contract change can break another lead's branch: say so in the PR description. +- One branch at a time bumps a given version; check the release notes on the default branch first. +- A lead claims the single-instance resources it needs in its plan; if another active plan already + claims one, ask the owner. diff --git a/.claude/workflow.json b/.claude/workflow.json new file mode 100644 index 0000000..944c557 --- /dev/null +++ b/.claude/workflow.json @@ -0,0 +1,6 @@ +{ + "claude_md_max_bytes": 14336, + "decisions_index_max_bytes": 20000, + "decisions_line_max_chars": 250, + "branch_guard_exempt": [] +} diff --git a/.claude/workflow.lock.json b/.claude/workflow.lock.json new file mode 100644 index 0000000..dde2903 --- /dev/null +++ b/.claude/workflow.lock.json @@ -0,0 +1,18 @@ +{ + "excluded": [], + "files": { + ".claude/agents/architect.md": "48fe7a09c04c46670265be39dab3c2f1152f84ee8e36875f0c7aabd99db91631", + ".claude/agents/executor.md": "2751d168f860ebb595301f4c08f8a7b2a303b64f3781716d516b35f8fc282cc9", + ".claude/agents/explore.md": "5ad8a5c9ea6d6975e572c0faaff6db9bc1345393eb1cce23392be2c3fc7c6693", + ".claude/agents/planner.md": "9b96c1c3be5cd8b220502ae437a14dc9e0c6c8e7b74ad01ecb386d323025f0c9", + ".claude/agents/reviewer.md": "d79012452a4a177372e58b30ce86df7a7db23dc162b8e43cbfdc4a33c9fe61be", + ".claude/hooks/guard-default-branch.py": "694d4a65cced3879ee6e125a6d039d60c572a8c7e6b205654d19dc4f7f03aba2", + ".claude/skills/checkpoint/SKILL.md": "e773f095fca977e9872d0b7c13cc7918b065443de8095896d0097aeb07c1e739", + ".claude/skills/handoff/SKILL.md": "a1710e7a140476f138537e7d0080e01e98fe3702f61a4ec2f9eae19ecea75e59", + ".claude/skills/parallel-work/SKILL.md": "da50f448eef1afb036b2df00a437236995e693b063931641016628692fc012eb", + ".claude/workflow/check.py": "c3e23ff58df8e13365d69c85a5a54d1bfbc9557f08d745e426426715e09d6220", + "docs/plans/_template.md": "88964018a9ae3f890d005940663373d78ad062c50b92dd2f2d80bb28b5f3b259" + }, + "kit": "a622a54 2026-10-24", + "level": "full" +} diff --git a/.claude/workflow/check.py b/.claude/workflow/check.py new file mode 100755 index 0000000..d8c367a --- /dev/null +++ b/.claude/workflow/check.py @@ -0,0 +1,232 @@ +#!/usr/bin/env python3 +"""Check that the workflow's docs and config agree with themselves. + + python3 .claude/workflow/check.py # from the repository root + python3 .claude/workflow/check.py --root DIR + +Copied into the repository by the workflow kit (managed: `workflow-kit sync` updates it). Run it +in CI or a test so a broken rule fails a build rather than relying on anyone to remember it. +Budgets come from `.claude/workflow.json`. Standard library only; exit 1 on any failure. + +Checks: +- CLAUDE.md stays under its byte budget (it loads in every session). +- Every skill in .claude/skills/ has a name matching its directory and a description, and is + named in backticks in CLAUDE.md or an agent file (an unnamed skill is never opened). +- Every path-scoped rule in .claude/rules/ has a strict `paths:` header whose fixed prefixes exist + (a header that fails to parse makes the rule load in every session instead). +- docs/decisions.md, if present: under its byte budget, lines within the length limit, each theme + heading names an existing detail file; every detail entry opens with a status line; every index + line is claimed by an entry marked `live, indexed`, and every claim names an index line. +- docs/plans/README.md, if present: its "Active plans" list agrees with each dated plan's + `Status:` line (a closed plan is not listed; an open one is). +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path + +DEFAULTS = { + "claude_md_max_bytes": 14336, + "decisions_index_max_bytes": 20000, + "decisions_line_max_chars": 250, +} +STATUS = re.compile( + r"^\*\*Status:\*\* (live, indexed|live, not indexed|superseded by \d{4}-\d{2}-\d{2}" + r"|superseded|withdrawn|record)(?: — (.+))?$" +) +INDEX_LINE = re.compile(r"^- (\d{4}-\d{2}(?:-\d{2})?) — ") +HEADING = re.compile(r"^## (\d{4}-\d{2}(?:-\d{2})?) — ") +TERMINAL = re.compile(r"^(Done|Closed|Superseded|Shelved)\b", re.I) + + +class Report: + def __init__(self) -> None: + self.failures: list[str] = [] + + def fail(self, check: str, detail: str) -> None: + self.failures.append(f"{check}: {detail}") + + +def frontmatter(text: str) -> str | None: + text = text.replace("\r\n", "\n") + m = re.match(r"^---\n(.*?)\n---(?:\n|$)", text, re.S) + return m.group(1) if m else None + + +def check_claude_md(root: Path, cfg: dict, r: Report) -> None: + path = root / "CLAUDE.md" + if not path.exists(): + r.fail("CLAUDE.md", "missing") + return + size = path.stat().st_size + if size > cfg["claude_md_max_bytes"]: + r.fail("CLAUDE.md", f"{size} bytes, over the {cfg['claude_md_max_bytes']} budget: " + "cut always-on text, or move procedures to a skill") + + +def check_skills(root: Path, r: Report) -> None: + skills_dir = root / ".claude/skills" + if not skills_dir.is_dir(): + return + always_on = [(root / "CLAUDE.md").read_text(encoding="utf-8-sig")] if (root / "CLAUDE.md").exists() else [] + agents = root / ".claude/agents" + if agents.is_dir(): + always_on += [p.read_text(encoding="utf-8-sig") for p in sorted(agents.glob("*.md"))] + text = "\n".join(always_on) + for skill in sorted(p for p in skills_dir.iterdir() if p.is_dir()): + file = skill / "SKILL.md" + if not file.exists(): + r.fail("skills", f"{skill.name}: missing SKILL.md") + continue + fm = frontmatter(file.read_text(encoding="utf-8-sig")) or "" + if not re.search(rf"^name: *[\"']?{re.escape(skill.name)}[\"']? *$", fm, re.M): + r.fail("skills", f"{skill.name}: frontmatter `name:` must match its directory") + if not re.search(r"^description: \S", fm, re.M): + r.fail("skills", f"{skill.name}: frontmatter needs a `description:`") + if f"`{skill.name}`" not in text: + r.fail("skills", f"{skill.name}: not named in backticks in CLAUDE.md or an agent " + "file, so nothing tells Claude to load it") + + +def check_rules(root: Path, r: Report) -> None: + rules_dir = root / ".claude/rules" + if not rules_dir.is_dir(): + return + for rule in sorted(rules_dir.rglob("*.md")): + name = rule.relative_to(rules_dir) + fm = frontmatter(rule.read_text(encoding="utf-8-sig")) + if fm is None: + continue # no header: loads in every session, which is allowed + first, *items = fm.split("\n") + if first != "paths:" or not items or not all( + re.match(r'^ {2}- "[^"]+"$', line) for line in items + ): + r.fail("rules", f"{name}: header must be `paths:` then lines of ` - \"glob\"`") + continue + for line in items: + glob = re.match(r'^ {2}- "([^"]+)"$', line).group(1) + prefix = re.split(r"[*?[{]", glob)[0].rstrip("/") + if prefix and not (root / prefix).exists(): + r.fail("rules", f"{name}: `{glob}` names `{prefix}`, which does not exist") + + +def check_decisions(root: Path, cfg: dict, r: Report) -> None: + index_path = root / "docs/decisions.md" + if not index_path.exists(): + return + index = index_path.read_text(encoding="utf-8-sig") + size = len(index.encode()) + if size > cfg["decisions_index_max_bytes"]: + r.fail("decisions", f"index is {size} bytes, over {cfg['decisions_index_max_bytes']}: " + "prune lines (set their entries to `live, not indexed`)") + named: set[str] = set() + for line in index.split("\n"): + if re.match(r"^- \d{4}-", line) and not INDEX_LINE.match(line): + r.fail("decisions", f"index line not in `- YYYY-MM-DD — text` form: {line[:70]}") + if INDEX_LINE.match(line) and len(line) > cfg["decisions_line_max_chars"]: + r.fail("decisions", f"line over {cfg['decisions_line_max_chars']} chars: {line[:70]}…") + for section in re.split(r"^(?=## )", index, flags=re.M): + m = re.match(r"^## .*`docs/decisions/([\w-]+\.md)`", section) + if not m: + stray = [l for l in section.split("\n") if INDEX_LINE.match(l)] + if stray: + r.fail("decisions", f"index line under a heading that names no theme file: " + f"{stray[0][:60]}") + continue + file = m.group(1) + named.add(file) + dates = [d.group(1) for d in (INDEX_LINE.match(l) for l in section.split("\n")) if d] + detail = root / "docs/decisions" / file + if not detail.exists(): + r.fail("decisions", f"theme heading names {file}, which does not exist") + continue + lines = detail.read_text(encoding="utf-8-sig").split("\n") + claims: list[tuple[int, str]] = [] + for i, line in enumerate(lines): + h = HEADING.match(line) + if not h: + continue + nxt = next((l for l in lines[i + 1:] if l.strip()), "") + s = STATUS.match(nxt) + if not s: + r.fail("decisions", f"{file}:{i + 1}: entry must open with a valid `**Status:**` " + f"line, found: {nxt[:60]}") + continue + if s.group(1) != "live, indexed": + continue + note = s.group(2) or "" + pointed = re.findall(r"under the (\d{4}-\d{2}-\d{2}) line", note) + own = [] if note.startswith("under ") else [h.group(1)] + claims += [(i + 1, d) for d in own + pointed] + claimed = {d for _, d in claims} + for d in dates: + if d not in claimed: + r.fail("decisions", f"{file}: index line {d} has no entry marked `live, indexed`") + for line_no, d in claims: + if d not in dates: + r.fail("decisions", f"{file}:{line_no}: marked `live, indexed` for {d}, but the " + "index has no line of that date") + theme_dir = root / "docs/decisions" + if theme_dir.is_dir(): + for p in sorted(theme_dir.glob("*.md")): + if p.name not in named: + r.fail("decisions", f"{p.name} is not named by any heading in docs/decisions.md") + + +def check_plans(root: Path, r: Report) -> None: + plans = root / "docs/plans" + readme = plans / "README.md" + if not readme.exists(): + return + text = readme.read_text(encoding="utf-8-sig") + parts = re.split(r"^## Active plans$", text, flags=re.M) + if len(parts) < 2: + r.fail("plans", "README.md has no `## Active plans` section") + return + active = parts[1].split("\n## ")[0] + listed = re.findall(r"^- `([^`]+\.md)`", active, re.M) + status = {} + for p in sorted(plans.glob("*.md")): + if not re.match(r"^\d{4}-\d{2}-\d{2}-.+\.md$", p.name) or p.name.endswith("-log.md"): + continue + head = "\n".join(p.read_text(encoding="utf-8-sig").split("\n")[:12]) + m = re.search(r"^- \*\*Status:\*\*\s*(.*)$", head, re.M) + if not m: + r.fail("plans", f"{p.name}: no `- **Status:**` line in its header") + continue + status[p.name] = m.group(1).replace("*", "").strip() + for name in listed: + if name not in status: + r.fail("plans", f"active list names {name}, which is not a dated plan here") + elif TERMINAL.match(status[name]): + r.fail("plans", f"{name} is listed as active but its status is: {status[name][:40]}") + for name, s in status.items(): + if not TERMINAL.match(s) and name not in listed: + r.fail("plans", f"{name} is open ({s[:30]}) but missing from the active list") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--root", default=".") + root = Path(parser.parse_args().root).resolve() + cfg_path = root / ".claude/workflow.json" + cfg = {**DEFAULTS, **(json.loads(cfg_path.read_text(encoding="utf-8-sig")) if cfg_path.exists() else {})} + r = Report() + check_claude_md(root, cfg, r) + check_skills(root, r) + check_rules(root, r) + check_decisions(root, cfg, r) + check_plans(root, r) + for failure in r.failures: + print(f"FAIL {failure}") + if not r.failures: + print("workflow check: all passed") + return 1 if r.failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9a00031..7429256 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -52,6 +52,15 @@ jobs: - name: Test run: uv run --no-sync pytest -q -rs + workflow: + name: Workflow check + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v7.0.1 + # CLAUDE.md budget, decisions index format, skills and the plans' active list. + - run: python3 .claude/workflow/check.py + viewer-and-wheel: name: Viewer and wheel runs-on: ubuntu-latest diff --git a/.gitignore b/.gitignore index 4efefb8..f8f029a 100644 --- a/.gitignore +++ b/.gitignore @@ -16,3 +16,7 @@ src/jepa_scope/static/ *.mp4 *.mov *.webm + +# Claude Code: per-machine settings and agent worktrees +.claude/settings.local.json +.claude/worktrees/ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..4dc9b89 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,136 @@ +# jepa-scope + +jepa-scope is an open-source (Apache-2.0, public at `ryangswe/jepa-scope`) local inspector for JEPA +video encoders. `jepa-scope clip.mp4` embeds a video with Meta's pre-trained V-JEPA 2 ViT-L on a +laptop and opens a browser viewer with PCA feature colours, a surprise overlay and a surprise +timeline. It is the first of three planned tools; the programme is in `docs/roadmap.md`. + +## Layout + +- `src/jepa_scope/`: the Python package. `video.py` decodes (PyAV, rotation, downscale), + `model.py` wraps the encoder and predictor, `embed.py` turns a video into a `.latent` file, + `server.py` serves it, `cli.py` is the entry point, `grid.py` owns the token layout. +- `web/`: the Vite, React and TypeScript viewer (WebGL2 overlay in `web/src/overlay.ts`). It builds + into `src/jepa_scope/static/` (git-ignored), which the wheel ships. +- `tests/`: fast tests on a tiny random V-JEPA 2; `test_real_model.py` is opt-in (real weights). +- `docs/`: `roadmap.md`, the decisions record and plans. +- **Shared contracts that change together:** the `.latent` file (`latent_file.py`: tensor names, + shapes, header keys, `FORMAT_VERSION`) with `embed.py` and `server.py`; and the `/api/meta` JSON + (`server.py`) with `web/src/types.ts`. + +## Commands (run from the repository root) + +- Install: `uv sync` and `cd web && npm ci` +- Lint: `uv run ruff check src tests && uv run ruff format --check src tests`; + viewer `cd web && npx oxlint src` +- Typecheck (viewer): `cd web && npx tsc -b` +- Test: `uv run pytest -q`. Real weights (about 1.3 GB download, then cached): + `JEPA_SCOPE_MODEL_TESTS=1 uv run pytest tests/test_real_model.py -s` +- Build: `./scripts/build.sh` (viewer, wheel, and a check that the wheel contains the viewer) +- Run: `uv run jepa-scope clip.mp4`, or `uv run jepa-scope view x.latent --no-browser` beside + `cd web && npm run dev` for viewer work +- Workflow check: `python3 .claude/workflow/check.py` + +## Working agreements + +- **Never change files on the default branch** (`main`/`master`). Branch first + (`git checkout -b `). `.claude/hooks/guard-default-branch.py` blocks Edit/Write there as + a backstop, but cannot see shell writes (`sed -i`, `>>`). +- Record decisions a later session could plausibly reopen (direction, architecture, contracts, + security, rejected alternatives): one line in the auto-loaded index @docs/decisions.md and the + full entry in its `docs/decisions/.md`. The index header sets the format and budget. +- For large milestones, write the plan to `docs/plans/YYYY-MM-DD-.md` from + `docs/plans/_template.md`, so a fresh session can execute it from the file alone. Keep its + `Status:` line and the active list in `docs/plans/README.md` current. +- **Output-changing edits bump `OUTPUT_VERSION`** in `src/jepa_scope/__init__.py` (energy, + preprocessing, windowing, PCA). It keys the cache; without a bump users see stale results. +- **A signal that runs is not a signal that works.** Wrong JEPA preprocessing or energy code + produces plausible numbers with no error. Check changes to `model.py`, `embed.py` or `video.py` + against the real-model test on natural video, not only the fast tests. +- **Test media and model weights never enter the repo.** Use the Kinetics clips the opt-in test + downloads, or local files outside the checkout. +- **This repository is public.** Write docs, commits and PRs for outside readers: full sentences, + no references to private repositories or internal tracking. +- `CLAUDE.md` loads in every session and is capped at 14 KB by the workflow check. Always-on text + is for a rule every session needs; add some only by cutting some. Procedures go in a skill, area + rules in `.claude/rules/` with a `paths:` header. + +## Orchestration and delegation + +For non-trivial work the lead session is the orchestrator: it owns the goal, the architecture, the +task graph, integration and every exchange with the owner. Workers are the agents in +`.claude/agents/` (`planner`, `architect`, `executor`, `reviewer`, `Explore`); their checklists +and report formats live there. + +- **Milestone specs: do not edit files until the architecture and the first small change are + explained and approved.** Inspect read-only first. +- Decompose work into tasks with dependencies, complexity class, success criteria, escalation + conditions and verification. **Price every task in the owner's hours**: they are the scarcest + resource. +- Work directly when a task is small, tightly coupled or cheaper to do than to brief. Delegate when + work can run concurrently, is self-contained, would flood the lead's context, or needs a fresh + judgment. Every worker gets a bounded task and ends with SUCCESS, ESCALATE or BLOCKED; on + ESCALATE or BLOCKED, stop the tasks that depend on it and update the plan. +- Present the task graph, delegation, model classes and approval gates **before dispatching + code-writing workers**, unless the owner has authorised autonomous work. Read-only workers need + no approval. Ask before anything that changes product behaviour, architecture, compatibility, + security or privacy posture, production data, scope or cost. + +### Owner-only actions (each time, never implied by plan approval) + +- Merging into the default branch. +- Publishing a release: tags, GitHub releases, PyPI uploads. +- Changing repository settings or visibility, or posting about the project anywhere public. +- Paid compute, such as renting a GPU for embedding runs or `eb_jepa` training. + +### Model routing + +**The lead runs on Opus at high effort**, chosen at the start of a session. Plans name capability +classes; map them at dispatch: ROUTINE → Sonnet (Haiku for lookups), STRONG → Opus, FRONTIER → +Fable. Update the mapping when models change. + +| Agent | Default | Use | +|---|---|---| +| `Explore` | Haiku, low | code search; fetching one decision entry, plan task or doc section | +| `executor` | Sonnet, medium | ROUTINE tasks; override to Opus for STRONG | +| `reviewer` | Opus, high | every review; Sonnet only for ROUTINE-only work | +| `planner` | Fable, high | milestone plans, re-plans after an ESCALATE | +| `architect` | Fable, high | judgement calls mid-task; resume it with `SendMessage` for follow-ups | + +- Call FRONTIER where it pays: a milestone plan, a rule or bar set before measuring, a re-plan, a + result the lead cannot explain, a root cause after two strikes. +- **Keep the lead's context small.** The lead does not read a decisions theme file or a plan over + ~30 KB itself; `Explore` fetches the entry or task it needs. +- **Don't switch the lead's model or effort mid-session**: each model has its own prompt cache, so + the next turn re-reads the whole conversation uncached. Ask `architect`, or start a session. +- Never set `CLAUDE_CODE_SUBAGENT_MODEL_FORCE`: it ignores the agent files' `model`. + +### Concurrency + +- Before running workers at the same time, or beside another lead session, load the + `parallel-work` skill. +- Every concurrent code-writing worker gets its own worktree. Stage explicit paths; never + `git add .` or a whole directory. +- Single-instance resources: the viewer's default port 8765 (pass `--port` in a second + worktree), and the Apple GPU (MPS) when two sessions embed at once. The Hugging Face cache + (`~/.cache/huggingface`) and the result cache (`~/.cache/jepa-scope`) are shared and safe. +- A new worktree lacks `.venv`, `web/node_modules` and the built viewer: run `uv sync`, + `cd web && npm ci && npm run build`. + +### Review, integration and closeout + +- Run `/checkpoint` (the `checkpoint` skill) after each integrated step. +- **Two strikes:** when the same test, typecheck or build error survives two fix attempts, stop + patching. Ask `reviewer` (or `architect` when the cause is unclear) for a root-cause read before + a third attempt; a worker reports ESCALATE instead. +- Before declaring non-trivial work done, run `reviewer` with fresh context on the plan and the + diff. Passing tests are not proof. Fix the material findings, re-run the checks. +- Ending a session mid-task, switching topic, or past ~300–400k tokens: run `/handoff` (the + `handoff` skill). It records state in the repository and prints a short prompt for the next + session. Don't let a conversation be the only place work's state lives. +- **Close out:** the plan's `Status:` line and the active list; the execution log compressed into + the completion summary with its lesson or `NO_ENTRY`; decision lines that meet the index bar; a + short summary for the owner. + + diff --git a/README.md b/README.md index 9f3e0ae..71b1ce3 100644 --- a/README.md +++ b/README.md @@ -150,14 +150,10 @@ arrays once, so scrubbing only changes a uniform. The first week's milestones are done. The model runs on a laptop, the token grid is pinned by a test, the PCA overlay and surprise timeline work, the file format is defined, and the viewer runs -in the browser. - -Next up: - -- A 3D view of each clip's path through latent space, synced to playback. -- Nearest-neighbour search across a folder of clips, using the pooled embeddings. -- Health checks for a set of embeddings, such as effective rank and per-dimension variance. -- Adapters for other encoders behind the same file format, starting with `eb_jepa` models. +in the browser. [docs/roadmap.md](docs/roadmap.md) has the full programme: the developer +bottleneck this addresses, the missing toolchain layers, the three planned tools of which +jepa-scope is the first, and what comes next. Settled design choices and their reasons are in +[docs/decisions.md](docs/decisions.md). ## License diff --git a/docs/decisions.md b/docs/decisions.md new file mode 100644 index 0000000..996a4e2 --- /dev/null +++ b/docs/decisions.md @@ -0,0 +1,43 @@ +# Decisions index + +One short line per **live** decision, so every session sees the settled surface cheaply and does +not re-litigate it. **This index is auto-loaded every session**, so every byte is paid by every +conversation. The *why*, the rejected alternatives and every past decision live in the per-theme +detail files under `docs/decisions/` named in each heading. **Before reopening or contradicting a +decision, read its detail entry**: `rg -A2 '^## 20'` on the theme file lists every entry with its +status; read that entry only, not the file. + +**What earns an entry:** a choice a future session could plausibly reopen: direction, +architecture, contracts, security and privacy posture, licensing, or an alternative that was tried +and rejected. **Not an entry:** experiment results, details settled inside an approved plan, and +anything the code already shows. + +**Format and budget** (checked by `.claude/workflow/check.py`): +- `- YYYY-MM-DD — `, newest first under its theme, at most 250 characters (aim for about + 150). +- Each detail entry opens with `**Status:**` `live, indexed`, `live, not indexed`, `superseded by + `, `withdrawn` or `record` (results only). Update it when its line goes. An entry covered + by a line of another date says `live, indexed — under the line`. +- Add the full entry (context, decision, rejected alternatives) to the matching + `docs/decisions/.md` as `## YYYY-MM-DD — `, newest first, `---` between + entries. Start a new theme file only if none fits, and add its section here. +- The whole index stays under 20 KB. Near it, prune lines first. + +--- + +## Method · `docs/decisions/method.md` + +- 2026-11-05 — Surprise = mean L1 between predictor output and layer_norm(target), with the context frames encoded on their own; never the HF full-clip forward +- 2026-11-05 — Windows of 16 frames at 8 fps, overlapping by half: context is the first half, every later tubelet is predicted exactly once +- 2026-11-05 — Any output-changing edit bumps OUTPUT_VERSION, which keys the result cache + +## Product & data · `docs/decisions/product.md` + +- 2026-11-05 — Apache-2.0, public at ryangswe/jepa-scope; V-JEPA 2 weights are downloaded at run time, never redistributed +- 2026-11-05 — `.latent` = safetensors plus a validated JSON header; the viewer refuses a video whose SHA-256 differs from the header +- 2026-11-03 — jepa-scope is the first tool; the order of the other two is a proposal; tools target a model-agnostic protocol rather than V-JEPA 2 + +## Process & tooling · `docs/decisions/process.md` + +- 2026-11-05 — Adopted the workflow kit at the full level, with its process files, the roadmap and this record published in the public repo +- 2026-11-05 — CI installs uv.lock versions as constraints with CPU torch; the real-model test stays opt-in and runs locally diff --git a/docs/decisions/method.md b/docs/decisions/method.md new file mode 100644 index 0000000..d890ac0 --- /dev/null +++ b/docs/decisions/method.md @@ -0,0 +1,73 @@ +# Decisions — Method + +Full rationale for how jepa-scope computes what it shows, newest first. The lean, auto-loaded +index is `docs/decisions.md`. + +--- + +## 2026-11-05 — Surprise = mean L1 between predictor output and layer_norm(target), with the context frames encoded on their own; never the HF full-clip forward + +**Status:** live, indexed + +**Context.** Surprise is the per-patch prediction error for the second half of each window. Two +mistakes each produce a flat, useless signal while running without any error, and both were made +and caught during Week 1. + +**Decision.** `Encoder.predict` runs the encoder on the context frames only, then calls the +predictor with token-id masks (the ids double as RoPE positions). `Encoder.surprise` compares that +prediction with `F.layer_norm` of the full-clip encoder output for the target tokens, using mean +absolute error. This matches `forward_target` and `loss_fn` in `app/vjepa/train.py` of +`facebookresearch/vjepa2`. + +**Evidence.** In one measured run on an archery clip spliced onto a bowling clip, surprise stayed +between about 0.57 and 0.62 within each scene and rose to 0.65–0.67 for the four tubelets after +the cut. `tests/test_real_model.py` reproduces that clip but asserts only the ordering: the four +tubelets after the cut score higher than every other tubelet. Without the layer norm, targets had +a standard deviation of about 3.3 against about 0.65 for predictions, and the timeline was flat. + +**Rejected.** +- *`VJEPA2Model.forward` with `context_mask`.* It encodes the whole clip and masks afterwards, so + context tokens have already attended to the future. `test_prediction_ignores_the_future` keeps a + negative control showing this path leaks. +- *L1 against raw encoder output.* It does not match the training objective and gave a flat line. +- *Correcting the per-horizon rise.* Within each window, tubelets further from the context are + about 0.01 harder to predict, which shows as a small sawtooth on the timeline. It is documented + in the README rather than corrected, to keep the raw energy. + +--- + +## 2026-11-05 — Windows of 16 frames at 8 fps, overlapping by half: context is the first half, every later tubelet is predicted exactly once + +**Status:** live, indexed + +**Context.** The viewer needs one continuous surprise timeline, while the model predicts within +fixed windows. + +**Decision.** Defaults are 8 fps and 16-frame windows (2 seconds, 2,048 tokens on ViT-L) with a +stride of half a window. `embed.timeline_owner` maps every tubelet to exactly one window. The first +half of the first window is context only and has no surprise. Sources slower than the requested +frame rate are refused, because each window would silently cover more time than the header says. + +**Rejected.** +- *64-frame windows,* the checkpoint's training length. They give 8,192 tokens, and attention + cost grows with the square of that count, which is too slow on a laptop. +- *Non-overlapping windows.* Half of every window would have no surprise value. + +--- + +## 2026-11-05 — Any output-changing edit bumps OUTPUT_VERSION, which keys the result cache + +**Status:** live, indexed + +**Context.** `jepa-scope clip.mp4` caches results in `~/.cache/jepa-scope`. A cache that ignores +code changes keeps showing old results. That is exactly how a pre-fix flat surprise line would +survive an upgrade. + +**Decision.** `OUTPUT_VERSION` in `src/jepa_scope/__init__.py` is part of the cache key and is +recorded in each file's header. A cached file from another output version is re-embedded. The key +also covers the video hash, model, frame rate, window, length limit, device, dtype and whether +patches are kept. + +**Rejected.** +- *Keying on the package version.* It changes for reasons unrelated to output and can stay the + same across output changes. diff --git a/docs/decisions/process.md b/docs/decisions/process.md new file mode 100644 index 0000000..a1053a2 --- /dev/null +++ b/docs/decisions/process.md @@ -0,0 +1,49 @@ +# Decisions — Process & tooling + +Full rationale for this theme's decisions, newest first. The lean, auto-loaded index is +`docs/decisions.md`. + +--- + +## 2026-11-05 — Adopted the workflow kit at the full level, with its process files, the roadmap and this record published in the public repo + +**Status:** live, indexed + +**Context.** jepa-scope is the first of three planned tools and will run over many sessions. Until +now its plan lived only in the conversation that produced it. The owner asked for the plan to be +written down and for the same written workflow used in their other projects. + +**Decision.** The repository uses the workflow kit (github.com/ryangswe/claude-workflow) at the +**full** level: a committed `.claude/` with the planner, architect, executor, reviewer and Explore +agents, the checkpoint, handoff and parallel-work skills, and a branch guard. It also has this +decisions record, plans in `docs/plans/`, and `.claude/workflow/check.py` keeping them consistent, +which CI runs. The kit's managed files are updated with `workflow-kit sync`; `CLAUDE.md` and these +docs are the repository's own. The owner approved publishing the process files, `docs/roadmap.md` +and this record in the public repository. + +**Rejected.** +- *The light level.* It has no plans or planner, and this is a multi-session programme. +- *Keeping the process files private.* The owner chose to build in public. The files describe how + the work is done and contain nothing sensitive. + +--- + +## 2026-11-05 — CI installs uv.lock versions as constraints with CPU torch; the real-model test stays opt-in and runs locally + +**Status:** live, indexed + +**Context.** The locked Linux torch pulls about 3 GB of CUDA packages that a CI runner cannot use. +A first version of the workflow installed fresh versions instead and so tested a different +environment from the lockfile's. + +**Decision.** `.github/workflows/ci.yml` checks the lockfile, exports it as constraints, and +installs it with `uv pip install --torch-backend cpu`. Every package except torch and torchvision +is pinned to its locked version, and those two use the CPU build. uv is pinned to 0.13.6, the +version used locally when the lockfile was written. The real-model test needs about 1.3 GB of +weights and is meaningful only on natural video, so it stays opt-in and is run locally before +changes to the model, embedding or decoding code. + +**Rejected.** +- *`uv sync --locked` in CI.* It downloads the CUDA packages on every uncached run. +- *Changing the lockfile to CPU torch on Linux.* That would give Linux users with NVIDIA GPUs a + CPU-only install. diff --git a/docs/decisions/product.md b/docs/decisions/product.md new file mode 100644 index 0000000..047b6da --- /dev/null +++ b/docs/decisions/product.md @@ -0,0 +1,67 @@ +# Decisions — Product & data + +Full rationale for what jepa-scope is, what it stores and how it is published, newest first. The +lean, auto-loaded index is `docs/decisions.md`. + +--- + +## 2026-11-05 — Apache-2.0, public at ryangswe/jepa-scope; V-JEPA 2 weights are downloaded at run time, never redistributed + +**Status:** live, indexed + +**Context.** The project's purpose is an open developer layer for JEPA models, so it is built in +public. + +**Decision.** The code is Apache-2.0 and the repository is public. The tool downloads V-JEPA 2 +weights from Hugging Face when it first runs, and neither the repository nor the wheel contains +them. Meta releases V-JEPA 2 under MIT, according to the model card and the +`facebookresearch/vjepa2` repository. Test media is never committed either: the opt-in test +downloads Kinetics sample clips at run time, and `.gitignore` blocks video files. + +**Rejected.** +- *Bundling a small checkpoint or sample video.* It is a licensing and repository-size burden with + no benefit, since the first run downloads what it needs. + +--- + +## 2026-11-05 — `.latent` = safetensors plus a validated JSON header; the viewer refuses a video whose SHA-256 differs from the header + +**Status:** live, indexed + +**Context.** Latents carry no record of their layout, model or preprocessing. Pairing them with the +wrong video or settings draws plausible but wrong overlays. + +**Decision.** A `.latent` file is a plain safetensors file with one JSON header under the metadata +key `jepa_scope` (`FORMAT_VERSION` 1). The header records the model, grid, window, preprocessing +recipe and hash, crop rectangle, source video hash, size and rotation, and the run. `load` validates +the header keys and every tensor shape against the grid and the window count. `jepa-scope view` +refuses a video whose size or SHA-256 differs from the header. Files are written to a temporary +name and renamed, so an interrupted write never leaves a half file behind. + +**Rejected.** +- *NumPy `.npz` or a directory of files.* Safetensors reads single tensors without loading the + whole file, has no code-execution path, and has readers for Rust and JavaScript, which later + tools may need. +- *Trusting the video path in the header.* Videos move and get re-encoded. + +--- + +## 2026-11-03 — jepa-scope is the first tool; the order of the other two is a proposal; tools target a model-agnostic protocol rather than V-JEPA 2 + +**Status:** live, indexed + +**Context.** `docs/roadmap.md` proposes three solo-sized tools: jepa-scope (a latent inspector), +jepa-probe (few-shot heads behind an HTTP API) and latent-plan (a planning runtime). The owner +asked for jepa-scope to be scaffolded first. + +**Decision.** jepa-scope comes first because it needs only pre-trained weights, has the clearest +demo, and produces the latent format and viewer shell the other two reuse. Tools target a protocol +(the `.latent` contract now, and `encode`, `predict` and `energy` for planning later), so new +weights from Meta, AMI Labs or others become an adapter rather than a rewrite. The order of +jepa-probe and latent-plan stays a proposal until a milestone for one of them is planned. + +**Rejected.** +- *Starting with latent-plan.* It carries the most research risk, because planning quality depends + on the model, and it has the smallest audience today. +- *Training a model.* That is out of reach for a solo developer and unnecessary for a developer + layer. diff --git a/docs/plans/README.md b/docs/plans/README.md new file mode 100644 index 0000000..5113f5c --- /dev/null +++ b/docs/plans/README.md @@ -0,0 +1,31 @@ +# Plans + +Durable implementation plans, so a **fresh session** — often on a different model from the one +that wrote the plan — can execute the work from the file alone. + +## Naming + +`YYYY-MM-DD-.md`, dated when the plan was written. One file per milestone or effort. + +## Active plans + +Add a plan as soon as its file is committed (a Draft counts); remove it when its status becomes +Done, Closed, Superseded or Shelved. Every other file here is a record. `.claude/workflow/check.py` fails when this list and a +plan's `Status:` line disagree. + +## Lifecycle + +- **Draft** while planning; reviewed by the owner before any code is touched. +- **In progress**: the implementing session marks tasks as it lands them. +- **Done** when every step is verified and committed; **Superseded** (name the successor) or + **Shelved** when it stops. Update the `Status:` line and the list above in the same commit. +- **The execution log lives beside the plan**, in `-log.md`, appended as work proceeds, so + the plan stays a lean spec. +- **On Done**, compress the log into the completion summary (outcome, deviations, follow-ups, + PRs), then delete the `-log.md` file. Git keeps the full log. +- **Write the lesson field** (at most one, or `NO_ENTRY`), then list earlier ones with + `rg -n '^- \*\*Lesson:\*\* cue' docs/plans`. When a lesson repeats one from an earlier plan, + promote it: a line in `CLAUDE.md` or an agent file, or better a hook or a check, which cannot be + skipped the way prose can. + +Small fixes and one-sitting features don't need a plan file. diff --git a/docs/plans/_template.md b/docs/plans/_template.md new file mode 100644 index 0000000..1e4b1e4 --- /dev/null +++ b/docs/plans/_template.md @@ -0,0 +1,169 @@ +# + +- **Date:** YYYY-MM-DD +- **Status:** Draft | Approved | In progress | Blocked | Done | Closed | Superseded (by …) | Shelved +- **Planned by:** +- **Spec / source:** +- **Branch:** `` + +## Goal + +Describe what this effort delivers in 2–4 sentences. State the scope boundary +explicitly: what this deliberately does **not** build yet, and why. + +## Approach + +Describe the architecture, affected workspaces, important seams, repository invariants, +and any shared-type coupling that requires coordinated changes. Explain the intended +data/control flow where relevant. + +## Alternatives considered + +- **** — + +Promote decisions worth preserving beyond this effort to `docs/decisions.md`. + +## Risks and open questions + +- **** — + +Resolve questions requiring user input before changing the plan from Draft to Approved. + +## Dependency summary + +Replace the example with the actual task graph. Keep it small enough to scan quickly. + +```mermaid +flowchart LR + T1 --> T2 + T1 --> T3 + T2 --> T4 + T3 --> T4 +``` + +- **Parallel group A:** +- **Parallel group B:** +- **Integration order:** + +## Task graph + +Create independently understandable work units, not one task per file. Separate tasks do +not automatically imply separate agents. + +### T1 — + +- **Status:** Pending | In progress | Complete | Escalated | Blocked +- **Dependencies:** None | T, ... +- **Parallel group:** | Sequential +- **Complexity:** ROUTINE | STRONG | FRONTIER +- **Execution:** Lead session | Executor | Research worker +- **Owner time:** None | +- **Expected files/workspaces:** `path/...` +- **Objective:** +- **Success criteria:** + - + - +- **Verification:** + - `` +- **Escalate if:** + - + +### T2 — + +- **Status:** Pending | In progress | Complete | Escalated | Blocked +- **Dependencies:** T1 +- **Parallel group:** | Sequential +- **Complexity:** ROUTINE | STRONG | FRONTIER +- **Execution:** Lead session | Executor | Research worker +- **Owner time:** None | +- **Expected files/workspaces:** `path/...` +- **Objective:** +- **Success criteria:** + - +- **Verification:** + - `` +- **Escalate if:** + - + + + +## Execution strategy + +- **Lead/orchestrator owns:** +- **Delegated tasks:** +- **Model routing:** +- **Owner time, total:** +- **Approval gates:** + - user approval before code-writing dispatch + - user approval before +- **Review strategy:** + +### Concurrency and isolation + +A parallel group identifies tasks that are eligible to overlap. It does not mean they +will automatically execute concurrently. For every parallelizable task, explicitly state +whether it will actually run concurrently or sequentially. + +| Task | Actually concurrent with | Worker/session | Writes files? | Branch/base | Isolation | Shared mutable resources | Integration | +|---|---|---|---:|---|---|---|---| +| T1 | None | Lead | Yes | `` | Main checkout | `` | Direct | +| T2 | T3 | Executor | Yes | `` | Worktree | None | Cherry-pick | +| T3 | T2 | Research worker | No | `` | Read-only shared checkout | None | Report only | + +For each task scheduled to run concurrently, confirm: + +- **Execution timing:** Concurrent | Sequential | Deferred +- **Reason:** +- **Isolation:** Main checkout | Read-only shared checkout | Worktree | External environment +- **Base ref:** `` +- **Tracked inputs available:** Yes | No; +- **Shared resources:** None | +- **Output ownership:** +- **Integration method:** Direct | Merge | Cherry-pick | Artifact handoff + +## Execution log + +Keep the log in a sibling file, `-log.md`, not here: the plan is the spec a +fresh session executes from, and the log grows with every task. Record material discoveries, +decisions, escalations, and deviations there while work proceeds, in this format. Keep routine +implementation detail out of it. + +- **YYYY-MM-DD — : ** + - **Evidence:** + - **Impact:** + - **Resolution:** + - **Permanent decision record needed:** Yes | No (the bar is in the `docs/decisions.md` header) + +## Integration and verification + +### Integrated checkpoint + +- [ ] `/checkpoint` completed (typecheck → test → build → verify) +- [ ] Any checkpoint failures resolved and the checkpoint rerun + +### End-to-end acceptance + +- [ ] +- [ ] +- [ ] + +### Independent review + +- **Reviewer:** +- **Verdict:** Pending | Pass | Changes required +- **Findings:** +- [ ] Material findings resolved +- [ ] Relevant verification rerun after fixes + +## Completion summary + +- **Delivered:** +- **Deviations from approved plan:** None | +- **Known risks / follow-up:** None | +- **Decision records updated:** None | `docs/decisions.md` entries +- **Lesson:** NO_ENTRY | cue: / lesson: / evidence: / confidence: + + diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..992dfe2 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,214 @@ +# Roadmap + +- **Written:** 2026-11-03, from the planning conversation that started this project. +- **Last updated:** 2026-11-05, after the Week 1 scaffold and CI landed. +- **Status:** living document. Executable milestone plans go in `docs/plans/`. Settled choices + go in `docs/decisions.md`. + +## Why this project exists + +Language models produce text, so a developer can tell whether a call worked by reading the output. +That property is why an application layer grew so quickly around them, through hosted APIs, Ollama, +LangChain, LlamaIndex and vLLM. JEPA models (Joint-Embedding Predictive Architectures, the line of +work led by Yann LeCun) predict in a learned latent space instead. Their output is a tensor of +embeddings and an energy, and nothing in it tells a developer whether it is right. + +Meta's V-JEPA 2 and the `eb_jepa` library are good research code. What is missing is the developer +layer that would let an engineer without a research background build on them. This project builds +that layer in the open, under Apache-2.0, as a sequence of small tools that each remove one +concrete bottleneck. + +## The bottleneck, concretely + +Cloning `facebookresearch/vjepa2` and trying to use it shows where things break down. + +1. **Environment.** The setup assumes a CUDA cluster. Video decoding has historically used + `decord`, which is hard to install on Apple Silicon. Configs are YAML files for distributed + training, not a path for "I have one video". +2. **Weights.** There are several loading routes with different code: Hugging Face checkpoints, + `torch.hub`, and raw files. The action-conditioned model (V-JEPA 2-AC) loads only through + `torch.hub`. +3. **Preprocessing.** Frame count, sampling stride, resize, crop and normalisation all have to + match training. When any of them is wrong, nothing errors. The model returns plausible-looking + vectors that are simply wrong. +4. **Output.** The encoder returns something like `float16[8192, 1408]` (ViT-g on 64 frames). + The tensor carries no record of its time, row and column layout, so every tool has to rebuild + it. +5. **Tasks.** There is no prompt. Every task needs a trained head (a probe), and the published + evaluation recipes need datasets of hundreds of gigabytes. + +`eb_jepa` (arXiv 2602.03604, February 2026) is much friendlier, with small examples that train on +one GPU in a few hours. It is still a training codebase, with no serving, no stable API and an +energy reported as a unitless number in a training log. + +| | `ollama run llama3` | V-JEPA 2 today | +| --- | --- | --- | +| Input contract | Text | An undocumented preprocessing recipe | +| Output | Readable, so you can check it yourself | A tensor you cannot check by looking | +| Task interface | The prompt | A probe you train per task | +| Knowing it worked | Read it | Needs instruments | +| Runs on a laptop | Yes | ViT-L does, but nobody packaged it | + +## The missing toolchain + +These are the layers between a research checkpoint and an application, roughly in the order they +are needed. + +- **A. Runtime and packaging.** One loader that bundles the model, its preprocessing recipe and its + token layout. Decoding that works everywhere, MPS and CPU fallbacks, streaming with latent reuse, + and later ONNX or Core ML export. +- **B. A latent data contract.** A file format that stores latents together with the model id, + grid, preprocessing hash and source hash, so tools can share data without guessing. jepa-scope's + `.latent` file is the first version. +- **C. Visualisation and observability.** PCA colour maps of patch features, surprise maps (the + per-patch prediction error), embedding health checks such as effective rank and per-dimension + variance, and latent trajectories over time. +- **D. Latent-to-task adapters.** Probes trained in seconds on cached embeddings, retrieval over + pooled embeddings, small decoders for debugging, and alignment to language models. +- **E. Latent-to-action planning.** A model-agnostic protocol (`encode(obs) → z`, + `predict(z, a) → z'`, `energy(z, z_goal)`) with planners such as CEM and MPPI written against + it, and bridges to simulators and robot-learning datasets. +- **F. A laptop-scale evaluation harness.** Small benchmarks that finish in minutes, so tools and + new checkpoints can be compared. + +The tools target the protocol in B and E rather than V-JEPA 2 itself. When Meta, AMI Labs or anyone +else releases new weights, they become one more adapter instead of a reason to start over. + +## The three projects + +jepa-scope comes first, because it produces the data contract and viewer shell that the other two +reuse. The order of the other two is a proposal until one of them is planned. + +### 1. jepa-scope, a local latent inspector (this repository) + +`jepa-scope clip.mp4` embeds a video with V-JEPA 2 on a laptop and opens a browser viewer. The +viewer shows PCA feature colours over the video, a surprise overlay, and a surprise timeline you can +click to seek. It addresses the "I got a tensor, now what?" problem and the silent preprocessing +problem. + +- **Frontend leverage.** The whole timeline is uploaded once into WebGL2 texture arrays, so + scrubbing changes only a uniform. Planned additions are a Three.js view of a clip's path through + latent space and nearest-neighbour search across a folder of clips. +- **Stack.** A Python backend with FastAPI, Hugging Face `transformers` and PyAV, packaged with uv. + A Vite, React and TypeScript viewer that ships inside the Python wheel, so users never need npm. + Latents travel to the browser as raw bytes, not JSON. +- **MVP, four to six weeks.** Weeks 1 and 2 cover inference, the grid, the PCA overlay, the file + format and the CLI. Week 3 adds surprise maps. Week 4 adds folder indexing and nearest-neighbour + search. Weeks 5 and 6 add health metrics, polish and a launch demo. Surprise maps arrived early, + in Week 1, and this roadmap now also proposes the latent trajectory view for Weeks 3 and 4. + +### 2. jepa-probe, few-shot task heads behind an HTTP API + +This project is the JEPA equivalent of a prompt. A user labels 30 to 100 clip segments in a browser +and gets a working classifier or detector in under a minute. The result is served through endpoints +shaped like familiar model APIs, such as `POST /v1/video/embeddings` and +`POST /v1/heads/{name}/predict`, so an application developer works with labels instead of vectors. + +- **Frontend leverage.** Timeline labelling with drag-to-select and hotkeys, active learning that + surfaces the clips the head is least sure about, a live confusion matrix, and probe attention + drawn over the frame. +- **Stack.** It reuses jepa-scope's latent cache. Linear and attentive probes in PyTorch, exported + to ONNX. +- **MVP.** Embed a folder, label it, train and evaluate both probe types, and serve the result. One + end-to-end example built from about 50 labelled home videos. + +### 3. latent-plan, a planning runtime and playground + +This project is a planning runtime written against the protocol in layer E, with a browser UI. A +user clicks a goal and watches CEM or MPPI sample action sequences, roll them out in latent space +and converge on the energy landscape. The first adapter is an `eb_jepa` Two Rooms world model +trained locally. The second is V-JEPA 2-AC, run offline on a rented GPU. + +- **Frontend leverage.** The energy over a two-dimensional slice of action space as an animated + WebGL heatmap, sampled trajectories with the best ones highlighted, and the real rollout beside + the predicted one to show where the model's prediction drifts. +- **Stack.** The Python protocol and planners in PyTorch, gymnasium-style environment bridges, and + the same web shell as jepa-scope. +- **MVP.** The protocol, CEM and MPPI, the `eb_jepa` adapter, the Two Rooms UI, and a written spec + so other people can add their own models. +- **Risk.** This is the most research-dependent of the three, because planning quality depends on + the model, and its audience is smaller today. + +## Compute strategy + +1. **Never pretrain. Freeze the encoder.** Projects 1 and 2 are inference plus small heads. +2. **Embed once, then work on cached arrays.** The expensive encoder runs once per clip. Probes, + PCA, search and the viewer all work from cached latents on a CPU. At the defaults, patch + features cost about 4 MB per window and pooled embeddings about 4 KB. +3. **Use ViT-L with 16-frame windows.** Sixteen frames give 2,048 tokens, against 8,192 for the + 64-frame setting, and attention cost grows with the square of the token count. In one measurement + on an M3 Max, a 10-second clip embedded in about 3.5 seconds once the model was loaded. Use the + giant model only to confirm that a conclusion holds at scale. +4. **Rent GPUs only for batch jobs,** such as bulk embedding or `eb_jepa` training runs. Use spot + instances and scripts that tear the machine down when the job ends. +5. **`eb_jepa`'s small environments are the world-model lab.** They retrain in hours on one GPU, + with no dependence on the large checkpoints. + +Expected cost for the first six weeks is close to zero beyond a laptop, plus perhaps tens of dollars +of GPU time for `eb_jepa` runs. + +## Progress + +### Week 1 (2026-11-03 to 2026-11-05) + +| Step | Status | +| --- | --- | +| Run ViT-L on Apple Silicon and check the token count | Done. Shapes are pinned by tests | +| Rebuild the time, row and column grid from the flat tokens | Done. A test lights one pixel and checks which token changes | +| PCA colour overlay | Done | +| Surprise from masked prediction | Done. A cut in a spliced clip is detected; see below | +| `.latent` format and CLI | Done | +| Web viewer with overlay and timeline | Done | +| Run the `eb_jepa` Two Rooms example and draft the planning protocol | **Not started** | +| Publish under Apache-2.0 | Done: `ryangswe/jepa-scope`, with CI | + +The first local milestone was "open a clip on a MacBook without CUDA and see a PCA overlay and a +surprise timeline whose spikes match what happens in the video". It was met on a clip that splices +an archery scene onto a bowling scene. In one run, surprise stayed between about 0.57 and 0.62 +within each scene and rose to between 0.65 and 0.67 for the second after the cut. + +### What Week 1 taught + +- **Two silent errors in the surprise score.** The Hugging Face forward pass lets context tokens + attend to the frames being predicted. Comparing predictions with raw targets instead of + layer-normalised ones also produces a flat line. Both ran without any error. The details are in + `docs/decisions/method.md`. +- **Test media matters.** The first test clip was two near-identical static title cards, and it + showed nothing useful. Natural video with motion is the minimum for judging a signal. +- **Expect a floor.** Surprise never reaches zero, and what carries information is the change + relative to the clip's own floor. + +### Next + +Choosing the next milestone is the owner's call. These are the candidates: + +1. The remaining Week 1 step: run the `eb_jepa` Two Rooms example and draft the planning protocol. +2. jepa-scope Weeks 3 and 4: folder indexing, nearest-neighbour search and the latent + trajectory view. +3. jepa-scope Weeks 5 and 6: embedding health metrics, polish and a launch demo. + +Each milestone gets a plan in `docs/plans/` before code changes, written by the `planner` agent and +approved by the owner. + +## Open questions and risks + +- **Predictor training distribution.** V-JEPA 2's predictor was trained mostly on masks that span + the whole clip in time. Predicting the future from the past alone falls outside that training, + so surprise is a useful signal but not a calibrated one. V-JEPA 2-AC is trained for future + prediction and may be a better source later. +- **New weights.** AMI Labs has said it will open-source some of its work. We know of no + open-weights model from it as of November 2026. Targeting a protocol instead of one model is the + hedge. +- **Who the users are.** The near-term audience is likely robotics and video analytics rather than + consumer apps. That favours jepa-probe's labelling-to-API path over chat-style demos. +- **Licences.** jepa-scope downloads V-JEPA 2 weights at run time and does not redistribute them. + Meta releases V-JEPA 2 under MIT. + +## Sources + +- EB-JEPA: [arXiv 2602.03604](https://arxiv.org/abs/2602.03604) +- V-JEPA 2: [facebookresearch/vjepa2](https://github.com/facebookresearch/vjepa2) and the + [Transformers documentation](https://huggingface.co/docs/transformers/model_doc/vjepa2) +- Prior art for viewers: `mosure/burn_jepa` (Rust and Bevy) and `keon/jepa` (teaching code with + t-SNE and PCA plots) +- AMI Labs funding: [SiliconANGLE, 2026-03-10](https://siliconangle.com/2026/03/10/yann-lecuns-new-startup-ami-labs-raises-1-03b-train-world-models/)