Skip to content

Run eb_jepa Two Rooms and draft the planning protocol - #4

Merged
ryangswe merged 14 commits into
mainfrom
week1-two-rooms-protocol
Nov 6, 2026
Merged

ryangswe merged 14 commits into
mainfrom
week1-two-rooms-protocol

Conversation

@ryangswe

@ryangswe ryangswe commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

This closes Week 1. It runs eb_jepa's Two Rooms example and drafts the planning protocol that latent-plan will build on, checked against that model. The protocol is experimental: it has been checked against one model, and its interface stays provisional until a second model's adapter reproduces that model's native rollouts.

What changes

  • A draft planning protocol in src/jepa_scope/planning/. A world model implements encode, a one-step stateless predict and energy over a fixed-shape planning state (state_shape). rollout keeps a trajectory, and plan_cost scores one with a "sum" or "last" objective, keeping only the current state. cem.py is a reference Cross-Entropy Method planner that mirrors eb_jepa's CEMPlanner; it validates its settings and inputs and fails loudly on NaN or infinite costs. The package depends only on torch.
  • An adapter for eb_jepa's Two Rooms model (eb_jepa_two_rooms.py). eb_jepa is imported only when an adapter is built, and it is never a dependency. Weights load with weights_only=True and strict key matching.
  • The spec, docs/planning-protocol.md, for people writing adapters for their own models. It covers the interface, the rules every adapter follows, what an action must document, what the reference planner supports, the Two Rooms adapter, the results with where each number came from and how to reproduce it, what is not yet checked, and the open points (including how V-JEPA 2-AC would fit).
  • Tests. tests/test_planning_protocol.py (35 tests, in CI) covers the protocol and planner on a point-mass model and on a model whose state packs a history buffer, plus the planner's validation. tests/test_eb_jepa_two_rooms.py is opt-in (it needs eb_jepa and a trained run) and is skipped in CI.
  • Docs. The roadmap marks Week 1 done, with Weeks 3 and 4 next. Four decision entries record the protocol's shape, where it lives, the milestone order and a rule for cloud GPU spend, which CLAUDE.md also carries. The plan is closed with its completion summary.
  • A build fix. scripts/build.sh reported a wheel without the viewer about half the time on bash, although the viewer was there: under pipefail, piping unzip -l into grep -q fails when grep exits early. It now reads the listing first.

Results

The model was trained with eb_jepa's own script (one NVIDIA L40S, about 1.6 hours). eb_jepa's own CEM evaluation then reached the goal in 18 of 20 episodes.

On the tested CPU cases, the adapter reproduces eb_jepa's rollouts (difference 0.0) and costs (9e-8 relative), and given the same noise the reference CEM reproduces eb_jepa's plans (6e-7). All 20 replayed start frames match eb_jepa's evaluation. In a separate closed-loop check on a different device, planning through the protocol solved 16 of 20 episodes against 18 of 20 natively and passed a threshold fixed before the run. That rule is a net-count engineering threshold, and the result does not establish that the protocol loses no planning quality. A greedy policy solved 9 of 20 and an untrained model 0 of 10.

Reviews

  • A fresh-context reviewer passed the branch twice, and its minor findings are fixed.
  • A Fable architect reviewed the protocol's design before it was recorded; its recommended changes were approved and applied.
  • An outside reviewer found no Critical issues and five Major ones about the contract and the claims. All eleven findings were addressed: the state_shape rename, the adapter rules and their tests, step-by-step cost, input validation, the action contract, and the more careful wording of the results. Four checks it suggested are listed in the spec as not yet done.

Checks

  • uv run ruff check src tests, uv run ruff format --check src tests and uv run pytest -q pass (62 passed; the opt-in tests skip).
  • ./scripts/build.sh and python3 .claude/workflow/check.py pass.
  • The full opt-in test passed earlier (5 tests, 82 minutes, on the code at 00e6919). The short opt-in checks were rerun on the final planner code:
jepa-scope revision: 3ac29ff8bc686e66a62fff9d819117f48152c3e3
eb_jepa revision: 966e61e9285b3a876f49b9774e9720d9a99a7925
machine: Apple M3 Max, CPU, float32
torch: 2.6.0
checkpoint sha256: 9dc746586ba996590e13186cfc72bf76d45d25f038d99266ece59ce01eb13bb4
Parity: max |protocol rollout - eb_jepa unroll| = 0.00e+00
Cost: max relative difference = 8.95e-08
Same-noise CEM, steps_left=None: max |protocol plan - eb_jepa plan| = 5.96e-07
Same-noise CEM, steps_left=7: max |protocol plan - eb_jepa plan| = 0.00e+00
======================= 4 passed, 3 deselected in 8.79s ========================

A planted deviation (a biased standard deviation in the CEM's elite update) makes the same-noise check fail by 0.45 and 0.16.

…otocol)

Draft for the owner's approval. It trains eb_jepa's Two Rooms world model in a lab directory outside the repository, drafts the encode/predict/energy protocol with a reference CEM planner, and checks an adapter against eb_jepa's own model and planner on replayed episodes.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
The owner has a RunPod account and allowed cloud GPUs for modest spend. Training now runs on a pod by default, with the laptop as the fallback: a CUDA 12.4 image, direct SSH, a restart-safe venv on the volume, a watchdog on the pod and one on the laptop, and teardown checked by a fresh read. CLAUDE.md now treats only spend outside an approved plan's cap as owner-only.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Add a model-agnostic WorldModel protocol (encode, one-step stateless
predict, energy), rollout and plan_cost, and a Cross-Entropy Method
planner that mirrors eb_jepa's CEMPlanner so the two can be compared.
A point-mass test checks that CEM reaches a known goal, beats random
plans and is reproducible from a seed. The spec is in
docs/planning-protocol.md. Ignore *.pth.tar and *.gif as a backstop.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
…pisodes

The reference model, trained with eb_jepa's own script, plans to the
goal in 90% of 20 episodes with eb_jepa's native CEM. The protocol spec
now gives the reproduction details and compares them with eb_jepa's
published figure.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
EbJepaTwoRooms wraps eb_jepa's action-conditioned Two Rooms model:
encode normalises and encodes an observation, predict is one GRU step
with the shared LayerNorm, and energy is the mean squared latent
distance. It loads only encoder and predictor weights, with
weights_only=True and strict key matching.

The opt-in test checks that rollouts and plan costs match eb_jepa's own
unroll and objective, that replayed episodes match eb_jepa's evaluation,
and that planning through the protocol succeeds as often as eb_jepa's
native planner, against a greedy baseline and an untrained control.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
encode now accepts any observation type, and the spec treats z as the
model's planning state, which may hold more than the latent that energy
compares (earlier predicted frames or a robot pose, for example).
plan_cost and CEMConfig gain an objective setting, 'sum' by default or
'last', so a model can be compared with a native planner that scores
only the final state. The open points sketch how V-JEPA 2-AC would fit
and name a possible latent(z) operation.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Under set -o pipefail, piping unzip -l into grep -q fails about half the
time on bash: grep exits at its first match and unzip then dies writing
to the closed pipe, so the script reported a wheel without the viewer
when the viewer was there. Read the listing first, then search it.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Planning eb_jepa's Two Rooms model through the protocol succeeded in 16
of 20 episodes, against 18 of 20 for eb_jepa's own planner on the same
episodes, with exact rollout and cost parity. The spec now gives these
numbers, the baselines and the commands to reproduce them. The roadmap
marks Week 1 done, and the decisions record the protocol's shape, where
it lives, the milestone order and the cloud spending rule.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Add an opt-in test that feeds the reference CEM and eb_jepa's CEMPlanner
the same noise and requires the same plan, so the planner cannot drift
from eb_jepa's unnoticed. Correct the spec's description of where the
protocol belongs, the run folder and the normaliser, and add the native
evaluation's machine and time. Fold the execution log into the plan's
completion summary and mark the plan done.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Rename latent_shape to state_shape, since it describes the whole planning
state. plan_cost now adds energies one step at a time and keeps only the
current state, so its memory no longer grows with the horizon. The
protocol states that methods treat inputs as read-only and keep batch
rows independent, and a toy model whose state packs a history checks
that, along with refusing a plan longer than its capacity. cem_plan
rejects non-finite or non-integer settings, a start and goal that do not
match, and NaN or infinite costs, and samples actions in float32.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
State the rules every adapter follows, what an action must document,
what the reference planner supports, and that the interface stays
experimental until a second model's adapter reproduces its native
rollouts. Describe the closed-loop result as an engineering check that
does not establish equivalence, give the provenance of each number, and
list what is not yet checked. The Two Rooms adapter's predict now moves
its inputs to its device and dtype. The roadmap lesson, the method
decision and the plan's completion summary follow.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
The float32-action test now sets torch's default dtype to float64, so it
fails if the planner stops choosing its action dtype explicitly. The
partially invalid cost case now makes only some candidates infinite. The
replay test checks that pairing each replay with the next episode fails,
as a built-in negative control. The spec no longer implies a conformance
check for outside adapters exists, and the plan notes the revised
reading of its pass rule.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
…om CI

The Python 3.12 job was cancelled at its 20-minute limit while apt-get downloaded ffmpeg from a
slow Ubuntu mirror (one 13.6 MB package took five and a half minutes). The same step stalled the
3.11 job in an earlier run. The only use of ffmpeg was one test writing rotation metadata, and
PyAV 18's set_display_rotation writes the same display matrix (checked with ffprobe). The test
now runs everywhere instead of skipping without ffmpeg, the CI job no longer touches apt, and its
time limit drops to 10 minutes so a future hang fails sooner.

Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
@ryangswe
ryangswe merged commit e5a37ad into main Nov 6, 2026
4 checks passed
@ryangswe
ryangswe deleted the week1-two-rooms-protocol branch November 6, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant