Repository navigation
Run eb_jepa Two Rooms and draft the planning protocol - #4
Merged
Merged
Conversation
…otocol) Draft for the owner's approval. It trains eb_jepa's Two Rooms world model in a lab directory outside the repository, drafts the encode/predict/energy protocol with a reference CEM planner, and checks an adapter against eb_jepa's own model and planner on replayed episodes. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
The owner has a RunPod account and allowed cloud GPUs for modest spend. Training now runs on a pod by default, with the laptop as the fallback: a CUDA 12.4 image, direct SSH, a restart-safe venv on the volume, a watchdog on the pod and one on the laptop, and teardown checked by a fresh read. CLAUDE.md now treats only spend outside an approved plan's cap as owner-only. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Add a model-agnostic WorldModel protocol (encode, one-step stateless predict, energy), rollout and plan_cost, and a Cross-Entropy Method planner that mirrors eb_jepa's CEMPlanner so the two can be compared. A point-mass test checks that CEM reaches a known goal, beats random plans and is reproducible from a seed. The spec is in docs/planning-protocol.md. Ignore *.pth.tar and *.gif as a backstop. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
…pisodes The reference model, trained with eb_jepa's own script, plans to the goal in 90% of 20 episodes with eb_jepa's native CEM. The protocol spec now gives the reproduction details and compares them with eb_jepa's published figure. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
EbJepaTwoRooms wraps eb_jepa's action-conditioned Two Rooms model: encode normalises and encodes an observation, predict is one GRU step with the shared LayerNorm, and energy is the mean squared latent distance. It loads only encoder and predictor weights, with weights_only=True and strict key matching. The opt-in test checks that rollouts and plan costs match eb_jepa's own unroll and objective, that replayed episodes match eb_jepa's evaluation, and that planning through the protocol succeeds as often as eb_jepa's native planner, against a greedy baseline and an untrained control. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
encode now accepts any observation type, and the spec treats z as the model's planning state, which may hold more than the latent that energy compares (earlier predicted frames or a robot pose, for example). plan_cost and CEMConfig gain an objective setting, 'sum' by default or 'last', so a model can be compared with a native planner that scores only the final state. The open points sketch how V-JEPA 2-AC would fit and name a possible latent(z) operation. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Under set -o pipefail, piping unzip -l into grep -q fails about half the time on bash: grep exits at its first match and unzip then dies writing to the closed pipe, so the script reported a wheel without the viewer when the viewer was there. Read the listing first, then search it. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Planning eb_jepa's Two Rooms model through the protocol succeeded in 16 of 20 episodes, against 18 of 20 for eb_jepa's own planner on the same episodes, with exact rollout and cost parity. The spec now gives these numbers, the baselines and the commands to reproduce them. The roadmap marks Week 1 done, and the decisions record the protocol's shape, where it lives, the milestone order and the cloud spending rule. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Add an opt-in test that feeds the reference CEM and eb_jepa's CEMPlanner the same noise and requires the same plan, so the planner cannot drift from eb_jepa's unnoticed. Correct the spec's description of where the protocol belongs, the run folder and the normaliser, and add the native evaluation's machine and time. Fold the execution log into the plan's completion summary and mark the plan done. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Rename latent_shape to state_shape, since it describes the whole planning state. plan_cost now adds energies one step at a time and keeps only the current state, so its memory no longer grows with the horizon. The protocol states that methods treat inputs as read-only and keep batch rows independent, and a toy model whose state packs a history checks that, along with refusing a plan longer than its capacity. cem_plan rejects non-finite or non-integer settings, a start and goal that do not match, and NaN or infinite costs, and samples actions in float32. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
State the rules every adapter follows, what an action must document, what the reference planner supports, and that the interface stays experimental until a second model's adapter reproduces its native rollouts. Describe the closed-loop result as an engineering check that does not establish equivalence, give the provenance of each number, and list what is not yet checked. The Two Rooms adapter's predict now moves its inputs to its device and dtype. The roadmap lesson, the method decision and the plan's completion summary follow. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
The float32-action test now sets torch's default dtype to float64, so it fails if the planner stops choosing its action dtype explicitly. The partially invalid cost case now makes only some candidates infinite. The replay test checks that pairing each replay with the next episode fails, as a built-in negative control. The spec no longer implies a conformance check for outside adapters exists, and the plan notes the revised reading of its pass rule. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
…om CI The Python 3.12 job was cancelled at its 20-minute limit while apt-get downloaded ffmpeg from a slow Ubuntu mirror (one 13.6 MB package took five and a half minutes). The same step stalled the 3.11 job in an earlier run. The only use of ffmpeg was one test writing rotation metadata, and PyAV 18's set_display_rotation writes the same display matrix (checked with ffprobe). The test now runs everywhere instead of skipping without ffmpeg, the CI job no longer touches apt, and its time limit drops to 10 minutes so a future hang fails sooner. Signed-off-by: RDCLder <raymondyang.dev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This closes Week 1. It runs eb_jepa's Two Rooms example and drafts the planning protocol that latent-plan will build on, checked against that model. The protocol is experimental: it has been checked against one model, and its interface stays provisional until a second model's adapter reproduces that model's native rollouts.
What changes
src/jepa_scope/planning/. A world model implementsencode, a one-step statelesspredictandenergyover a fixed-shape planning state (state_shape).rolloutkeeps a trajectory, andplan_costscores one with a"sum"or"last"objective, keeping only the current state.cem.pyis a reference Cross-Entropy Method planner that mirrors eb_jepa'sCEMPlanner; it validates its settings and inputs and fails loudly on NaN or infinite costs. The package depends only on torch.eb_jepa_two_rooms.py). eb_jepa is imported only when an adapter is built, and it is never a dependency. Weights load withweights_only=Trueand strict key matching.docs/planning-protocol.md, for people writing adapters for their own models. It covers the interface, the rules every adapter follows, what an action must document, what the reference planner supports, the Two Rooms adapter, the results with where each number came from and how to reproduce it, what is not yet checked, and the open points (including how V-JEPA 2-AC would fit).tests/test_planning_protocol.py(35 tests, in CI) covers the protocol and planner on a point-mass model and on a model whose state packs a history buffer, plus the planner's validation.tests/test_eb_jepa_two_rooms.pyis opt-in (it needs eb_jepa and a trained run) and is skipped in CI.CLAUDE.mdalso carries. The plan is closed with its completion summary.scripts/build.shreported a wheel without the viewer about half the time on bash, although the viewer was there: underpipefail, pipingunzip -lintogrep -qfails when grep exits early. It now reads the listing first.Results
The model was trained with eb_jepa's own script (one NVIDIA L40S, about 1.6 hours). eb_jepa's own CEM evaluation then reached the goal in 18 of 20 episodes.
On the tested CPU cases, the adapter reproduces eb_jepa's rollouts (difference 0.0) and costs (9e-8 relative), and given the same noise the reference CEM reproduces eb_jepa's plans (6e-7). All 20 replayed start frames match eb_jepa's evaluation. In a separate closed-loop check on a different device, planning through the protocol solved 16 of 20 episodes against 18 of 20 natively and passed a threshold fixed before the run. That rule is a net-count engineering threshold, and the result does not establish that the protocol loses no planning quality. A greedy policy solved 9 of 20 and an untrained model 0 of 10.
Reviews
state_shaperename, the adapter rules and their tests, step-by-step cost, input validation, the action contract, and the more careful wording of the results. Four checks it suggested are listed in the spec as not yet done.Checks
uv run ruff check src tests,uv run ruff format --check src testsanduv run pytest -qpass (62 passed; the opt-in tests skip)../scripts/build.shandpython3 .claude/workflow/check.pypass.A planted deviation (a biased standard deviation in the CEM's elite update) makes the same-noise check fail by 0.45 and 0.16.