Repository navigation
Conversation
… command `recipes.droid` used the state of a frame as its action label. Two problems came from this: - The arm label was the measured pose and joints of the same frame. The arm reaches a target some time after the command, so a model trained on these labels commands a pose that the arm already has. At inference the arm then moves slower than in the demonstrations. - The gripper label was the measured `grip`. When the fingers stop on an object, the measured value stays low, so the label never shows a closed gripper. Now the arm label of a frame is the measured pose and joints `action_lead_sec` later (default 0.14 s). Past the end of the episode the last state holds. The gripper label is `target_grip`, thresholded at 0.5 (`binarize_grip`, default on). The recorded arm command is not used, because a recording commands poses or joints, not both. The default lead comes from a replay of MolmoSpaces teleop recordings through the eval's arm controller: joint targets 2 steps ahead of the recorded state gave the smallest tracking error (0.002 rad RMS, against 0.009-0.012 rad with no lead). MolmoSpaces steps are 66 ms. The default of 0.14 s gives 2 steps both there and at 15 Hz, where 2/15 s in nanoseconds falls 1 ns short of the second step. `droid_three_cameras` gets the same labels. Only the training labels change. Serving and the recorded model settings do not change. Ticket: none
A frame before the first `target_grip` sample has no gripper label, so the converted episode starts at the first command. The action codecs of the other vendors do the same. In headset teleop recordings the first command comes 1-4 steps after the first state. A test pins this start. The codecs README now names the GR00T labels.
… command The trajectory codecs label actions with state, so they start at the first state. The start test now gives one `target_grip` sample, so the labels after it come from the held command.
dmatov
marked this pull request as ready for review
October 27, 2026 14:30
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a7058c278e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The lead went on the pose and the joints one at a time, before they were joined. When the two signals have different timestamps, each part of a label came from its own last sample plus the lead. With a pose at 0 and 130 ms, a joint sample at 50 ms and a lead of 100 ms, the label at 50 ms took the pose of 0 ms. Now the recipe joins the state columns and the gripper command first, then takes the arm parts from that joined signal one lead later. The gripper part stays at the frame. A test gives this case. The threshold of the gripper command goes inline at its one use.
`droid` labels each frame with its own state and the raw `target_grip`. `droid_molmo` and `droid_molmo_three_cameras` set the 0.14 s lead and the 0.5 grip threshold measured on MolmoSpaces.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
recipes.droiduses the state of a frame as its action label. The labels have two problems:grip. When the fingers stop on an object, the measured value stays low. In three MolmoSpaces teleop recordings with a grasp, the measured grip never goes above 0.57-0.81. So the label never shows a closed gripper.This PR changes the labels of the GR00T recipes:
target_gripin every recipe.binarize_grip=Truethresholds it at 0.5.action_lead_seclater. The recipe joins the state columns and the gripper command first, then takes the arm parts from the joined signal one lead later (tf.TimeOffsets). Past the end of the episode, the last state holds.action_lead_secbinarize_gripdroiddroid_three_camerasdroid_molmodroid_molmo_three_camerasdroidwith a lead of 0 keeps the arm labels ofmain, so a rig without a measured lead keeps today's arm labels. A preset for our Franka can follow once its lead is measured, for example from command and state in phail recordings. Checkpoints of the_molmopresets serve withdroidanddroid_three_cameras.The recorded arm command is not used, because a recording commands poses or joints, not both: teleop records
robot_command.pose, a joint-space policy recordsrobot_command.joints.The labels change. Serving and the recorded model settings do not change. A frame before the first
target_gripsample has no gripper label, so the converted episode starts at the first command, as withabsolute_pos_action. Before, a GR00T episode started at the first state. In headset teleop recordings the first command comes 1-4 steps after the first state, so those episodes lose 1-4 leading frames.The MolmoSpaces lead
A replay of MolmoSpaces teleop recordings through the eval's arm controller sent the recorded joints
ksteps ahead as joint targets.k = 2gave the smallest joint tracking error:(RMS, rad.) The teleop command also leads the measured state by 2 steps. MolmoSpaces steps are 66 ms. The preset's 0.14 s gives 2 steps both there and at 15 Hz. 2/15 s in nanoseconds falls 1 ns short of the second 15 Hz step, so it gives 1 step there.
The lead is measured on the MolmoSpaces controller only. A rig with another controller needs its own preset.
Verification
uv run --locked pytest -q positronic/vendors/gr00t(at 8686a19, after the merge ofmain): 97 passed, 1 skipped. The four recipes resolve with the cameras and the label values of the table above. The 2-steps-ahead test and the training-dataset test run ondroid_molmoanddroid_molmo_three_cameras.test_codecs.py: the arm labels are the state 2 steps ahead at 66 ms and at 66.67 ms steps, and the last state holds; the gripper label istarget_grip, binary or raw, and not the measured grip; the actions start at the firsttarget_gripsample; with pose and joint samples at different times, every arm part of a label comes from one time ahead. The tests that expectedaction == statenow expect the new labels. Onmaincode, 7 of these tests fail.uv run --locked pytest -q -n 4(with thelerobot-0-3-3extra, at 290b5ae): 5479 passed, 31 skipped.lerobot_0_3_3.to_lerobot convertat 290b5ae, whendroid_three_camerashad the values ofdroid_molmo_three_camerasnow, on 3 MolmoSpaces teleop recordings (1177 frames, 15 fps, v2.1): exit 0. In 98% of frames, the pose and joint labels equal the state 2 frames later. The other frames come from the resample of 66 ms steps to 15 fps. The gripper labels are 0 and 1 only. The recordings predateee_frames, so for this check I addedee_framesat thegrippersite and that site to the URDF, as a recording onmainhas.Known limits
lerobot_0_3_3.to_lerobot appendcompares only the model settings and the GR00T modality.action_lead_secandbinarize_gripare in neither. So an append of new episodes to a dataset converted before this change mixes the two label conventions with no error. Convert such a dataset again.Not measured
No training run compares these labels with the old labels on one dataset. A GR00T fine-tune with these labels and the base DROID normalization statistics did better than the base model on MolmoSpaces pick and place. That run also used other data, so it does not isolate this change.
A human must review this change before it merges.