Adapting and evaluating Vision-Language-Action models on real robot demonstrations.
EmbodiedVLA-Lab is a compact robotics ML engineering study covering the full VLA lifecycle: real-robot demonstration loading, SmolVLA adaptation, held-out action prediction, failure analysis, and reproducible inference.
| Robot data | 50 real SO-100 pick-and-place episodes, 19,631 frames at 30 fps |
| Split | 40 episodes for training, 10 held out for evaluation |
| Model | lerobot/smolvla_base, 450M parameters, fine-tuned for 2,000 steps |
| Result | Held-out action MAE 5.876 → 4.419 between step 500 and step 2,000 |
| Evaluation | 377 held-out observations across 10 unseen episodes |
| Serving | FastAPI action-chunk endpoint, ~255 ms p50 latency on an RTX 4090 |
Every 10th frame of each held-out episode is passed to the policy as a robot would see it — two camera views, the joint state, and the task instruction — and the predicted 50-step action chunk is compared against the teleoperator's actions. Action units are raw SO-100 joint values.
| Checkpoint | Action MAE ↓ | RMSE ↓ | Chunk cosine ↑ | p50 latency |
|---|---|---|---|---|
| step 500 | 5.876 | 8.913 | 0.9894 | 254 ms |
| step 1,000 | 4.965 | 7.992 | 0.9917 | 251 ms |
| step 1,500 | 4.659 | 7.574 | 0.9926 | 265 ms |
| step 2,000 | 4.419 | 7.387 | 0.9929 | 255 ms |
Action prediction improves consistently across evaluated checkpoints on the held-out episodes, a 24.8% MAE reduction from the first evaluated checkpoint to the last, with no sign of overfitting within this budget.
Raw pretrained MAE is omitted because the base checkpoint and the target SO-100 dataset use non-comparable action normalization; see Evaluation scope.
Aggregate improvement does not mean that all parts of the manipulation trajectory are equally difficult. Breaking the same predictions down by joint, by trajectory phase, along the prediction horizon and by episode separates what the model learned from what it did not.
Error is highest during early reaching and lowest late in the trajectory. Mean MAE is 5.45 in the early third of an episode, 4.24 in the middle and 2.88 in the late third — a 1.89× spread. This runs against the intuition that the grasp is the hard part.
The gripper is the hardest dimension. Normalized by each joint's own spread, error is
highest on main_gripper (0.299) and lowest on main_shoulder_pan (0.158). The gripper
command is close to binary, with a sharp open/close transition.
Error grows across the prediction horizon, from 1.83 at the first chunk step to 7.48 at the 50th. Expected for open-loop chunking: every step is predicted from one observation up to 1.67 s earlier, with no feedback.
Episodes vary more than checkpoints do. The worst held-out episode (39, MAE 5.21) is 1.5× the best (20, MAE 3.47) — a wider gap than between the first and last checkpoint, and the dominant source of uncertainty across 10 validation episodes.
The trajectory plots below show predicted (dashed) against demonstrated (solid) chunks at the highest-error moment of the three worst episodes. Look at the amplitude: where the demonstration sweeps through a large reach, the prediction stays comparatively flat, and the gripper channel holds near zero while the demonstration steps to its closed value.
Together with the phase breakdown, this pattern suggests that perception-conditioned reaching — resolving where the cube is before the approach — may be a more useful target for improvement than the grasp itself. That is a hypothesis this offline setup cannot settle; testing it calls for multi-seed runs and closed-loop evaluation.
Per-observation errors are in artifacts/error_analysis.csv, one row per evaluated frame
broken down by action dimension and trajectory phase.
The final checkpoint re-evaluated under three synthetic corruptions:
| Perturbation | Action MAE | vs clean |
|---|---|---|
| clean | 4.251 | — |
| brightness −0.2 | 4.322 | +1.7% |
| brightness +0.2 | 4.240 | −0.3% |
| Gaussian blur σ=1 | 5.298 | +24.6% |
| Gaussian blur σ=3 | 5.766 | +35.6% |
| 20% center occlusion | 5.726 | +34.7% |
Brightness shifts barely move the error, consistent with the frozen SigLIP encoder's normalization. Blur and occlusion cost 25–36%, and blur reaches most of its damage already at σ=1 — camera focus and lens obstruction would deserve more attention than lighting in a deployment. The clean baseline here is 4.251 rather than 4.419 because the sweep runs on a cheaper subsample (5 episodes at stride 20), so rows are comparable within this table only.
lerobot/svla_so100_pickplace —
real SO-100 arm teleoperation in LeRobotDataset v3.0 format.
| Episodes / frames | 50 / 19,631 |
| Split | 40 train / 10 held out (seed 42) → 15,911 / 3,720 frames |
| Episode length | 326–575 frames (10.9–19.2 s, mean 13.1 s) |
| Cameras | top and wrist, 480×640 RGB, AV1 |
| State / action | 6-D each: shoulder pan/lift, elbow flex, wrist flex/roll, gripper |
| Task | "Pick up the cube and place it in the box." |
One sample as the policy consumes it — language, two camera views and proprioceptive state in, a 50-step ground-truth action chunk out:
lerobot/smolvla_base is a 450M-parameter
SmolVLM2 vision-language backbone with a flow-matching action expert. One forward pass
emits 50 future actions, 1.67 s of control at 30 fps. Fine-tuning follows the SmolVLA
defaults: the vision encoder stays frozen and the action expert is trained.
| Training | |
|---|---|
| Steps / batch size | 2,000 / 8 |
| Learning rate | 1e-4, 100 warm-up steps, decayed over 2,000 |
| Precision | bfloat16 autocast |
| Hardware | RTX 4090 (24 GB) |
| Wall clock | 6 min 18 s |
| Training loss | 0.764 → 0.121 (min 0.092) |
The pipeline is a sequence of commands, each writing artifacts the next one reads:
| Stage | Command | Output |
|---|---|---|
| Inspect + split | evla-inspect |
artifacts/dataset/, artifacts/splits/ |
| Fine-tune | evla-train |
outputs/<profile>/checkpoints/, artifacts/training/ |
| Evaluate | evla-evaluate |
artifacts/metrics/evaluation.{json,csv} |
| Error analysis | evla-error-analysis |
artifacts/error_analysis.csv, figures |
| Robustness | evla-robustness |
artifacts/robustness.csv |
| Report | evla-report |
assets/results/, artifacts/provenance.json |
| Serve | evla-infer, uvicorn embodied_vla.api:app |
action chunks over HTTP |
Four decisions shape what the numbers mean:
- Episode-level splitting. At 30 Hz adjacent frames are near-duplicates, so the split is over whole episodes and written to manifests that every later stage reads.
- Padding is masked. LeRobot pads action chunks past the end of an episode by repeating the final frame; every metric ignores those steps rather than scoring against them. In this evaluation 6.6% of chunk steps are padding.
- Schedule fitting. SmolVLA's shipped 1,000-step warm-up and 30,000-step decay are rescaled to the requested step budget. The effective training configuration and run provenance are captured with each experiment.
- Configuration over code. Every dataset id, model id, revision, split rule and
hyper-parameter lives in
configs/*.yaml, so retargeting a different LeRobot-format dataset is a config change.
Training delegates to the official LeRobot trainer rather than reimplementing it, and platform-specific compatibility handling is isolated from upstream source. Details in docs/implementation.md and docs/platform_notes.md.
A trained checkpoint is served behind a typed HTTP interface: one observation in, one action chunk out.
| Endpoint | Purpose |
|---|---|
GET /healthz |
Liveness; answers 200 with a reason even when the model failed to load |
GET /model-info |
Served checkpoint, device, chunk size, expected cameras and state dim |
POST /v1/predict |
One observation → one action chunk |
latency_ms is measured per request and varies call to call. Measured inference
performance: 254.5 ms p50 on an RTX 4090 for the held-out evaluation workload (372 timed
calls at batch size 1, covering preprocess → forward → un-normalize).
The same loading path backs a dataset-replay mode, so inference can be exercised without robot hardware:
uv run python -m embodied_vla.inference --episode 39 --frame 100Instruction : "Pick up the cube and place it in the box."
Episode 39, frame 100 | pretrained_model on cuda
Chunk shape : (50, 6) (chunk_size x action_dim)
MAE vs demo : 3.4757
The Docker image carries code and a frozen dependency set; datasets, checkpoints, the Hugging Face cache and credentials are mounted at run time.
Requires Python 3.12.x, uv, and a CUDA GPU for training.
uv sync --extra devscripts/inspect_data.sh # dataset summary, statistics, split manifests
scripts/train_smoke.sh # ~1 min, validates the pipeline end to end
scripts/train_portfolio.sh # ~6 min on an RTX 4090, the reported run
scripts/evaluate.sh # evaluation, error analysis, robustness, report
scripts/serve.sh # inference API on :8000
uv run pytest # 176 tests, no GPU and no dataset downloaddocker build -t evla:latest . && docker run --rm -p 8000:8000 -v "$PWD/outputs:/app/outputs:ro" evla:latest| Model | lerobot/smolvla_base @ c83c3163b8ca9b7e67c509fffd9121e66cb96205 |
| Dataset | lerobot/svla_so100_pickplace @ 728583b5eaf9e739a7f119e2def466fa1d552402 |
| Split | 40 train / 10 held out, seed 42, written to artifacts/splits/ |
| Seeds | split 42, training 1000, evaluation 42 |
| Dependencies | uv.lock, Python 3.12.7, PyTorch 2.11.0+cu128, LeRobot 0.6.1 |
| Provenance | artifacts/provenance.json |
Model and dataset snapshots, package versions, episode splits and runtime configuration are pinned, and the revisions are consumed by the loaders rather than only declared in YAML. Full mechanics, Docker usage and the environment record are in docs/reproducibility.md.
Metrics here are offline, open-loop action-prediction metrics: how closely the policy reproduces recorded demonstrations on episodes it never saw during training. They are not closed-loop robot task-success rates — the policy never acts on a robot in this study, so compounding error, recovery and contact dynamics are untested.
Raw pretrained SmolVLA MAE is not reported. The base checkpoint un-normalizes actions with statistics from its pretraining mixture rather than SO-100 joint ranges, so its error would measure a normalization mismatch rather than policy quality. The checkpoint progression serves the comparison instead, since all checkpoints share the dataset's normalization.
Scope is one public dataset, one task, one seed and 10 held-out episodes, with 2,000 training steps — roughly one pass over the training frames, enough to adapt the action expert and well short of convergence. The perturbation sweep uses synthetic corruptions rather than real lighting or occlusion shifts.
Natural next steps: closed-loop evaluation in simulation to measure task success rather than action similarity; multi-seed runs with confidence intervals so checkpoint differences become testable; and targeted diagnostics for the early-reaching phase identified above.
configs/ smoke / portfolio / full experiment profiles
src/embodied_vla/ data, training, evaluation, error analysis, inference, API
scripts/ one entry point per pipeline stage
tests/ 176 tests, no GPU or dataset required
artifacts/ generated metrics, splits, run records, provenance
assets/ architecture figure and generated result figures
docs/ reproducibility, implementation and platform notes
Built on LeRobot and
SmolVLA; no LeRobot source is vendored.
Robot data from lerobot/svla_so100_pickplace,
distributed under Apache-2.0 according to its dataset card and downloaded from the Hub at
run time rather than redistributed here. Code in this repository is MIT licensed
(see LICENSE); the dataset's license governs that data and does not change it.