R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation

CVPR Workshop on Autonomous Driving 2026

Project page · Paper · Setup · Usage · Citation

News

• September 2026: Following the update to the Waymo Dataset License permitting trained model releases, R3D2 and R3D2-big weights are now available for non-commercial use on Hugging Face 🤗.

About

This is the official code release for R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation. R3D2 is a lightweight, one-step diffusion model that inserts complete 3D assets into existing neural-rendered scenes by generating plausible rendering effects — shadows and consistent lighting — in real time.

Abstract

Validating autonomous driving (AD) systems requires diverse and safety-critical testing, making photorealistic virtual environments essential. Traditional simulation platforms, while controllable, are resource-intensive to scale and often suffer from a domain gap with real-world data. In contrast, neural reconstruction methods like 3D Gaussian Splatting (3DGS) offer a scalable solution for creating photorealistic digital twins of real-world driving scenes. However, they struggle with dynamic object manipulation and reusability as their per-scene optimization-based methodology tends to result in incomplete object models with integrated illumination effects. This paper introduces R3D2, a lightweight, one-step diffusion model designed to overcome these limitations and enable realistic insertion of complete 3D assets into existing scenes by generating plausible rendering effects—such as shadows and consistent lighting—in real time. This is achieved by training R3D2 on a novel dataset: 3DGS object assets are generated from in-the-wild AD data using an image-conditioned 3D generative model, and then synthetically placed into neural rendering-based virtual environments, allowing R3D2 to learn realistic integration. Quantitative and qualitative evaluations demonstrate that R3D2 significantly enhances the realism of inserted assets, enabling use-cases like text-to-3D asset insertion and cross-scene/dataset object transfer, allowing for true scalability in AD validation.

Setup

Dependencies are managed with [uv](https://github.com/astral-sh/uv) and locked in uv.lock.

# install uv if needed: pip install uv
uv sync

This installs the r3d2 package in editable mode and all dependencies into a local .venv. Base weights (stabilityai/sd-turbo and, for the tiny-VAE variants, madebyollin/taesd) are downloaded from the Hugging Face Hub on first run.

Dataset

Point a config at a directory containing a metadata.json with this structure:

{
  "root": "/absolute/path/to/render/root",
  "train": [
    {"seq": "scene-0001", "gt": "rel/path/gt/FRONT_00000001.jpg", "wo_shadow": "rel/path/pred/FRONT_00000001.jpg"}
  ],
  "val": [
    {"seq": "scene-0002", "gt": "...", "wo_shadow": "..."}
  ]
}

• gt is the ground-truth image; wo_shadow is the rendered prediction used as the model input.
• gt/wo_shadow paths are relative to the top-level root.
• By default only FRONT camera frames are used (cams_to_use=("FRONT",)); set --data.cams-to-use to change this, or pass None to use every frame. Filenames are expected to be <CAMERA>_<index>.

Usage

Scripts are configured with [tyro](https://github.com/brentyi/tyro). Three predefined configurations are available:

Config	VAE	Resolution	Notes
R3D2	tiny	1080×1920	Main model from the paper; fast inference.
R3D2-big	regular	1080×1920	Higher quality, slower.
R3D2-fast	tiny	512×512	Quick experiments / lower compute.

Training

uv run r3d2/scripts/train.py R3D2 --data.data-path /path/to/dataset

Any config field can be overridden on the command line, e.g.:

uv run r3d2/scripts/train.py R3D2-big --data.data-path /path/to/dataset --trainer.train-cfg.batch-size 2

Training logs to Weights & Biases. Set --trainer.train-cfg.wandb-entity and --trainer.train-cfg.project-name, or run wandb offline to disable uploads. Checkpoints and the exported pipeline are written under outputs/.

Validation

uv run r3d2/scripts/validate.py R3D2 --data.data-path /path/to/dataset \
    --trainer.train-cfg.resume-from-checkpoint /path/to/checkpoint

Inference

Download R3D2 and run on a single image after setup:

uv run huggingface-cli download bertaveira/R3D2 --local-dir checkpoints/R3D2
uv run r3d2/scripts/eval.py --model-path checkpoints/R3D2 --image-path input.jpg --output-path out.png

For R3D2-big, replace R3D2 with R3D2-big in both commands. You can also point --model-path at your own exported pipeline. The CLI uses FP32; see the model cards for standalone Diffusers instructions using FP16 to reduce GPU memory use.

Video (requires ffmpeg on the system path):

uv run r3d2/scripts/run_on_video.py --model-path outputs/<run>/model --video-path in.mp4 --output-path out.mp4

FPS benchmark

measure_fps.py benchmarks inference throughput (real-time speed is a core feature of R3D2):

uv run r3d2/scripts/measure_fps.py --model-path outputs/<run>/model
# or benchmark the sd-turbo baseline without a trained model:
uv run r3d2/scripts/measure_fps.py --no-r3d2 --tiny-vae

Optional acceleration via [stable-fast](https://github.com/chengzeyi/stable-fast) (--compile-type sfast) and Triton.

Citation

If you find this work useful, please consider citing:

@inproceedings{ljungbergh2025r3d2,
  title        = {R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation},
  author       = {Ljungbergh, William and Taveira, Bernardo and Zheng, Wenzhao and Tonderski, Adam and Peng, Chensheng and Kahl, Fredrik and Petersson, Christoffer and Felsberg, Michael and Keutzer, Kurt and Tomizuka, Masayoshi and Zhan, Wei},
  booktitle    = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  year         = {2026}
}

License

Code: Apache License 2.0. See LICENSE.

Pretrained weights are for non-commercial use only under the Waymo Dataset License Agreement; all downstream use and modification are subject to its terms. See the full licenses for R3D2 and R3D2-big.