WorldSculpt: Generating Compositional Worlds from Grounded Videos

Muyao Niu^1,2   Jixuan He¹   Ruihan Yu¹   Lian Fu¹   Yonghao Yu¹   Zheng-Hui Huang¹   Yifan Zhan¹   Fengbo Lan¹  

Yongtao Ge¹   Yinqiang Zheng²   Kaipeng Zhang^1,✉   Zhixiang Wang^1,✉  

¹ Alaya Lab   ² The University of Tokyo   ^✉ Corresponding Authors  

Given RGB images with instance masks and 3D boxes, WorldSculpt produces a compositional mesh representation for very complex scenes consisting of hundreds of individual objects.

✨ News

• Sep 14, 2026: 🔥🔥 ComfyUI integration is now available 🎨. Many thanks to Terry Jia (@jtydhr88) for the wonderful implementation!
• Sep 2, 2026: 🔥🔥 Released inference codes 📟, checkpoints 🤗, arXiv 📚, and project page 🏠.

TL;DR

• Single-object prior for complex compositional scenes. We demonstrate that a single-object generative prior can be leveraged for compositional meshification of very complex scenes with hundreds of objects.
• Adapting Single-Object Prior. We finetune Pixal3D to consume occluded multiple-view input.
• Benchmark. We release the UE-MeshyScene dataset with per-frame, per-instance annotations.
• Application to Marble. The same pipeline turns a Marble 3DGS world into object-level compositional meshes.

🧭 Pipeline

🚀 Getting Started

⚙️ Installation

Step 1: Follow the TRELLIS.2 Installation Guide

Please first follow the installation guide of TRELLIS.2 to set up the base environment.

Step 2: Installing Additional Dependencies

NATTEN_CUDA_ARCH is the compute capability of your GPU. Print yours with

python -c "import torch; print('%d.%d' % torch.cuda.get_device_capability())"

# H100; replace 9.0 with your own compute capability
NATTEN_CUDA_ARCH="9.0" NATTEN_N_WORKERS=8 pip install natten==0.21.0 --no-build-isolation
pip install https://github.com/LDYang694/Storages/releases/download/20260430/utils3d-0.0.2-py3-none-any.whl
pip install peft pillow imageio imageio-ffmpeg tqdm easydict opencv-python-headless trimesh transformers==4.57.1 zstandard kornia timm diffusers accelerate gradio plyfile matplotlib scikit-image scikit-learn fpsample iopath pycocotools ftfy
pip install "setuptools<81"

Step 3: Downloading Pretrained Checkpoints

hf download AlayaLab/WorldSculpt --local-dir ./pretrained
hf download TencentARC/Pixal3D --local-dir ./pretrained/Pixal3D

Step 4: Downloading Data

# UE-MeshyScene
hf download AlayaLab/WorldSculpt_data --include "UE-MeshyScene/*" --local-dir ./input --repo-type=dataset
cd ./input/UE-MeshyScene
tar xvf scene*.tar
cd ../..

# Marble DEMO data
hf download AlayaLab/WorldSculpt_data --include "Marble/*" --local-dir ./input --repo-type=dataset
cd ./input/Marble
tar xvf Marble.tar.gz
cd ../..

🧠 Running Inference

Every option is passed explicitly — the script has no defaults, so a run is fully described by its own command line. Step 4 extracts each dataset into its own subdirectory, so INPUT_ROOT points at that subdirectory, not at ./input (the script reads $INPUT_ROOT/<scene>/transforms.json).

# UE-MeshyScene, scene_00001
INPUT_ROOT=./input/UE-MeshyScene OUTPUT_ROOT=./output CKPT_ROOT=./pretrained \
SAMPLER=official SS_STEP=15000 SHAPE_STEP=15000 \
RENDER=1 FACE_BUDGET=1000000 GPU=0 \
./inference.sh scene_00001

# Marble demo data, marble_serene_living_room_countryside_view
INPUT_ROOT=./input/Marble OUTPUT_ROOT=./output CKPT_ROOT=./pretrained \
SAMPLER=official SS_STEP=15000 SHAPE_STEP=15000 \
RENDER=1 FACE_BUDGET=1000000 GPU=0 \
./inference.sh marble_serene_living_room_countryside_view

🎨 ComfyUI Integration

A community ComfyUI implementation of WorldSculpt is available at ComfyUI-WorldSculpt, developed by Terry Jia (@jtydhr88).

It provides node-based workflows for running WorldSculpt within ComfyUI. Please refer to the repository for installation instructions, example workflows, and the latest usage details.

🙏 Acknowledgements

We thank Pixal3D, TRELLIS.2, and DINOv3 for their wonderful work and open-source repositories.

We also thank Terry Jia for developing and open-sourcing the community ComfyUI integration for WorldSculpt.

📖 Citation

If you find this work useful, please cite:

@misc{niu2026worldsculptgeneratingcompositionalworlds,
      title={WorldSculpt: Generating Compositional Worlds from Grounded Videos}, 
      author={Muyao Niu and Jixuan He and Ruihan Yu and Lian Fu and Yonghao Yu and Zheng-Hui Huang and Yifan Zhan and Fengbo Lan and Yongtao Ge and Yinqiang Zheng and Kaipeng Zhang and Zhixiang Wang},
      year={2026},
      eprint={2609.05416},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.05416}, 
}