egoallo for iPhone 📱

This repository is an AECTech London 2026 fork/adaptation of brentyi/egoallo, with a focus on making inference easier on iPhone LiDAR captures via Record3D.

Overview

We use egocentric SLAM poses and images to estimate 3D human body pose, height, and hands.

egoallo_coffee.mov

This repository is structured as follows:

.
├── download_checkpoint_and_data.sh
│                            - Download model checkpoint and sample data.
├── 0a_preprocess_training_data.py
│                            - Stage 1 preprocessing script for training datasets.
├── 0b_preprocess_training_data.py
│                            - Stage 2 preprocessing script (NPZ -> HDF5).
├── 1_train_motion_prior.py
│                            - Training script for motion diffusion model.
├── 2_run_hamer_on_vrs.py
│                            - Run HaMeR on Aria VRS trajectories.
├── 3_aria_inference.py
│                            - Run full pipeline on inference data.
├── 3_record3d_inference.py
│                            - Run full pipeline on Record3D iPhone LiDAR data.
├── 4_visualize_outputs.py
│                            - Visualize outputs from inference.
├── 5_eval_body_metrics.py
│                            - Compute and print body estimation accuracy metrics.
├── environment_files/
│   ├── environment.yml      - Conda environment for reproducible setup.
│   └── spec-file.txt        - Explicit conda package list.
│
├── src/egoallo/
│   ├── data/                - Dataset utilities.
│   ├── transforms/          - SO(3) / SE(3) transformation helpers.
│   └── *.py                 - All core implementation.
│
└── pyproject.toml          - Python dependencies/package metadata.

Getting started

EgoAllo requires Python 3.12 or newer.

We recommend using a Conda environment (recommended for this fork on Windows/NVIDIA systems):

conda env create -f environment_files/environment.yml
conda activate egoallo
pip install -e .

1. Clone the repository.

   git clone https://github.com/modellingpeople/egoallo.git

2. Download+unzip model checkpoint and sample data.

   Here are links to the checkpoint and example trajectories.

3. Download the SMPL-H model file.

   You can find the "Extended SMPL+H model" (16 shape parameters) from the MANO project webpage. Our scripts assumes an npz file located at ./data/smplh/neutral/model.npz, but this can be overridden at the command-line (--smplh-npz-path {your path}).

4. Visualize model outputs.

   The example trajectories directory includes example outputs from our model. You can visualize them with:

   python 4_visualize_outputs.py --search-root-dir ./egoallo_example_trajectories

Running inference

1. Installing inference dependencies.

   Our guidance optimization uses a Levenberg-Marquardt optimizer that's implemented in JAX. If you want to run this on an NVIDIA GPU, you'll need to install JAX with CUDA support:

   # Also see: https://jax.readthedocs.io/en/latest/installation.html
   pip install "jax[cuda12]==0.6.1"

   You'll also need jaxls:

   pip install git+https://github.com/brentyi/jaxls.git

2. Running inference on example data.

   Here's an example command for running EgoAllo on the "coffeemachine" sequence:

   python 3_aria_inference.py --traj-root ./egoallo_example_trajectories/coffeemachine

   You can run python 3_aria_inference.py --help to see the full list of options.

3. Running inference on your own data.

   To run inference on your own data, you can copy the structure of the example trajectories. The key files are:

   • A VRS file from Project Aria, which contains calibrations and images.
   • SLAM outputs from Project Aria's MPS: closed_loop_trajectory.csv and semidense_points.csv.gz.
   • (optional) HaMeR outputs, which we save to a hamer_outputs.pkl.
   • (optional) Project Aria wrist and palm tracking outputs.

4. Running HaMeR on your own data.

   To generate the hamer_outputs.pkl file, you'll need to install hamer_helper.

   Then, as an example for running on our coffeemachine sequence:

   python 2_run_hamer_on_vrs.py --traj-root ./egoallo_example_trajectories/coffeemachine

Record3D / iPhone Inference

This fork adds 3_record3d_inference.py, which converts a Record3D .r3d capture (iPhone with LiDAR) into EgoAllo-compatible trajectory inputs, then optionally runs full diffusion inference.

1. Install extra dependency for depth decoding.

   pip install pyliblzfse

2. Optional: install HaMeR dependencies for hand-guided Record3D inference.

   pip install hamer_helper

   --guidance-mode no_hands does not require HaMeR dependencies and remains the default behavior.

3. Convert only (no model inference yet).

   python 3_record3d_inference.py --r3d-path recording.r3d --no-run-inference

4. Run full conversion + inference.

   python 3_record3d_inference.py --r3d-path recording.r3d

5. Common options you will likely tweak.

   • --traj-length 256: increase/decrease trajectory length.
   • --guidance-mode no_hands|hamer_wrist|hamer_reproj2: choose guidance mode.
   • --visualize-traj: open viser visualization after inference.
   • --cpf-offset-y / --cpf-offset-z: adjust camera-to-CPF offset.

When --guidance-mode is hamer_wrist or hamer_reproj2, the script will:

• run HaMeR on sampled Record3D RGB frames,
• save detections to <output_dir>/hamer_outputs.pkl,
• save debug renders to <output_dir>/hamer_outputs_render/,
• reuse hamer_outputs.pkl if it already exists.

hamer_wrist uses only HaMeR's 3D wrist pose estimates to guide the optimizer. hamer_reproj2 adds a reprojection cost so hand poses better align with the image; it is typically more accurate but slower.

If HaMeR dependencies are missing and a HaMeR guidance mode is requested, the job fails early with a clear dependency error.

CPF is the "central pupil frame" proxy used by EgoAllo for egocentric head pose. Defaults in this fork are good starting points for handheld iPhone captures, but if the body appears vertically shifted, adjust --cpf-offset-y first.

Preprocessing Training Data

To train the motion prior model, we use data from the AMASS dataset. Due to licensing constraints, we cannot redistribute the preprocessed data. Instead, we provide two sequential preprocessing scripts:

1. Download the AMASS dataset.

   Download the AMASS dataset from the official website. We use the following splits:

   • Training: ACCAD, BioMotionLab_NTroje, BMLhandball, BMLmovi, CMU, DanceDB, DFaust_67EKUT, Eyes_Japan_Dataset, KIT, MPI_Limits, TCD_handMocap, TotalCapture
   • Validation: HumanEva, MPI_HDM05, SFU, MPI_mosh
   • Testing: Transitions_mocap, SSM_synced

2. Run the first preprocessing script.

   python 0a_preprocess_training_data.py --help
   python 0a_preprocess_training_data.py --data-root /path/to/amass --smplh-root ./data/smplh

   This script, adapted from HuMoR, processes raw AMASS data by:

   • Converting to gender-neutral SMPL-H parameters
   • Computing contact labels for feet, hands, and knees
   • Filtering out problematic sequences (treadmill walking, sequences with foot skating)
   • Downsampling to 30fps

3. Run the second preprocessing script.

   python 0b_preprocess_training_data.py --help
   python 0b_preprocess_training_data.py --data-npz-dir ./data/processed_30fps_no_skating/

   This converts the processed NPZ files to a unified HDF5 format for more efficient training, with optimized chunk sizes for reading sequences.

Status

This repository currently contains:

• egoallo package, which contains reference training and sampling implementation details.
• Training script.
• Model checkpoints.
• Dataset preprocessing script.
• Inference scripts for Project Aria and iPhone Record3D.
• Visualization script.
• Setup instructions.

While we've put effort into cleaning up our code for release, this is research code and there's room for improvement. If you have questions or comments, please reach out!