Grow with the Flow: 4D Reconstruction of Growing Plants with Gaussian Flow Fields

TL;DR: 4D reconstruction of plant growth from multi-view timelapse measurements using neural ODE.

Weihan Luo, Lily Goli, Sherwin Bahmani, Felix Taubner, Andrea Tagliasacchi, David Lindell

 

Installation

1. Clone the repository

git clone https://github.com/weihan1/growflow.git
cd growflow/

2. Create a new conda environment (make sure miniconda3 is installed beforehand). We tested on python version 3.10.8

conda create -yn grow_env python=3.10.8
conda activate grow_env

3. Additionally please install PyTorch from (https://pytorch.org/get-started/previous-versions/).
4. Install the first set of requirements file with

pip install -r requirements.txt

5. Install the second set with

pip install -r requirements_no_iso.txt --no-build-isolation

Dataset and checkpoints

To download the dataset, run python -m scripts.download_data from the repository root.

The dataset structure is

├── captured
│   ├── pi_corn_full_subset4
│   ├── pi_paperwhite_full_subset4
│   └── pi_rose
└── synthetic
    ├── clematis_subsample_6
    ├── plant_1_subsample_6
    ├── plant_2_subsample_6
    ├── plant_3_subsample_6
    ├── plant_4_subsample_6
    ├── plant_5_subsample_6
    └── tulip_subsample_6

NOTE: the synthetic datasets are already subsampled with interpolation factor 6, whereas the captured datasets are not subsampled, so you need to manually set the subsample_factor argument (see the boundary reconstruction and global optimization stage).

To download the checkpoints, run python -m scripts.download_checkpoints from the repository root. The checkpoint structure is

├── ckpts
│   ├── gaussian_ckpt_29999_t0.pt
│   └── neural_ode_29999.pt
└── fixed_pc_traj
    └── full_traj_{some_number}.npy

neural_ode_29999.pt is the final neural ODE checkpoint, whereas gaussian_ckpt_29999_t0.pt and full_traj_{some_number}.npy are the cached Gaussians that you can use to directly train the neural ODE in the global optimization stage.

Usage

Training

Our model is trained in 3 stages (see sect. 3.3 of paper), the static reconstruction stage, the boundary reconstruction stage, and the global optimization stage.

Static reconstruction stage

Synthetic:

python -m commands.train synthetic default --data-dir <your_data_dir>

Captured:

python -m commands.train captured default --data-dir <your_data_dir>

Boundary reconstruction stage

Synthetic:

python -m commands.trajectory synthetic default --data-dir <your_data_dir> --static-ckpt <your_ckpt_from_static_stage> --no-adjoint

Captured:

python -m commands.trajectory captured default --data-dir <your_data_dir> --static-ckpt <your_ckpt_from_static_stage> --no-adjoint --subsample-factor <desired_subsample_factor>

NOTE: In the paper, for the rose scene, desired_subsample_factor=17 and for the corn scene, desired_subsample_factor=10. You can also choose your own subsample_factor, however, if it's not a divisor of the total number of timesteps, you need to add the --include-end flag. For the paperwhite scene, use the flag --encoding freq throughout.

Global optimization stage

Synthetic:

python -m commands.train synthetic default --data-dir <your_data_dir> --static-ckpt <your_ckpt_from_static_stage> --full-trajectory-path <your_ckpt_from_boundary_stage> --rtol 1e-5 --atol 1e-6

Captured:

python -m commands.train captured default --data-dir <your_data_dir> --static-ckpt <your_ckpt_from_last_stage> --unscaled-encoder-lr-init 5e-4 --subsample-factor <desired_subsample_factor> --full-trajectory-path <your_ckpt_from_boundary_stage>

Evaluation

Synthetic:

python -m commands.render synthetic --data-dir <your_data_dir> --dynamic-ckpt <your_final_checkpoint_from_global>

Captured:

python -m commands.render captured --data-dir <your_data_dir> --dynamic-ckpt <your_final_checkpoint_from_global>

Use --ground-truth with either render command to export ground-truth views instead. Training and trajectory commands also accept mcmc in place of default.

Metrics

NOTE: You can only run the metrics code after running the eval code.

Synthetic:

python -m commands.metrics_synthetic --skip-dynamic3dgs --skip-4dgs --skip-4dgaussians

Captured:

python -m commands.metrics_captured --skip-dynamic3dgs --skip-4dgs --skip-4dgaussians

The results will be saved in ./growflow/final_results_{scene}.

Tips for training on custom data

• Use bounding box. For all of our experiments, we hand-craft custom bounding boxes to constrain the neural ODE to only learn the the flow field of the foreground Gaussians. On your custom data, simply add the bounds of a bounding box that roughly includes the foreground plant.
• Decrease subsample factor. For the real scenes, decreasing the temporal stride --subsample-factor and the image resolution --data-factor will lead to faster training and overall better reconstruction.
• Swap to MLP. For all experiments, we parametrize the neural ODE using an Hexplane as it performs slightly better in the synthetic scenes (see ablation studies in paper). However, on some real scenes, we find that using an MLP with fourier-encoding performs better. You can swap the representation in the boundary reconstruction stage and global optimization stage by using the flag --encoding freq.

Reproducing numbers

Download the dataset and checkpoints, then run python -m scripts.render_all to get the renderings and then run the metrics code. Run all commands from the repository root.

Credits

This code is built on top of gsplat and torchdiffeq. Thanks to the maintainers for their contribution to the community!

Citing

If you find growflow helpful, please consider citing:

@article{luo2026grow,
  title={Grow with the Flow: 4D Reconstruction of Growing Plants with Gaussian Flow Fields},
  author={Luo, Weihan and Goli, Lily and Bahmani, Sherwin and Taubner, Felix and Tagliasacchi, Andrea and Lindell, David B},
  journal={arXiv preprint arXiv:2602.08958},
  year={2026}
}