[ICLR 2023] DFlow: Learning to Synthesize Better Optical Flow Datasets via a Differentiable Pipeline (Paper)
Comprehensive studies of synthetic optical flow datasets have attempted to reveal what properties lead to accuracy improvement in learning-based optical flow estimation. However, manually identifying and verifying the properties that contribute to accurate optical flow estimation require large-scale trial-and-error experiments with iteratively generating whole synthetic datasets and training on them, i.e., impractical. To address this challenge, we propose a differentiable optical flow data generation pipeline and a loss function to drive the pipeline, called DFlow. DFlow efficiently synthesizes a dataset effective for a target domain without the need for cumbersome try-and-errors. This favorable property is achieved by proposing an efficient dataset comparison method that uses neural networks to approximately encode each dataset and compares the proxy networks instead of explicitly comparing datasets in a pairwise way. Our experiments show the competitive performance of our DFlow against the prior arts in pre-training. Furthermore, compared to competing datasets, DFlow achieves the best fine-tuning performance on the Sintel public benchmark with RAFT.
- torch=2.0.1
- torchvision=0.10.0
- numpy=1.21.4
- kornia==0.7.0
You can download the our DFlow dataset and the pretrained RAFT model from the link below.
- DFlow dataset (https://drive.google.com/drive/folders/1_gu-N_Dfaywd-o-Ag9CMPoMfXurdLCUN?usp=sharing)
- Pretrained model (https://drive.google.com/drive/folders/1LLpxyGcgE9rUoWsZMdC07bdrV4QQBMEV?usp=sharing)
Download the DFlow dataset and place it in the "dataset" folder. Then, enter the "bash train_standard.sh" command. When training with the DFlow dataset, the data augmentation used during dataset creation is applied. Please refer to section B.1 in our paper for more details.
If you find this work useful for your research, please cite:
@inproceedings{byung2022dflow,
title={DFlow: Learning to Synthesize Better Optical Flow Datasets via a Differentiable Pipeline},
author={Byung-Ki, Kwon and Hyeon-Woo, Nam and Kim, Ji-Yun and Oh, Tae-Hyun},
booktitle={The Eleventh International Conference on Learning Representations},
year={2022}
}
The majority of the code has been adapted from previous works:
DFlow's original contributions are released under the MIT License.
Third-party components retain their original licenses and terms. In particular, code adapted from RAFT remains subject to the BSD 3-Clause License. Any code derived from Softmax Splatting remains subject to the academic-use terms stated by its authors. Datasets, pretrained models, and other assets are subject to their respective upstream terms.