Repository navigation
Metrics reproduction yields significant differences for CollisionRate #131
Description
Activity
@w-poh
Hi~ I found out that in the README file at Results of released checkpoint, the authors found some errors in the collision rate calculation, so they released the updated checkpoints, logs, and results.You can check the updated planning results at Detailed results for planning, where the collision rate (3s) = 0.23 (result in paper was 0.18).
However, yeah, compared to the reproduction results you shared, some of the collision rate results (3s & Avg) do show high differences.
@w-poh
Hi, I try reproducing the results on my system (only 3 4090 GPUs) and the training loss behaviors are quite different from the published logs.
Could you please share me the training logs of your reproductions?
I am trying to reproduce Sparsedrive. My setup is to use similar number of GPUs, but for my HPC just wrapped in a docker. The docker image and environment is according to the repository's requirements file.
When training, I get non-deterministic results, which is explainable by atomic operations with the CUDA kernel.
But for some metrics such as Collision Rate I obtained a lot worse results. Has anyone already a solution to that? I have taken the seeds as in the repository.
In the Chinese issue #124 it is suggested to change post-processing, did anyone find settings that lead to the paper metrics?
My results so far are:
Detection and Tracking
Mapping
Planning
I'd be thankful for experience or an author's comment on that @swc-17.