Unable to reproduce the reported performance / Cannot reach the official accuracy

[hhhjjy]

Hello, thanks for your great work!
I am trying to reproduce the results reported in the paper, but I am currently unable to reach the official accuracy. I have strictly followed the instructions in the README, but the performance is still much lower than reported.
Total loss converges and stays around ~1.0
Evaluation metrics are much lower than reported

this is the logs:
2026-04-01 18:35:07 - INFO - Total Loss: 1.7167, Obj Loss: 1.5992, Pr Loss: 0.1176, -TCR: -82.7548 Object Sim Loss: 0.2805
2026-04-01 18:35:07 - INFO - Processing video 42446529, progress 99/200
2026-04-01 18:35:08 - INFO - {
"running_loss_total": 1.769169569015503,
"tcr": -75.73298899332683,
"obj_sim_loss": 0.27877716223398846,
"mean_dis": 0.5410435597101847,
"running_loss_obj": 1.635296146074931,
"running_loss_pr": 0.13387343287467957
}
2026-04-01 18:35:08 - INFO - Total Loss: 1.7692, Obj Loss: 1.6353, Pr Loss: 0.1339, -TCR: -75.7330 Object Sim Loss: 0.2788
2026-04-01 18:35:08 - INFO - Processing video 42446532, progress 100/200
2026-04-01 18:35:10 - INFO - {
"running_loss_total": 1.8055452108383179,
"tcr": -77.96596908569336,
"obj_sim_loss": 0.27827901393175125,
"mean_dis": 0.5986064672470093,
"running_loss_obj": 1.6633397042751312,
"running_loss_pr": 0.14220549166202545
}
2026-04-01 18:35:10 - INFO - Total Loss: 1.8055, Obj Loss: 1.6633, Pr Loss: 0.1422, -TCR: -77.9660 Object Sim Loss: 0.2783
2026-04-01 18:35:10 - INFO - Evaluation done. Final Average avg pp: 0.2996, avg_po: 0.4536 avg_graph_iou: 0.3030
2026-04-01 18:35:10 - INFO - Training on video ['47331343', '48458447', '43896512', '41126729', '41124923', '47204291'], progress 3325/4492 Epoch 4
2026-04-01 18:35:17 - INFO - videos=0002801, steps=0014462, Total Loss: 0.9525, Obj Loss: 0.9255, Pr Loss: 0.0270, -TCR: -409.6833 Object Sim Loss: 0.6012
2026-04-01 18:35:17 - INFO - Training on video ['42444703', '48458318', '42445916', '42899596', '47332184', '42444908'], progress 3331/4492 Epoch 4
2026-04-01 18:35:26 - INFO - videos=0002802, steps=0014467, Total Loss: 1.0280, Obj Loss: 1.0082, Pr Loss: 0.0198, -TCR: -322.3859 Object Sim Loss: 0.5146
2026-04-01 18:35:26 - INFO - Training on video ['44358718', '45261457', '47334068', '45663397', '47334025', '44358295'], progress 3337/4492 Epoch 4
2026-04-01 18:35:32 - INFO - videos=0002803, steps=0014470, Total Loss: 1.0516, Obj Loss: 1.0257, Pr Loss: 0.0260, -TCR: -366.1333 Object Sim Loss: 0.5912
2026-04-01 18:35:32 - INFO - Training on video ['41098145', '47430137', '47333803', '44796473', '43828505', '47115391'], progress 3343/4492 Epoch 4

Could you tell me where the problem might be and what suggestions you have?

[hhhjjy]

It seems that the model has converged after 4 epochs and the loss no longer decreases.

[juexZZ]

Hi @hhhjjy
Thank you for your interest. Please look at this reproduce report.
https://wandb.ai/juexiao/MSG-repr/reports/MSG--VmlldzoxMzU3MzIwMQ?accessToken=kgvxvn705h7343iskny0gpjtkafory148is4sfxayjk89crvb8qgsr9k57l1dm1h

[juexZZ]

I'd say the training is a bit sensitive to the learning rate and batch size, given the fact that the data loading is sampling-based, and the objective is batch-wise supervised contrastive. So pay attention to these two key hyperparameters.

[hhhjjy]

I noticed that during the later training stage, only obj_sim_loss keeps decreasing. Could this mean that this loss has a greater impact on the evaluation metrics?