nan values in training

[cxym-yyn]

Hello, thodan.
First of all, thanks for your sharing.
I'm trying to training model with 'python train.py --model='ycbv_custom'', but i got nan values in losses. Like:
step: 0 total_loss: 9.91135 obj_cls: 3.09097505 frag_cls: 4.15891361 frag_loc: 2.40971112
step: 100 total_loss: nan obj_cls: nan frag_cls: nan frag_loc: nan
INFO:tensorflow:global_step/sec: 1.57567
step: 200 total_loss: nan obj_cls: nan frag_cls: nan frag_loc: nan
INFO:tensorflow:global_step/sec: 1.66842

And i check the input data with visualization, it works. I trying to fix for days, But I don't where the error is...
Can you give me some advice for that?

[athnzc]

@cxym-yyn hello, I know it's been a long time but were you able to figure out the issue? I am facing the exact same issue and I have tried numerous parameter configurations but have not been successful.