Repository navigation
I have successed in running the code with 4 a800 80g #9
Description
Activity
--num_generations 4
--per_device_train_batch_size 8 \means not 8 pictures but 2 pictures ,while means it may decrease to dpo ,but with 4 a800 80g, it is my only way to run the code, i think this may hurt performence badly, i will continue focusing the performanceThanks. Maybe there are some settings in your code are different. 4 A800 80G can run 8 images with num_generation 6 in my settings.
Hi, I ran into the same OOM issue. For reference, with H20 96GB GPUs, the maximum batch size I can push per device is 12.
Curious whether you ended up seeing a noticeable performance drop after the 48-hour run — would love to hear your results when they're ready.
and it takes 48hours ,I will report if this change will down the performance