the problem of training curve

[yingchao-piao]

Hi
I try to train the model, and the accuracy reward curve go up from about 0.75 to 0.79(smoothed) in 47k steps. I think it is to narrow. Is it correct? how about your experiment?

my settings:
torchrun --nproc_per_node="1"
--nnodes="1"
--node_rank="0"
--master_addr="127.0.0.1"
--master_port="12345"
src/open_r1/grpo_jsonl.py
--output_dir output/$RUN_NAME
--model_name_or_path /VisualQuality-R1-master/finetune_model/Models/Qwen2.5-VL-3B-Instruct
--question_template scoring
--dataset_name KADID-10K
--image_folders /VisualQuality-R1-master/datasets/KADID-10K/images
--data_file_paths /VisualQuality-R1-master/datasets/KADID-10K/scoring/RL-KADID-10K_train_scoring.jsonl
--freeze_vision_modules false
--max_prompt_length 1024
--num_generations 4
--per_device_train_batch_size 24
--gradient_accumulation_steps 3
--logging_steps 1
--bf16
--torch_dtype bfloat16
--data_seed 42
--report_to tensorboard
--gradient_checkpointing true
--attn_implementation flash_attention_2
--num_train_epochs 40
--run_name $RUN_NAME
--save_steps 400
--save_only_model true

[TianheWu]

Thanks. It is better to observe the validation srcc and plcc.

[yingchao-piao]

Thanks. It is better to observe the validation srcc and plcc.

Thanks for replay.
I will validate srcc later. I saw other method for image quality assesment based on GRPO, the reward go up like from 0.4 to 0.7 which seems improving a lot.

I am wondering the what is the normal range of accuracy reward during your training process.