multiprompt-longvideo Benchmark

[zhw-zhang]

Hi, thank you for your work—this paper has been very helpful to me. However, I’m a bit confused about the evaluation metrics. I’ve attached some screenshots summarizing what I found, and I’d really appreciate it if you could take a look and help clarify when you have time.

[SihuiJi]

Hi! Thanks for your interest.

Quality Score: Calculated via VBench on dimensions supporting customized videos, using weights defined in scripts/constant.py.
Consistency Score: The average of Subject Consistency and Background Consistency.
Aesthetic Score: The Aesthetic Quality value.

[zhw-zhang]

Thank you for your response, but I am still a bit confused.

Regarding the Quality Score, it seems that for customized videos, only the imaging_quality dimension is supported. In this case, how is the weighting applied? Could you clarify how the weights in scripts/constant.py are defined? For these specific metrics, could you provide a calculation formula?

Additionally, it appears that the Consistency Score also includes motion smoothness. Why was this metric omitted?

[SihuiJi]

1. As claimed in Vbench-long, 6 dimentions support customized videos.
2. See cal_final_score for usage of weights in scripts/constant.py.
3. We define Consistency Score mainly for visual consistency, so we include background and foreground consistency, you can define you own metric for consistency evaluation.

[zhw-zhang]

1. As claimed in Vbench-long, 6 dimentions support customized videos.
2. See cal_final_score for usage of weights in scripts/constant.py.
3. We define Consistency Score mainly for visual consistency, so we include background and foreground consistency, you can define you own metric for consistency evaluation.

Thank you for your response.

re_1. I am aware there are 6 dimensions. However, it seems that the only sub-dimension corresponding to the original Quality Score is imaging_quality, as shown in my first screenshot.

re_3. Thanks for the clarification. I previously assumed it was the same as the Consistency Score defined in VBench, as I don't see this specific point mentioned in the paper.