Motivation.
First off — thank you very much for the excellent work on FastVideo. I'm currently integrating VSA into our internal pipeline, and two questions came up that I'd really value your input on.
- Running inference with VSA_TILE_SIZE=(4,8,4) using weights trained with (4,4,4)
- As I understand it, the released Wan2.1-VSA-T2V-14B-720P checkpoint was trained with VSA_TILE_SIZE=(4,4,4). Our inference platform, however, only has kernels that support VSA_TILE_SIZE=(4,8,4) — so I'm effectively running the stock (4,4,4)-trained weights through a (4,8,4) VSA kernel at sampling time. Qualitatively the videos still look reasonable, but I wanted to check whether this weight/kernel tile-size mismatch is actually expected to be benign
- Are the non-to_gate_compress weights retrained, or just precision-cast from the original fp32?
-
For reasons explained in question 1, I'm planning to retrain the VSA model with VSA_TILE_SIZE=(4,8,4) instead of the (4,4,4) used for the released checkpoint. To keep training cost manageable, I'd like to confirm the minimal training recipe:
-
Is it correct/sufficient to train only the to_gate_compress.* parameters in each transformer layer and freeze everything else (all attention projections to_q/k/v/out, FFN, embeddings, norms, scale_shift_table, etc.)?
-
Is to_gate_compress the only component whose role depends on the tile-size configuration, with the rest of the transformer effectively tile-size-agnostic?
-
Or does changing VSA_TILE_SIZE also shift the optimal values of other layers, meaning they should be trainable as well for best quality?
-
If only training to_gate_compress is suboptimal, which other module groups would you recommend unfreezing, and in what order of priority?
Thanks again for open-sourcing this work, and thank you in advance for any time you can spare to answer.
Proposed Change.
Looking forward to your reply
Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...
Motivation.
First off — thank you very much for the excellent work on FastVideo. I'm currently integrating VSA into our internal pipeline, and two questions came up that I'd really value your input on.
For reasons explained in question 1, I'm planning to retrain the VSA model with VSA_TILE_SIZE=(4,8,4) instead of the (4,4,4) used for the released checkpoint. To keep training cost manageable, I'd like to confirm the minimal training recipe:
Is it correct/sufficient to train only the to_gate_compress.* parameters in each transformer layer and freeze everything else (all attention projections to_q/k/v/out, FFN, embeddings, norms, scale_shift_table, etc.)?
Is to_gate_compress the only component whose role depends on the tile-size configuration, with the rest of the transformer effectively tile-size-agnostic?
Or does changing VSA_TILE_SIZE also shift the optimal values of other layers, meaning they should be trainable as well for best quality?
If only training to_gate_compress is suboptimal, which other module groups would you recommend unfreezing, and in what order of priority?
Thanks again for open-sourcing this work, and thank you in advance for any time you can spare to answer.
Proposed Change.
Looking forward to your reply
Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...