[feat] Add HunyuanVideo 1.5 embedding preprocessing pipeline

[klhhhhh]

Summary

This PR adds the embedding preprocessing pipeline for HunyuanVideo 1.5 training and extends the existing T2V dataset pipeline to support dual text encoders.

Changes

Dual text embedding support

• Extend the T2V parquet schema by introducing text_embedding_2_* fields for the ByT5 encoder while keeping backward compatibility with the existing dataset format.
• Update the parquet collator to load both Qwen and ByT5 embeddings, generate their corresponding attention masks, and batch them into the training input.
• Apply synchronized CFG dropout across both text encoders so the conditional and unconditional branches remain consistent during training.
• Support samples without quoted text by storing an empty ByT5 embedding (shape=[0, hidden_size]) instead of removing the field, allowing downstream components to handle all samples through a unified data format.

HunyuanVideo 1.5 embedding preprocessing

• Add a preprocessing pipeline for HunyuanVideo 1.5 that generates:
  • VAE latents
  • Qwen text embeddings
  • ByT5 text embeddings
• Serialize all generated embeddings and metadata into the FastVideo parquet format for training.
• Extract quoted text from captions for the ByT5 encoder while preserving the original caption for the Qwen encoder.

Validation

• Verified that the preprocessing pipeline correctly generates VAE latents, Qwen embeddings, and ByT5 embeddings.
• Verified that both text embeddings and attention masks are correctly loaded by the collator.
• Verified that samples without quoted text produce valid empty ByT5 embeddings and can still be batched correctly.
• Verified compatibility with the existing T2V parquet loading pipeline.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[mergify]

⚠️ PR title format required

Your PR title must start with a type tag in brackets. Examples:

• [feat] Add new model support
• [bugfix] Fix VAE tiling corruption
• [refactor] Restructure training pipeline
• [perf] Optimize attention kernel
• [ci] Update test infrastructure
• [infra] Add activation trace hooks
• [docs] Add inference guide
• [misc] Clean up configs
• [new-model] Port Flux2 to FastVideo
• [skill] Add add-model agent skill

Valid tags: feat, feature, bugfix, fix, refactor, perf, ci, infra, doc, docs, misc, chore, kernel, new-model, skill, skills

Please update your PR title and the merge protection check will pass automatically.

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=fastcheck-passed
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=fastcheck-passed
• check-success=full-suite-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

[mergify]

Pre-commit checks failed

Hi @klhhhhh, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

• yapf: yapf -i <file> (formatting)
• ruff: ruff check --fix <file> (linting)
• codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

[KyleNeverGivesUp]

The training side that consumes this is #1662.

[SolitaryThinker]

Kun Lin — pushed three commits' worth of changes to your branch rather than sending a list back. Revert anything you disagree with.

The blocker. The three new fields break the five existing overfit preprocess scripts. pa.table(mapping, schema=...) requires the mapping to contain every field in the schema, and nullable=True doesn't exempt it, so preprocess_cosmos_overfit.py:157, preprocess_cosmos25_overfit.py:169, preprocess_hunyuan_overfit.py:184, preprocess_kandinsky5_overfit.py:287 and preprocess_ltx2_overfit.py:233 all raise:

KeyError: "The passed mapping doesn't contain the following field(s) of the schema:
text_embedding_2_bytes, text_embedding_2_shape, text_embedding_2_dtype"

All five succeed on base, so it's a regression. The LTX2 and Kandinsky5 nightly overfit e2e tests invoke these scripts directly; fastcheck stays green because it doesn't run nightly.

Fixed by switching them to records_to_table(records, pyarrow_schema_t2v), which already exists in dataset/dataloader/parquet_io.py and goes through from_pylist — that fills absent keys with null. Your new hunyuan15 script already avoids the problem by iterating the schema with .get(); this just propagates the idea. Confirmed the resulting table is schema-identical and the collator skips the null column. Dropped the now-unused import pyarrow as pa from each.

Two smaller ones:

dataset/utils.py:175 used torch.empty for a zero-byte payload. For a truncated row that declares a non-empty shape, base failed loudly in stack(); empty() silently stacks uninitialized memory. Changed to torch.zeros.

test_hunyuan15_secondary_text_embedding_fields_are_nullable couldn't fail. nullable=True is pyarrow's default for pa.field, so it holds for every field in every schema and would still hold with the explicit flag deleted. Replaced it with the behavior it was reaching for: a pre-secondary-encoder record raises under the old pattern and round-trips under records_to_table.

Verified: ruff clean on every changed file, fastvideo/tests/dataset/ 40 passed.

Credit where it's due — sizing the empty-ByT5 placeholder from encoder.config.d_model on the loaded checkpoint instead of a config default is the right call and avoids a real class of bug, and replacing the hardcoded 768-wide fallback with a shape-derived dimension fixed a latent bug affecting every model, not just this one.

Two things I left alone: _decode_tensor_from_parquet + _NUMPY_DTYPES (dataset/utils.py:310-365) have no callers and no tests, and are also the only code honoring the _dtype column while the live path hardcodes float32 — wire it in or drop it, your call. And there's no round-trip test that writes a new-schema parquet and reads it back through the collator; I did it by hand and it works, but nothing guards it.

Note this PR and #1662 form a contract in both directions — see the comment there.

[klhhhhh]

@SolitaryThinker, Thanks for the detailed review and fixes! I’ve addressed the remaining points by adding the parquet round-trip tests for the dual text embeddings, including the empty ByT5 case, and removing the unused _decode_tensor_from_parquet / _NUMPY_DTYPES code. Could you take another look when you have a chance? Thanks!