fix(dflash): honor config.is_causal in DFlash training masks [1/3] - #934
Open
maocheng23 wants to merge 1 commit into
Open
maocheng23 wants to merge 1 commit into
maocheng23 wants to merge 1 commit into
Conversation
SGLang's DFlash model (v0.5.18+) decides in-block causality from config.is_causal: - unset: full layers bidirectional, sliding layers causal - False: every layer bidirectional - True: every layer causal SpecForge ignored the field and always trained sliding layers causal and full layers bidirectional, so drafts with `is_causal: false` and sliding layers were trained causal but served bidirectional. create_dflash_sdpa_mask / create_dflash_block_mask now take is_causal and resolve it with SGLang's rule, and the online wrapper passes the draft config's value. Sliding layers also get the window's left bound inside the draft block (no right bound: SGLang backends disagree on one for non-causal windows); this only differs from before when sliding_window < block_size. Split out of #905. Co-authored-by: Inhyuk Cho <ihcho@lgresearch.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 6, 2026
cih9088
approved these changes
Oct 6, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split of #905, part 1 of 3 (the actual bug fix). Stacked on upstream main
53398a8.Motivation
SGLang's DFlash model (v0.5.18+,
_get_dflash_attention_type) decides causality inside the draft block fromconfig.is_causal:false: all layers bidirectionaltrue: all layers causalSpecForge ignores the field. It always trains sliding layers causal and full layers bidirectional, so any draft with
is_causal: falseand sliding layers trains causal but is served bidirectional.Changes
create_dflash_sdpa_mask/create_dflash_block_masktakeis_causaland resolve it with SGLang's rule viaresolve_dflash_is_causal(is_causal, layer_type): an explicit value applies to every layer, unset is the per-layer-type default.is_causal.kv_off >= q_off - (W-1)). There is no right bound, because SGLang backends disagree on one for non-causal windows (FA3 symmetric(W-1, W-1), FlashInfer/Triton left-only). This only differs from before whensliding_window < block_size, which no shipped config has (W = 2048/4096, block 8/16).benchmark_dflash2_conv.pypasses the config'sis_causaltoo.configs/qwen3.6-27b-dflash2.jsonandconfigs/qwen3.8-27b-dflash2.jsonsetis_causal: falseand use only sliding layers. After this PR they train bidirectional inside the block (which is what SGLang v0.5.18+ already serves). This needs a decision before merging:is_causalfrom both configs, which keeps today's causal training and makes serving causal too.Causal vs bidirectional inside the block has never been ablated cleanly. Existing checkpoints can be served to match how they were trained, without retraining, by deleting
is_causalfrom the servedconfig.json.Tests
tests/test_utils/test_dflash_mask.py: the reference mask is extended withis_causal. New tests cover dense vs reference, flexmask_modvs dense, and flex block occupancy for window ∈ {None, 1, 3, 9} × is_causal ∈ {None, False, True}, plus an in-block spot check and a check that the training wrapper readsdraft_model.config.is_causal.Run on CPU (torch 2.14, transformers 5.12.1): every
tests/test_utils/test_dflash*andtests/test_modeling/test_dflash*file passes when run alone.Co-authored-by @cih9088 (original #905); every commit carries a
Co-authored-by: Inhyuk Chotrailer.🤖 Generated with Claude Code