Skip to content

Fix canonical FP32 gradients and BF16 branch graph lifetime - #56

Merged
CompleteDotTech merged 1 commit into
mainfrom
fix/canonical-fp32-and-branch-graph-lifetime
Sep 30, 2026
Merged

CompleteDotTech merged 1 commit into
mainfrom
fix/canonical-fp32-and-branch-graph-lifetime

Conversation

@CompleteDotTech

Copy link
Copy Markdown
Collaborator

Changes

Refs #3 and #4. This is a numerical/source repair and scoped evidence delivery; neither issue is closed.

  • Make CUDA FP32 query dispatch depend on absolute positions, with fixed short eager padding, CPU-initialized RoPE bits, and coherent double shared K/V and LoRA accumulation.
  • Charge materialized double GQA storage correctly and include the cache planner in CUDA resume identity.
  • Release completed BF16 branch graph owners before optimizer allocation, with an actual CUDA lifetime regression and a failing old-source control.
  • Publish opt-in FP32 HF/seam reproducers, portable numerical/mechanics/failure receipts, and preserve the original 45-feature matrix.
  • Exclude generated TypeScript dependencies/build output from the source distribution.

Verification

  • Final current-source CPU suite: 858 passed, 55 explicit skips (69.85 s); compilation, critical Ruff, whitespace, source-index audit and 45-feature/450-cell verifier pass.
  • Independent source review found no blocking defect. New lifetime regression passes; proper old-source control fails; all 14 selected BF16 CUDA tests pass (counts overlap).
  • Actual pinned-weight FP32: independent HF hidden result exact and all 96 Q/V adapter gradients pass; four short/seam cases plus 8191/8192 boundaries and a 10520-token state pass all 100 trainable gradients in each batched/serial/full comparison. Frozen tolerances remain unchanged.
  • Public HF command and all four public short/seam cases pass on the frozen trained checkpoint.
  • Noncheckpointed native rank-8 FP32 and BF16 each complete training, reload, same-run resume to step two, and reload. These are generated short mechanics, with independently initialized heads.
  • Rebuilt wheel matches all 32 runtime modules and four license files. Source archive inspection found zero generated client dependency/build members.

Limits retained

The exact 32768-branch/65536-aggregate FP32 training resource probe OOMs during forward at the unchanged 70% allocator cap. Full-weight noncheckpointed FP32 resume also OOMs. The graph cleanup allows the same-init BF16 first optimizer step and reload, but resumed backward still OOMs. An allocator experiment is explicitly unsuccessful: expandable segments are unsupported on this Windows runtime. All failures and artifact/attempt histories are preserved.

Historical numerical/rank-8 receipts retain their original source hashes; the later graph cleanup has a separate source-bound receipt. This does not establish full-weight resume, installed serving for every mode, representative predictive quality, or Jev benefit. Overwatch remains archived by user direction.

@CompleteDotTech
CompleteDotTech merged commit 4f31cc7 into main Sep 30, 2026
5 checks passed
@CompleteDotTech
CompleteDotTech deleted the fix/canonical-fp32-and-branch-graph-lifetime branch October 1, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant