Skip to content

Add experimental persistent BF16 full fine-tuning - #3

Open
muthu813 wants to merge 1 commit into
Edge-Intelligence-Lab:mainfrom
muthu813:main
Open

Add experimental persistent BF16 full fine-tuning#3
muthu813 wants to merge 1 commit into
Edge-Intelligence-Lab:mainfrom
muthu813:main

Conversation

@muthu813

@muthu813 muthu813 commented Sep 8, 2026

Copy link
Copy Markdown

This PR adds an experimental persistent-BF16 parameter path for native C++ Full-FT.

Key changes:

  • Persistent BF16 parameter storage
  • FP32 gradients and FP32 Adam moments
  • BF16-aware LayerNorm, RMSNorm, MatMul, and gradient accumulation paths
  • FP32-to-BF16 SafeTensors loading
  • BF16 validation tests
  • GPT-2 124M full-model validation
  • README documentation of the experimental extension

Validation performed on an ARM64 Android device:

  • 148/148 GPT-2 124M parameter tensors loaded as BF16
  • 0 FP32 parameter tensors
  • Full 124M forward pass completed successfully
  • 10/10 forward passes
  • 10/10 backward passes
  • 10/10 FP32 Adam updates
  • Parameters remained BF16 throughout the 10-step run
  • Separate 20-step real-data causal Full-FT experiment completed

This is an engineering extension of MobileFineTuner, not a claim of a new optimization algorithm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant