Load-bearing caveats. Every item below is something that would change the story if someone else reran this on a different machine or with different software.
- Single machine, single chip. Apple M5 with 32 GB unified memory, macOS
26.1,
torch==2.11.0,python==3.14.3. Earlier M-series chips and earlier torch releases are known to handle MPS, bf16, andtorch.compiledifferently. - No CUDA. None of the "real" int4/int8 matmul kernels (bitsandbytes,
AWQ kernels, GPTQ kernels), nor FlashAttention, nor CUDA-only compile paths
were tried. Every "real int8 math" claim in this study is via
torch.ao.quantization.quantize_dynamic+qnnpackon CPU. - No thermal control. Runs are sequential on a laptop; long tails may
throttle.
hardware.thermal_noterecords this on every JSON. No cold-machine re-run is performed to amortize - sequential wall-clock is the answer being measured.
- Tiny models only. Conclusions from Qwen2.5 0.5B / 1.5B / 3B do not extrapolate to frontier-scale models. A 50% prune here detonates PPL; a 50% prune of a 70B model is a different regime.
- Wikitext-2 PPL is a narrow signal. It captures language-modeling loss on a single domain. It will miss regressions that downstream tasks, code, or non-English inputs would catch.
- Single-machine, single-day measurements. No cross-run statistical comparisons; variance is reported within a run (forward-latency stddev) but not across runs.
The export matrix ships torch.compile and ONNX runs but not CoreML. On this
venv, coremltools==9.0 fails to load its native bindings
(_MLModelProxy, BlobWriter) under torch==2.11.0 + python==3.14.3, so
ct.convert() cannot produce an .mlpackage at all. Before that,
torch.jit.trace through transformers>=5.x's masking_utils.sdpa_mask hits a
compat bug (q_length.shape[0] on a 0-d trace proxy). Both failures are deep
in dependencies. Rather than ship a half-working number or a silent fallback
to a different path, the matrix ships without CoreML and this file records why.
Revisit when coremltools catches up to the installed torch version.
Four methods in this repo are reimplementations from scratch:
int8_wo_cpu- per-channel symmetric weight-only int8.int4_wo_cpu- group-wise (128) weight-only int4.gptq_like- an educational GPTQ-style error-compensation pass.int8_kv_*- per-token/per-head symmetric int8 KV cache.
These are written to be understandable, not competitive. They do pack bytes
honestly and dequantize on every forward; they do not use fused int kernels.
Numbers from these rows are not a benchmark of the technique - they are a
benchmark of this implementation of the technique.
speculative.py decides acceptance by exact argmax match between draft and
target. This is the simplest correct formulation - zero divergence from the
target on accepted tokens. It does not cover sampling-based speculative
decoding (draft and target sample, acceptance uses probability ratios). Accept
rate and speedup will look different with sampling; that regime is out of scope.
200 KL steps at seq_len=128, batch_size=1, lr=1e-4, T=2.0. This is a
"did it move?" experiment, not a training run. A negative result (PPL worse
after distillation than before) is expected and is reported as-is.
The 3B → 1.5B cell is intentionally omitted on this machine. Holding the 3B teacher (~12 GB in fp32) plus the 1.5B student with gradients and AdamW state (~12 GB) plus activations pushes resident + compressed memory past the 32 GB unified RAM budget. In practice the process falls into swap thrash: step 1 finishes in 26 s, step 10 in 18 min, and wall-clock keeps growing superlinearly as ~23 GB of swap fills. Rather than ship a number that is really a measurement of macOS swap, the cell is dropped. The 1.5B → 0.5B signed-delta result stands on its own.
- No claim of compile/ONNX/int8 parity with a production serving stack.
- No claim that any method reaches a particular accuracy floor (e.g., "close to fp32"). Where PPL delta is small, the number is reported; where it is large, the number is reported.
- No claim of a best configuration - only measurements of the configurations that ran.