Name and Version
This is all from my Mac studio m4 max 36gb:
Operating systems
Mac
GGML backends
Metal
Hardware
0.3.1
Models
No response
Problem description & steps to reproduce
I tried like https://github.com/Anbeeld/beellama.cpp/blob/main/docs/quickstart-qwen36-dflash.md
but not work
First Bad Commit
No response
Relevant log output
tien@TiensMacStudio beellama-v0.3.0 % ./llama-server
-m "/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-Q4_K_M.gguf"
--spec-draft-model "/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-DFlash-Q4_K_M.gguf"
--spec-type dflash
--spec-dflash-cross-ctx 512
--port 8080
-np 1
--kv-unified
-ngl all
--spec-draft-ngl all
-b 2048 -ub 512
--ctx-size 102400
--cache-type-k q5_0 --cache-type-v q4_1
--flash-attn on
--jinja
--no-mmap --mlock
--no-host
--reasoning on
--chat-template-kwargs '{"preserve_thinking":true}'
--temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0
0.00.035.194 I dflash: setting -cd to 256 (drafter doesn't need the full main ctx; pass -cd N to override)
0.00.035.219 I log_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.035.220 I device_info:
0.00.035.224 I - MTL0 : Apple M4 Max (28753 MiB, 28753 MiB free)
0.00.035.224 I - BLAS : Accelerate (0 MiB, 0 MiB free)
0.00.035.228 I - CPU : Apple M4 Max (36864 MiB, 36864 MiB free)
0.00.035.238 I system_info: n_threads = 10 (n_threads_batch = 10) / 14 | MTL : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | REPACK = 1 |
0.00.035.249 I srv init: running without SSL
0.00.035.285 I srv init: using 13 threads for HTTP server
0.00.035.333 I srv start: binding port with default address family
0.00.036.654 I srv llama_server: loading model
0.00.036.679 I srv load_model: loading model '/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-Q4_K_M.gguf'
0.00.208.967 I srv load_model: [spec] estimated memory usage of draft model is 2839.85 MiB
0.00.208.976 I common_init_result: fitting params to device memory ...
0.00.208.977 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.05.254.594 W llama_context: n_ctx_seq (102400) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.05.711.047 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.06.600.998 I srv load_model: shrunk recurrent state to 1 cells before draft load (deferred 1 backup cells)
0.06.601.002 I srv load_model: loading draft model '/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-DFlash-Q4_K_M.gguf'
0.06.601.013 I srv load_model: DFlash draft model will use target output device MTL0 by default; pass --spec-draft-device to override
0.07.004.778 I srv load_model: expanded recurrent state to 2 cells before speculative GPU buffers
0.07.004.783 I srv load_model: initializing slots, n_slots = 1
0.07.137.424 W seq_rm: RS-ROLLBACK-OVERFLOW seq=0 p0=1 cell.pos=1 rollback=1 > n_rs_seq=0 -> CANNOT ROLLBACK
0.07.148.342 W srv load_model: speculative decoding will use checkpoints
0.07.148.343 I srv load_model: DFlash enabled for all 1 slots
0.07.148.363 W llama_context: n_ctx_seq (256) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.07.257.771 I common_speculative_create_ctx_dft: draft model warmup complete
0.07.257.791 I common_speculative_init: adding implementation dflash
0.07.257.815 I dflash: contract ok: block_size=16 mask_token=248070 target_layer_ids=[1,16,31,46,61] n_target_layers=5 n_embd=5120 n_target_features=25600 target_layers=65 cross_ctx=512
0.07.257.817 I dflash: target/drafter info: target_ctx_train=262144 target_vocab=248320 drafter_vocab=248320 vocab_match=1 capture_min=1 capture_max=61
0.07.284.759 I dflash: GPU hidden capture policy: allowed=1 forced_cpu=0 requested=1 target_devices=1 drafter_devices=1
0.07.284.805 W dflash: GPU cross ring unavailable; using CPU hidden capture
0.07.285.737 I slot load_model: id 0 | task -1 | speculative decoding context initialized
0.07.285.738 I slot load_model: id 0 | task -1 | new slot, n_ctx = 102400
0.07.285.824 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
0.07.285.825 I srv load_model: use --cache-ram 0 to disable the prompt cache
0.07.285.825 I srv load_model: for more info see ggml-org#16391
0.07.285.825 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
0.07.285.890 I srv init: idle slots will be saved to prompt cache and cleared upon starting a new task
0.07.296.320 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
0.07.302.530 I srv init: init: chat template, thinking = 1
0.07.302.543 I srv llama_server: model loaded
0.07.302.547 I srv llama_server: server is listening on http://127.0.0.1:8080
0.07.302.551 I srv update_slots: all slots are idle
0.47.174.111 I srv params_from_: Chat format: peg-native
0.47.174.391 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
0.47.174.394 I slot get_availabl: id 0 | task -1 | adaptive dm: reset state for LRU slot selection
0.47.194.983 I srv recurrent_sh: shrunk recurrent state to 1 cells for prompt cache (before prompt cache save/load, removed 1 backup cells)
0.47.194.987 I srv get_availabl: updating prompt cache
0.47.194.995 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.47.194.997 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 102400 tokens, 8589934592 est)
0.47.219.429 I srv recurrent_ex: expanded recurrent state to 2 cells after prompt cache (after prompt cache save/load)
0.47.219.436 I srv get_availabl: prompt cache update took 24.45 ms
0.47.219.479 I reasoning-budget: activated, budget=2147483647 tokens
0.47.219.481 I reasoning-budget: deactivated (natural end)
0.47.219.491 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.47.658.221 I slot create_check: id 0 | task 0 | created context checkpoint 1 of 32 (pos_min = 12, pos_max = 12, n_tokens = 13, size = 150.896 MiB)
0.48.079.230 I slot operator(): id 0 | task 0 | adaptive dm profit: cur=0 recommended=4 score=13.5 action=apply
WARNING: Using native backtrace. Set GGML_BACKTRACE_LLDB for more info.
WARNING: GGML_BACKTRACE_LLDB may cause native MacOS Terminal.app to crash.
See: ggml-org#17869
0 libggml-base.0.13.1.dylib 0x00000001011162c0 ggml_print_backtrace + 276
1 libggml-base.0.13.1.dylib 0x000000010112cef4 _ZL23ggml_uncaught_exceptionv + 12
2 libc++abi.dylib 0x0000000187c2f75c _ZSt11__terminatePFvvE + 16
3 libc++abi.dylib 0x0000000187c31be4 __cxa_get_exception_ptr + 0
4 libc++abi.dylib 0x0000000187c1e09c __cxa_get_globals + 0
5 libllama.0.0.10070.dylib 0x0000000100d79ed4 _ZNSt3__120__throw_out_of_rangeB8ue170006EPKc + 60
6 libllama.0.0.10070.dylib 0x0000000100e64234 ZNKSt3__112__hash_tableINS_17__hash_value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEiEENS_22__unordered_map_hasherIS7_S8_NS_4hashIS7_EENS_8equal_toIS7_EELb1EEENS_21__unordered_map_equalIS7_S8_SD_SB_Lb1EEENS5_IS8_EEE4findIS7_EENS_21__hash_const_iteratorIPNS_11__hash_nodeIS8_PvEEEERKT + 0
7 libllama.0.0.10070.dylib 0x0000000100e5bae4 _ZNK11llama_vocab4impl14token_to_pieceEiPciib + 1300
8 libllama-common.0.0.10070.dylib 0x00000001021802b0 _Z21common_token_to_piecePK11llama_vocabib + 112
9 libllama-server-impl.dylib 0x000000010160d6d8 _ZN19server_context_impl12update_slotsEv + 46548
10 libllama-server-impl.dylib 0x00000001015cb264 _ZN12server_queue10start_loopEx + 500
11 libllama-server-impl.dylib 0x000000010151e1a4 _Z12llama_serveriPPc + 16656
12 dyld 0x00000001878bfda4 start + 6992
libc++abi: terminating due to uncaught exception of type std::out_of_range: vector
zsh: abort ./llama-server -m "/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-Q4_K_M.gguf"
tien@TiensMacStudio beellama-v0.3.0 %
Name and Version
This is all from my Mac studio m4 max 36gb:
Operating systems
Mac
GGML backends
Metal
Hardware
0.3.1
Models
No response
Problem description & steps to reproduce
I tried like https://github.com/Anbeeld/beellama.cpp/blob/main/docs/quickstart-qwen36-dflash.md
but not work
First Bad Commit
No response
Relevant log output
tien@TiensMacStudio beellama-v0.3.0 % ./llama-server
-m "/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-Q4_K_M.gguf"
--spec-draft-model "/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-DFlash-Q4_K_M.gguf"
--spec-type dflash
--spec-dflash-cross-ctx 512
--port 8080
-np 1
--kv-unified
-ngl all
--spec-draft-ngl all
-b 2048 -ub 512
--ctx-size 102400
--cache-type-k q5_0 --cache-type-v q4_1
--flash-attn on
--jinja
--no-mmap --mlock
--no-host
--reasoning on
--chat-template-kwargs '{"preserve_thinking":true}'
--temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0
0.00.035.194 I dflash: setting -cd to 256 (drafter doesn't need the full main ctx; pass -cd N to override)
0.00.035.219 I log_info: verbosity = 3 (adjust with the
-lv NCLI arg)0.00.035.220 I device_info:
0.00.035.224 I - MTL0 : Apple M4 Max (28753 MiB, 28753 MiB free)
0.00.035.224 I - BLAS : Accelerate (0 MiB, 0 MiB free)
0.00.035.228 I - CPU : Apple M4 Max (36864 MiB, 36864 MiB free)
0.00.035.238 I system_info: n_threads = 10 (n_threads_batch = 10) / 14 | MTL : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | REPACK = 1 |
0.00.035.249 I srv init: running without SSL
0.00.035.285 I srv init: using 13 threads for HTTP server
0.00.035.333 I srv start: binding port with default address family
0.00.036.654 I srv llama_server: loading model
0.00.036.679 I srv load_model: loading model '/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-Q4_K_M.gguf'
0.00.208.967 I srv load_model: [spec] estimated memory usage of draft model is 2839.85 MiB
0.00.208.976 I common_init_result: fitting params to device memory ...
0.00.208.977 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.05.254.594 W llama_context: n_ctx_seq (102400) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.05.711.047 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.06.600.998 I srv load_model: shrunk recurrent state to 1 cells before draft load (deferred 1 backup cells)
0.06.601.002 I srv load_model: loading draft model '/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-DFlash-Q4_K_M.gguf'
0.06.601.013 I srv load_model: DFlash draft model will use target output device MTL0 by default; pass --spec-draft-device to override
0.07.004.778 I srv load_model: expanded recurrent state to 2 cells before speculative GPU buffers
0.07.004.783 I srv load_model: initializing slots, n_slots = 1
0.07.137.424 W seq_rm: RS-ROLLBACK-OVERFLOW seq=0 p0=1 cell.pos=1 rollback=1 > n_rs_seq=0 -> CANNOT ROLLBACK
0.07.148.342 W srv load_model: speculative decoding will use checkpoints
0.07.148.343 I srv load_model: DFlash enabled for all 1 slots
0.07.148.363 W llama_context: n_ctx_seq (256) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.07.257.771 I common_speculative_create_ctx_dft: draft model warmup complete
0.07.257.791 I common_speculative_init: adding implementation dflash
0.07.257.815 I dflash: contract ok: block_size=16 mask_token=248070 target_layer_ids=[1,16,31,46,61] n_target_layers=5 n_embd=5120 n_target_features=25600 target_layers=65 cross_ctx=512
0.07.257.817 I dflash: target/drafter info: target_ctx_train=262144 target_vocab=248320 drafter_vocab=248320 vocab_match=1 capture_min=1 capture_max=61
0.07.284.759 I dflash: GPU hidden capture policy: allowed=1 forced_cpu=0 requested=1 target_devices=1 drafter_devices=1
0.07.284.805 W dflash: GPU cross ring unavailable; using CPU hidden capture
0.07.285.737 I slot load_model: id 0 | task -1 | speculative decoding context initialized
0.07.285.738 I slot load_model: id 0 | task -1 | new slot, n_ctx = 102400
0.07.285.824 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
0.07.285.825 I srv load_model: use
--cache-ram 0to disable the prompt cache0.07.285.825 I srv load_model: for more info see ggml-org#16391
0.07.285.825 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
0.07.285.890 I srv init: idle slots will be saved to prompt cache and cleared upon starting a new task
0.07.296.320 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
0.07.302.530 I srv init: init: chat template, thinking = 1
0.07.302.543 I srv llama_server: model loaded
0.07.302.547 I srv llama_server: server is listening on http://127.0.0.1:8080
0.07.302.551 I srv update_slots: all slots are idle
0.47.174.111 I srv params_from_: Chat format: peg-native
0.47.174.391 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
0.47.174.394 I slot get_availabl: id 0 | task -1 | adaptive dm: reset state for LRU slot selection
0.47.194.983 I srv recurrent_sh: shrunk recurrent state to 1 cells for prompt cache (before prompt cache save/load, removed 1 backup cells)
0.47.194.987 I srv get_availabl: updating prompt cache
0.47.194.995 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.47.194.997 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 102400 tokens, 8589934592 est)
0.47.219.429 I srv recurrent_ex: expanded recurrent state to 2 cells after prompt cache (after prompt cache save/load)
0.47.219.436 I srv get_availabl: prompt cache update took 24.45 ms
0.47.219.479 I reasoning-budget: activated, budget=2147483647 tokens
0.47.219.481 I reasoning-budget: deactivated (natural end)
0.47.219.491 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.47.658.221 I slot create_check: id 0 | task 0 | created context checkpoint 1 of 32 (pos_min = 12, pos_max = 12, n_tokens = 13, size = 150.896 MiB)
0.48.079.230 I slot operator(): id 0 | task 0 | adaptive dm profit: cur=0 recommended=4 score=13.5 action=apply
WARNING: Using native backtrace. Set GGML_BACKTRACE_LLDB for more info.
WARNING: GGML_BACKTRACE_LLDB may cause native MacOS Terminal.app to crash.
See: ggml-org#17869
0 libggml-base.0.13.1.dylib 0x00000001011162c0 ggml_print_backtrace + 276
1 libggml-base.0.13.1.dylib 0x000000010112cef4 _ZL23ggml_uncaught_exceptionv + 12
2 libc++abi.dylib 0x0000000187c2f75c _ZSt11__terminatePFvvE + 16
3 libc++abi.dylib 0x0000000187c31be4 __cxa_get_exception_ptr + 0
4 libc++abi.dylib 0x0000000187c1e09c __cxa_get_globals + 0
5 libllama.0.0.10070.dylib 0x0000000100d79ed4 _ZNSt3__120__throw_out_of_rangeB8ue170006EPKc + 60
6 libllama.0.0.10070.dylib 0x0000000100e64234 ZNKSt3__112__hash_tableINS_17__hash_value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEiEENS_22__unordered_map_hasherIS7_S8_NS_4hashIS7_EENS_8equal_toIS7_EELb1EEENS_21__unordered_map_equalIS7_S8_SD_SB_Lb1EEENS5_IS8_EEE4findIS7_EENS_21__hash_const_iteratorIPNS_11__hash_nodeIS8_PvEEEERKT + 0
7 libllama.0.0.10070.dylib 0x0000000100e5bae4 _ZNK11llama_vocab4impl14token_to_pieceEiPciib + 1300
8 libllama-common.0.0.10070.dylib 0x00000001021802b0 _Z21common_token_to_piecePK11llama_vocabib + 112
9 libllama-server-impl.dylib 0x000000010160d6d8 _ZN19server_context_impl12update_slotsEv + 46548
10 libllama-server-impl.dylib 0x00000001015cb264 _ZN12server_queue10start_loopEx + 500
11 libllama-server-impl.dylib 0x000000010151e1a4 _Z12llama_serveriPPc + 16656
12 dyld 0x00000001878bfda4 start + 6992
libc++abi: terminating due to uncaught exception of type std::out_of_range: vector
zsh: abort ./llama-server -m "/Users/tien/Setup/AI/unsloth/Qwen3.6-27B-Q4_K_M.gguf"
tien@TiensMacStudio beellama-v0.3.0 %