Basalt is a local inference engine for mixture-of-experts models: Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash. It runs on an NVIDIA Blackwell GPU (sm_120) on Linux and Windows x64, with the experts that do not fit in VRAM in system RAM (and optionally on more GPUs). The server speaks the OpenAI and Anthropic APIs. Website: basalt.jgulfo.com.

Help wanted: spare Blackwell compute for testing? See Contributing.

Requirements

• Linux x86-64 with AVX2, or Windows 10 1903+ x64 (platforms).
• An NVIDIA Blackwell GPU (compute capability 12.0) with CUDA 13.
• CMake 3.24+, GCC 13+ (Linux), Python 3.10+, and enough system RAM for the experts that do not fit in VRAM.

Quick start

Prebuilt Linux and Windows archives are attached to each release (what they need); otherwise build from source:

git clone https://github.com/jesdga95/basalt && cd basalt
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j                  # bin/basalt, bin/basalt-bench, bin/basalt-probe, bin/basalt-vision
pip install -r requirements.txt
hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4-DQ6_K.basalt --local-dir models

Put the settings in basalt_config.json:

{
  "version": 2,
  "models": [{ "model": "models/Qwen3.8-FN-NVFP4-DQ6_K.basalt", "gpu": "0,1", "kv": "int8" }],
  "engine": { "resident-experts": true },
  "server": { "port": 8080 }
}

bin/basalt serve --config basalt_config.json

The server listens on 127.0.0.1. Every setting: configuration. More examples: example configurations.

Models

	Qwen3.8-Flash-Next	DeepSeek-V4.1-Flash
Packs	default Qwen3.8-FN-NVFP4-DQ6_K, Hugging Face	default DeepSeek-V4.1-Flash-1017-UD-IQ1_M, Hugging Face
Context	default 8,192; --max-context sets it	default 8,192; --max-context sets it, up to 1,048,576
Drafting	MTP, on by default	DSpark, on by default
Images	yes	yes
Guide	guide	guide
Precision	precision	precision

All models compared: models.

Documentation

• Run: running, configuration, models and the router, sampling, image sources, diagnostics, basalt-probe, platforms.
• Models: models, Flash-Next, DeepSeek, the .basalt file, file format changes.
• Quality: precision, method.
• Build and develop: building, adding a model, community runs, changelog, contributing.

Credits

• Strata: the project Basalt forks (MIT).
• Qwen and DeepSeek: the models. Their packs carry the models' own licences (see the Hugging Face cards).
• unsloth (GGUFs), NVIDIA (NVFP4 experts).
• @giveen: the BF16 reference for the DeepSeek quality numbers.
• ggml and llama.cpp: quantization formats and kernels in src/ (MIT, THIRD_PARTY_LICENSES).

License

MIT, see LICENSE.