ML Stack

A local machine learning stack for model inference, fine-tuning, and evaluation.

Model inference • Fine-tuning • Evaluation — all local, all modular

Hardware target: Optimized for NVIDIA RTX 5090 (32 GB VRAM) with CUDA ≥ 13.

Training environment: The training env uses PyTorch 2.4.0 + CUDA 12.1. A separate training-cuda13 env (PyTorch cu130 + CUDA 13.0) is available for optimal Flash Attention 2 performance on the RTX 5090. It uses a community-built flash-attn 2.8.3 cu130 wheel (source). Switch between environments with ml use-training-env. See Conda Environments for setup.

⚙️ Requirements • 🚀 Quick Start • 🛠️ CLI Commands • 📚 Model Storage • 🐳 Docker Services • 🔍 Web Search • 📖 Training • 🐍 Conda Environments • 🤖 Claude Code • 📊 Evaluation • ⚡ GPU Optimization • Runtimes

⚙️ Requirements

Hardware

Component	Requirement	Notes
GPU	1× NVIDIA RTX 5090 (32 GB VRAM)	Blackwell GB202; lower-end GPUs may not fit 27B int4 models
CPU	8+ cores	For dataset processing and llama.cpp multi-threading
RAM	32 GB	Shared between system, inference and training; running both simultaneously will be tight
Storage	~50 GB free	Model weights, datasets, conda envs, Docker images

Software

Component	Version	Notes
OS	Linux / WSL2	Ubuntu 22.04+ recommended
NVIDIA Driver	≥ 580.x	Required for CUDA 13 runtime in the vLLM Docker image
CUDA Toolkit	≥ 13	System CUDA ≥ 13 is backward-compatible with PyTorch CUDA 12.1 wheels
Docker	Latest stable	With NVIDIA Container Toolkit installed for GPU passthrough
Git	≥ 2.x	For cloning and model management
Conda (Miniconda)	Latest	For `training` and `inference-vllm` environments
Python	3.10+	3.10 for stable envs, 3.12 for experimental training env

Optional

Component	Purpose
`huggingface_hub` (hf CLI)	Model/dataset downloads (`pip install huggingface_hub`)
`lm-eval`	Benchmarking (`pip install 'lm-eval[multitask]'`)

NVIDIA Drivers

Windows / WSL2 — NVIDIA App (Recommended)

Install the NVIDIA App (or the legacy GeForce driver installer) — both provide the correct driver with CUDA support. On WSL2, the driver is shared with the Windows host, so no additional setup is needed inside WSL2.

Linux (Ubuntu) — Open Kernel Module

On Ubuntu 24.04+ with RTX 50-series (Blackwell), the open kernel module (nvidia-driver-open) is required for driver stability. The proprietary/proprietary-open drivers may fail to detect the GPU.

# Ensure Secure Boot is disabled (required for open driver on fresh installs)
# Check current driver status
ubuntu-drivers devices

# Install the open driver
sudo apt install nvidia-driver-open

# After reboot, verify
nvidia-smi

For CUDA, install the toolkit from your distro's repos (e.g. cuda-toolkit-13 for CUDA 13.x) and add to your ~/.bashrc:

export PATH="/usr/local/cuda-13.3/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-13.3/lib64:${LD_LIBRARY_PATH}"

Troubleshooting: If nvidia-smi shows no GPU or returns "No devices found," switch to nvidia-driver-open (see above). On systems without an iGPU, you cannot enroll MOK keys at boot (no display), so either disable Secure Boot or use the open driver which doesn't require key enrollment.

🚀 Quick Start

1. Clone the repository:

git clone https://github.com/PhilipEriksson/ml-stack.git
cd ml-stack

2. Run the setup check:

bash scripts/init.sh

This checks your system (Python, pip, jq, curl, GPU, Docker, conda, huggingface_hub, llama.cpp, Docker images) and reports what's missing.

3. Auto-install what it can:

bash scripts/init.sh --install

This will:

Add ml to your PATH (in ~/.bashrc)
Install missing system tools (jq)
Install Python packages (huggingface_hub)
Create Docker wrapper on WSL2 (if Docker Desktop is installed but not in PATH)
Create missing conda environments
Build missing Docker images (vllm, api-webui)
Create missing project directories

Note: Docker Desktop itself must be installed manually (download).

4. Source your shell:

source ~/.bashrc
ml          # shows all available commands
ml help add-model

5. Launch vLLM inference:

docker compose -f services/docker/docker-compose.yml up -d vllm

The first launch will download the model weights (~20 GB for a 27B int4 model). Wait for "Application startup complete" in the logs.

Performance: Qwen3.6-27B int4 AutoRound with MTP speculative decoding delivers 100+ tokens/sec on a single RTX 5090 (32 GB) with 256K context.

6. Verify it's working:

curl http://localhost:8080/v1/models | jq

7. Optional — start the web interface:

docker compose -f services/docker/docker-compose.yml up -d api-webui

Open http://localhost:3000 for the Open WebUI.

📁 Project Structure

.
├── cli/                ← CLI entry point (the `ml` command)
│   └── ml
├── configs/            ← JSON registries and configuration files
│   ├── datasets/       ← dataset registry
│   ├── dflash/         ← dflash_server environment config (server.env)
│   ├── evals/          ← benchmark eval registry (created on first `ml eval`)
│   ├── llama/          ← active llama.cpp model state (created by `serve-model`)
│   ├── models/         ← model registry (used by `serve-model`)
│   ├── runs/           ← training run registry
│   ├── searxng/        ← SearXNG search configuration
│   └── vllm/           ← vLLM environment config files (.env)
├── datasets/           ← downloaded datasets (raw/ + processed/)
├── engine/             ← script orchestration
│   └── runner.sh       ← dispatches commands to scripts/
├── envs/               ← conda environment YAML files
│   ├── training.yml    ← PyTorch + Unsloth + PEFT + TRL + huggingface_hub
│   └── inference-vllm.yml    ← vLLM inference (conda env: inference-vllm)
├── models/             ← local model storage
│   ├── base/           ← full-precision, unmodified weights
│   ├── quantized/      ← compressed model variants (GGUF, GPTQ, AWQ, etc)
│   └── finetuned/      ← LoRA adapters and merged fine-tuned models
├── outputs/            ← experiment outputs and training artifacts
├── runtimes/           ← third-party runtime dependencies (e.g. llama.cpp)
├── scripts/            ← entrypoints and orchestration
│   ├── build.sh        ← alias for docker compose build
│   ├── init.sh         ← one-time setup (adds `ml` to PATH)
│   ├── train/          ← training scripts (finetune.py)
│   ├── eval/           ← benchmark scripts (run-benchmark)
│   └── utils/          ← utility scripts
│       ├── add-model       ← download and register models from HF
│       ├── add-dataset     ← download and register datasets from HF
│       ├── compare-runs    ← compare two training runs
│       ├── kill-training   ← find and kill running finetune.py processes
│       ├── ml-runs         ← list all recorded training runs
│       ├── process-dataset  ← convert raw datasets to alpaca format (instruction, input, output)
│       ├── sample-dataset   ← create a smaller version of a processed dataset for quick tests
│       ├── serve-model     ← serve a registered GGUF model via llama.cpp
│       ├── set-vllm-env    ← set the active vLLM env file
│       ├── use-training-env ← switch the active training conda environment
│       ├── create-training-run  ← create and register a new training run
│       └── execute-training-run ← run the training in the conda training env
└── services/           ← Docker services
    ├── api-webui/      ← FastAPI proxy + Open WebUI
    ├── docker/         ← Docker Compose orchestration
    │   └── docker-compose.yml
    ├── searxng/        ← SearXNG search engine
    └── vllm/           ← vLLM Docker service (env-driven config)

🛠️ CLI Commands

The ml command (in cli/ml) is the unified entry point for all operations. Add it to your PATH:

export PATH="$HOME/ml-stack/cli:$PATH"

Model Management

Download and register a model from Hugging Face (auto-detect):

ml add-model auto <hf-repo>

Auto-detects the family (qwen, llama3), quantization format, and bit-width. Places the model in the correct folder under models/.

# Examples:
ml add-model auto Lorbus/Qwen3.6-27B-int4-AutoRound
# → models/quantized/qwen/auto-round/4bit/...

ml add-model auto bartowski/Llama-3.1-8B-Instruct-GGUF
# → models/quantized/llama3/gguf/...

Download with manual family + variant names:

ml add-model <family> <variant> <hf-repo>

# Examples:
ml add-model qwen qwen3.6-27b Lorbus/Qwen3.6-27B-int4-AutoRound
ml add-model qwen qwen3.5-32b HuggingFaceTB/SmolLM2-1.7B-Instruct --type base

Override auto-detection type:

ml add-model auto <hf-repo> --type <base|quantized|finetuned>

Serving Models

Serve a GGUF model via llama.cpp (local):

ml serve-model <family> <variant>

# Example:
ml serve-model qwen qwen3.6-27b-gguf

The script reads the registry entry, resolves the .gguf file path, and launches llama-server from runtimes/llama.cpp/.

VRAM safety: Before launching, the script checks if the vLLM Docker container is running. If so, it warns you and offers to stop it — running both vLLM and llama.cpp simultaneously on a consumer GPU will likely exceed your VRAM.

For HF models (serve via Docker vLLM):

docker compose -f services/docker/docker-compose.yml up -d vllm

This launches vLLM in Docker with the model from configs/vllm/qwen3.6-27b-int4.env. Swap models by changing the .env file and rebuilding.

Optional flags for serve-model:

--ctx <n>     Context length (auto-detected from GPU VRAM, default 16384)
--ngl <n>     Number of GPU layers (default 999 = all layers on GPU)
--port <n>    Port to serve on (default 8000)
--host <addr> Bind address (default 0.0.0.0)
--fa          Enable flash attention

If the registry entry has an mmproj field (vision projection), it is auto-loaded.

Dataset Management

Download a dataset from Hugging Face:

ml add-dataset <family> <hf-dataset-repo>

# Example:
ml add-dataset qwen HuggingFaceTB/cnn_dailymail
# → datasets/raw/qwen/cnn_dailymail/

Convert to alpaca format for training:

ml process-dataset <family> <dataset-name>

Auto-detects column mapping from common dataset patterns. Override with --instruction, --input, --output flags. Outputs alpaca-format dataset (instruction, input, output) to datasets/processed/ with train/ and eval/ splits.

# Examples:
ml process-dataset qwen cnn_dailymail
#   auto-detects: instruction=(none), input=article, output=highlights

ml process-dataset qwen my_alpaca --instruction prompt --output response

Training

Uses Unsloth's FastLanguageModel for efficient 4-bit LoRA fine-tuning with SFTTrainer. The dataset is converted from alpaca format into chat messages and formatted with the model's native chat template.

Full workflow — download, process, train:

# 1. Download raw dataset
ml add-dataset qwen yahma/alpaca-cleaned

# 2. Convert to alpaca format (instruction, input, output)
ml process-dataset qwen alpaca-cleaned

# 3. Download a base model for fine-tuning
ml add-model auto Qwen/Qwen3-0.6B --type base

# 4. Create a training run
ml create-training-run my-run qwen/qwen3-0.6b qwen/alpaca-cleaned

# 5. Optionally switch training environment (default: training)
ml use-training-env                     # show current env + available options
ml use-training-env training-cuda13     # switch to CUDA 13 env (Flash Attention 2)
ml use-training-env training            # switch back to stable env

# 6. Execute the training run
ml execute-training-run my-run

# 7. Kill a running training job
ml kill-training

Quick Test Run

For testing the pipeline before committing to a full training run, sample a small subset of the dataset. Datasets under 10K examples automatically trigger fast settings (1 epoch, batch=8, sparse logging) — reducing training from ~40 min to ~30 seconds.

# Create a 1000-example version of the processed dataset
ml sample-dataset qwen alpaca-cleaned 1000

# Use it for a fast test run
ml create-training-run my-test qwen/qwen3-0.6b qwen/alpaca-cleaned-1000
ml execute-training-run my-test

Sample sizes for different use cases:

Samples	Train/eval split	Est. time (0.6B, RTX 5090)	Use case
1,000	1,000 / 50	~30 sec	Pipeline smoke test
5,000	5,000 / 250	~2 min	Quick sanity check
10,000	10,000 / 500	~4 min	Preliminary results
51,760 (full)	49,172 / 2,588	~40 min	Production run

Force fast settings on a full dataset:

ml execute-training-run my-run --sample

Recommended for Qwen3-0.6B: Start with a 1,000 or 5,000 sample to verify the pipeline, then run the full dataset with ml execute-training-run my-run (4 epochs, batch=2) once satisfied.

Run Tracking

List all training runs:

ml runs

Compare two runs:

ml compare-runs <run1> <run2>

Shows model version, dataset, and model hash for each run. Warns if both use identical weights.

Help

ml                # Shows global help with all commands
ml help <cmd>     # Shows detailed help for a specific command

📚 Model Storage Structure

Models are organized by category (base/quantized/finetuned), then by family (qwen, llama3), then by format/bit-width.

`models/base/` — Full-Precision Weights

Original, unmodified model weights. Used as the starting point for fine-tuning.

base/
└── <family>/
    └── <variant>/
        ├── config.json
        ├── *.safetensors
        ├── tokenizer.json
        └── ...

`models/quantized/` — Compressed Variants

Organized by quantization format and bit-width:

quantized/
└── <family>/
    ├── gguf/                ← GGUF format (bit in filename: Q3_K_XL, Q4_K_M)
    │   └── <variant>/
    │       └── *.gguf
    │
    ├── auto-round/          ← AutoRound quantization
    │   ├── 4bit/
    │   ├── 8bit/
    │   └── other/
    │
    ├── gptq/                ← GPTQ quantization
    │   ├── 4bit/
    │   ├── 8bit/
    │   └── other/
    │
    ├── awq/                 ← AWQ quantization
    │   ├── 4bit/
    │   └── other/
    │
    ├── nvfp/                ← NVIDIA FP4/FP8 formats
    │   ├── fp4/
    │   └── fp8/
    │
    ├── mx/                  ← MX FP4/FP8 formats
    │   ├── fp4/
    │   └── fp8/
    │
    └── other/               ← exotic/custom quantizations (PARO, etc)

The add-model script auto-detects format and bit-width from the HF repo name and file contents, placing files in the correct folder.

`models/finetuned/` — LoRA & Merged Models

finetuned/
└── <family>/
    ├── lora/                ← LoRA adapter weights
    ├── checkpoints/         ← training checkpoints (saved per epoch)
    └── merged/              ← adapter merged back into base weights

Model Registry

configs/models/registry.json tracks all local models:

{
  "qwen": {
    "models": {
      "qwen3.6-27b-gguf": {
        "type": "gguf",
        "variant": "instruct",
        "path": "/path/to/model/dir",
        "file": "model.gguf",
        "mmproj": "/path/to/vision-projection.gguf",
        "status": "ready",
        "hash": "sha256..."
      }
    }
  }
}

Variant types:

Variant	Description	Auto-Detect Hint
`pretrained`	Raw token predictor (no fine-tuning)	No "Instruct"/"Reasoning" in name
`instruct`	SFT-finetuned for instruction following	Repo name contains `Instruct`
`reasoning`	CoT-trained + RLHF/RLVR for chain-of-thought	Repo name contains `Thinking`, `Reasoning`, `R1`

The serve-model script reads this registry to resolve the model path, type, and optional vision projection.

🐳 Docker Services

Docker Setup

Choose the Docker setup that matches your OS:

Windows / WSL2 — Docker Desktop (Recommended)

Install Docker Desktop for Windows, enable the WSL2 backend, and make sure the NVIDIA GPU is toggled on in Settings → General. Docker Desktop handles GPU passthrough automatically on WSL2 — no manual configuration needed.

Linux Native — Docker Engine + NVIDIA Container Toolkit

Install Docker Engine from your distribution's repos and the NVIDIA Container Toolkit for GPU passthrough:

# Ubuntu/Debian
sudo apt install docker.io nvidia-container-toolkit
sudo systemctl enable --now docker
# Configure nvidia as the default runtime
sudo mkdir -p /etc/docker
echo '{"runtimes":{"nvidia":{"path":"nvidia-container-runtime","args":[]}}}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker

For a terminal-based UI to manage containers (alternative to Docker Desktop's GUI), install LazyDocker:

# Download the latest release
LATEST=$(curl -s https://api.github.com/repos/jesseduffield/lazydocker/releases/latest | grep -oP '"tag_name":\K[^"]+')
curl -L https://github.com/jesseduffield/lazydocker/releases/download/${LATEST}/lazydocker_${LATEST#v}_Linux_x86_64.tar.gz | tar xz
sudo mv lazydocker /usr/local/bin/
lazydocker --version

Docker Desktop on Linux (Optional)

Docker Desktop is also available for Linux and works as a drop-in replacement. If you prefer the GUI, install it from docker.com — the NVIDIA Container Toolkit is still required for GPU access.

Auto-Start with Windows Task Scheduler

On WSL2, Docker containers don't survive a reboot. To auto-start vLLM and API+WebUI on login, create two scheduled tasks in Windows:

Start vLLM on login:

schtasks /create /tn "ML-Stack-vLLM" /tr "wsl bash -c 'cd /home/YOUR_USER/ml-stack && docker compose -f services/docker/docker-compose.yml up -d vllm'" /sc onlogon /ru YOUR_USER

Start API + WebUI on login:

schtasks /create /tn "ML-Stack-API-WebUI" /tr "wsl bash -c 'cd /home/YOUR_USER/ml-stack && docker compose -f services/docker/docker-compose.yml up -d api-webui'" /sc onlogon /ru YOUR_USER

Note: Replace YOUR_USER with your actual WSL username. Also replace docker with "/mnt/c/Program Files/Docker/Docker/resources/bin/docker.exe" if you use the WSL2 wrapper script.

Architecture

Three modular Docker services:

Service	Dockerfile	Description
vllm	`services/vllm/Dockerfile`	vLLM model serving (GPU)
searxng	`services/searxng/Dockerfile`	SearXNG metasearch engine
api-webui	`services/api-webui/Dockerfile`	FastAPI proxy + Open WebUI

vLLM Service

Based on vllm/vllm-openai:v0.20.1. All model settings are driven by environment variables — no rebuild needed to swap models.

Config file: configs/vllm/qwen3.6-27b-int4.env

Contains all vLLM flags (MODEL_NAME, MAX_MODEL_LEN, GPU_MEMORY_UTILIZATION, etc). To swap models, create a new .env file with different settings.

How it works:

The start.sh entrypoint reads env vars and builds the vllm serve command
HF_HOME=/hf-cache is set; the HF cache is volume-mounted from the host
vLLM resolves MODEL_NAME through its native HF model resolution

Env file variables:

Variable	Description
`MODEL_NAME`	HF model name or local path
`MAX_MODEL_LEN`	Maximum context length
`GPU_MEMORY_UTILIZATION`	GPU memory fraction (0.0–1.0)
`ATTENTION_BACKEND`	e.g. `flashinfer`
`PERFORMANCE_MODE`	e.g. `interactivity`
`LANGUAGE_MODEL_ONLY`	`true` / omitted
`KV_CACHE_DTYPE`	e.g. `fp8_e4m3`
`MAX_NUM_SEQS`	Max concurrent sequences
`SKIP_MM_PROFILING`	`true` / omitted
`QUANTIZATION`	e.g. `auto_round`
`REASONING_PARSER`	e.g. `qwen3`
`ENABLE_AUTO_TOOL_CHOICE`	`true` / omitted
`TOOL_CALL_PARSER`	e.g. `qwen3_coder`
`ENABLE_PREFIX_CACHING`	`true` / omitted
`ENABLE_CHUNKED_PREFILL`	`true` / omitted
`SPECULATIVE_CONFIG`	JSON config string
`HOST`	Bind address
`PORT`	Container port (default 8000)

Swap models at runtime:

Option A — from HF cache (model already downloaded):

# Create configs/vllm/other-model.env with new MODEL_NAME + settings
# MODEL_NAME can be an HF repo ID (uses ~/.cache/huggingface/hub)
docker compose -f services/docker/docker-compose.yml --env-file configs/vllm/other-model.env up -d vllm

Option B — from local model storage (~/ml-stack/models/):

# Point MODEL_NAME to the path inside the container:
MODEL_NAME=/opt/models/quantized/qwen/auto-round/4bit/your-model-folder

Both the HF cache (~/.cache/huggingface/hub) and local model directory (~/ml-stack/models) are mounted read-only into the container.

API + WebUI Service

Based on ghcr.io/open-webui/open-webui:main. Adds a FastAPI proxy that:

Translates between OpenAI API format and Responses API format
Proxies all requests to the vLLM backend
Runs on port 8000 alongside Open WebUI on port 8080

Ports:

Host Port	Container Port	Service
8080	8000	vLLM (direct API access)
8000	8000	FastAPI proxy
3000	8080	Open WebUI (web interface)
8888	8080	SearXNG (direct search access)

Running with Docker Compose

# Build images
docker compose -f services/docker/docker-compose.yml build

# Start everything
docker compose -f services/docker/docker-compose.yml up -d

# Start only vLLM
docker compose -f services/docker/docker-compose.yml up -d vllm

# Start only the API+WebUI
docker compose -f services/docker/docker-compose.yml up -d api-webui

# Stop everything
docker compose -f services/docker/docker-compose.yml down

# View logs
docker compose -f services/docker/docker-compose.yml logs -f vllm
docker compose -f services/docker/docker-compose.yml logs -f api-webui

Web Search

The stack includes a SearXNG search service that provides web search results to the LLM via the FastAPI proxy. SearXNG is a metasearch engine aggregating results from 70+ search engines (Google, Bing, Wikipedia, and more). A DuckDuckGo fallback is available if SearXNG is unavailable.

Architecture:

Service	Host Port	Container Port	Purpose
vllm	8080	8000	vLLM model serving (direct API)
api-webui	8000	8000	FastAPI proxy + Open WebUI (8080→3000)
searxng	8888	8080	SearXNG search engine

The FastAPI proxy at port 8000 routes search requests to the SearXNG container and exposes a clean REST API with an OpenAPI spec for auto-discovery.

Enabling web search in Open WebUI:

Open Open WebUI (port 3000) and navigate to Settings → Integrations → Tool Servers
Click Add New and enter the following:
- URL: http://<hostname>:8000 — Use your server's hostname or IP. If accessing over Tailscale from another machine, use your Tailscale URL (e.g., http://p5090.taila6ea3a.ts.net:8000). If accessing locally, use http://localhost:8000
- Name: Web Search
Save — Open WebUI will auto-discover the search_web tool from the OpenAPI spec at /openapi.json

Why not localhost when accessing remotely? The Tool Server URL is resolved by your browser, not the server. When connecting from a MacBook over Tailscale, localhost resolves to the MacBook — not the server hosting the containers.

Search endpoint:

# Search via GET
curl "http://localhost:8000/search?q=what+is+the+current+date"

# Search via POST
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "DDR5 RAM prices", "max_results": 5}'

Start all services including search:

docker compose -f services/docker/docker-compose.yml up -d

🤖 Using with Claude Code

init.sh --install automatically sets up claude-local in your ~/.bashrc. It dynamically detects your active inference backend — no manual config needed.

How it works:

vLLM Docker — claude-local checks if the vllm-server container is running and reads MODEL_NAME from its environment
llama.cpp (GGUF) — When you run ml serve-model, it records the active model. claude-local reads this on next launch

Switch models for vLLM:

# Point to a different vLLM env file
ml set-vllm-env qwen3.6-27b-int4.env
docker compose -f services/docker/docker-compose.yml --env-file configs/vllm/qwen3.6-27b-int4.env up -d vllm
claude-local   # picks up the new model automatically

Serve a GGUF model and connect:

ml serve-model qwen qwen3.6-27b-gguf
# In another terminal:
claude-local   # detects the running llama.cpp server

Note: The .claude_env file in ~/ml-stack/ is created by init.sh and sources scripts/utils/get-claude-env to detect the active model. Don't edit it manually.

🐍 Conda Environments

File	Conda Env Name	Purpose
`envs/inference-vllm.yml`	`inference-vllm`	vLLM inference
`envs/training.yml`	`training`	PyTorch 2.4.0 + CUDA 12.1 + Unsloth
(manual)	`training-cuda13`	PyTorch (cu130) + CUDA 13.0 + Flash Attention 2

# Create environments
conda env create -f envs/inference-vllm.yml
conda env create -f envs/training.yml

# Optional: CUDA 13 training environment (recommended for RTX 5090)
conda create -n training-cuda13 python=3.12 -y
conda activate training-cuda13
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130
# Install the community flash-attn cu130 wheel (get latest from flash-attn GitHub releases)
pip install flash-attn --no-cache-dir
pip install unsloth --no-deps
pip install unsloth_zoo --no-deps
pip install trl datasets accelerate peft sentence-transformers
# Switch to use it:
ml use-training-env training-cuda13

📊 Evaluation

Evaluate your local model against standard benchmarks using lm-eval.

Install once:

pip install 'lm-eval[multitask]'
# or inside your conda env:
conda activate training
pip install 'lm-eval[multitask]'

Run against your active backend:

# Default suite (MMLU, GSM8K, HellaSwag, ARC-Challenge)
ml eval

# Specific tasks
ml eval mmlu,gsm8k

# With custom batch size (higher = faster but more VRAM)
ml eval mmlu,ifeval 8

The ml eval script automatically detects your running vLLM container or llama.cpp server, sends all benchmark prompts through your API, and saves result JSONs to outputs/evals/.

Available Tasks

Task	Description	Approx. Questions	Time on RTX 5090
`mmlu`	Massive Multitask Language Understanding (57 subjects)	~5,700	~10 min
`gsm8k`	Grade School Math	1,319	~3 min
`hellaswag`	Commonsense NLI	10,042	~15 min
`arc_challenge`	AI2 Reasoning Challenge (hard)	1,102	~3 min
`arc_easy`	AI2 Reasoning Challenge (easy)	2,258	~5 min
`truthfulqa_gen`	TruthfulQA (generation)	817	~2 min
`ifeval`	Instruction Following Eval	3,132	~8 min
`winogrande`	Winograd Schema	1,267	~3 min
`bbh`	Beyond Benchmark-Hard (17 tasks)	~3,900	~10 min

Note: Times are approximate for a 27B int4 model at batch_size=4. Your results may vary.

Benchmark datasets (~200 MB total) are downloaded on first run and cached in ~/.cache/huggingface/datasets/.

Eval Registry

Each benchmark run is recorded in configs/evals/registry.json with the model, backend, timestamp, tasks, scores, and result file path. This lets you track how models improve over time without digging through raw JSON files.

# View all recorded evals
ml evals

# View a single eval result in detail
cat outputs/evals/qwen3.6-27b-int4-2026-05-08_22-00.json | jq .results

# Compare two results (manually or with a diff tool)
diff <(jq '.results | keys' file1.json) <(jq '.results | keys' file2.json)

⚡ GPU Optimization

Power Management

The RTX 5090 has a 575W TDP by default. For quieter, cooler operation (especially during long eval runs or idle serving), you can lower the power limit.

Set a power limit:

# Check current power limit
nvidia-smi -q | grep "Power"

# Set a custom power limit (watts)
sudo nvidia-smi -pl 400

# Reset to default
sudo nvidia-smi -pl 575

Recommended: 400W is the best everyday power limit on both Linux and Windows — noticeably cooler and quieter with only a minor performance hit (~7-12%). The 400W floor is hardware-enforced on both platforms. On Windows/WSL2 you can combine it with undervolting (MSI Afterburner) for better performance-per-watt. On Linux, nvidia-smi doesn't expose voltage curves.

Common power profiles:

Target Power	Use Case	Performance Impact
575W (default)	Max throughput (training, large batch evals)	100% baseline
450W	Balanced — good for serving with lower noise	~5-10% slower token gen
400W (recommended)	Best overall — quiet serving, minimal perf loss	~7-12% slower alone

Note: The RTX 5090's minimum power limit is 400W (hardware-enforced on both Linux and Windows). On Linux, set it via sudo nvidia-smi -pl 400. On Windows/WSL2, set the MSI Afterburner power slider to 69% (~400W). Undervolting doesn't let you go below 400W — it adjusts the voltage/frequency curve for better performance-per-watt at the same power limit (see below).

Make power limit persistent:

# Add to ~/.bashrc
sudo nvidia-smi -pl 400

Clock Speed Management

For fine-grained control, you can also limit the maximum GPU clock:

# List supported clock speeds
nvidia-smi -q -d CLOCK | grep "Graphics"

# Set max graphics clock (MHz)
sudo nvidia-smi -lgc 2000

# Remove clock limit
sudo nvidia-smi -lgc 0

Undervolting (Advanced — WSL2 Only)

Undervolting is only available on Windows/WSL2. NVIDIA's Linux driver does not expose the voltage/frequency curve table, so tools like nvidia-smi on native Linux cannot undervolt. If you're on WSL2, apply undervolting from the Windows host — the GPU sees it regardless of which OS is running the workload.

For advanced users on WSL2, undervolting delivers near-default performance with better thermals and efficiency at the same 400W limit — the voltage/frequency curve lets the GPU maintain higher clocks per watt.

Suggested starting point for RTX 5090:

Core voltage cap: 875 mV at 2827 MHz
This retains boost clocks close to the default while dramatically reducing power consumption and temperatures

How to do it (Windows host): Use MSI Afterburner — open the Tweaker tab, go to Settings (gear icon), raise the voltage limit ceiling, then drag individual voltage/frequency curve points down. Set the target point to 875 mV @ 2827 MHz and flatten all higher-frequency points to the same voltage. In the Tweaker tab, set the Power Limit slider to 69% (≈ 400W on the RTX 5090) to cap power without nvidia-smi. Apply, save to profile, and verify stability under load.

vLLM Settings for RTX 5090

In your vLLM .env file, these settings affect power usage:

Variable	Power-Conscious	Performance	Description
`GPU_MEMORY_UTILIZATION`	`0.85`	`0.95`	Lower = less VRAM pressure, less compute work
`MAX_NUM_SEQS`	`1`	`4`	Fewer concurrent sequences = less power
`PERFORMANCE_MODE`	`interactivity`	`throughput`	Interactivity prioritizes low latency over batch size

For a power-conscious setup serving a single model interactively:

GPU_MEMORY_UTILIZATION=0.85
MAX_NUM_SEQS=1
PERFORMANCE_MODE=interactivity

Runtimes

Optimized configurations for specific GPU targets, validated with end-to-end benchmarks.

RTX 5090 — Long-Context Runtime

For workloads with 40–100K context on a single RTX 5090 (32 GB). Combines KVFlash (bounded residency), PFlash (prefill compression), and DFlash DDTree (speculative decode) from lucebox-hub.

Prerequisites:

Clone the lucebox-hub repo into runtimes/:

cd ~/ml-stack/runtimes
git clone https://github.com/Luce-Org/lucebox-hub.git
cd lucebox-hub
git submodule update --init --recursive

Build the server (follow runtimes/lucebox-hub/README.md for cmake/dependencies):

conda create -n dflash -y "gcc=15" cuda-toolkit ninja cmake make && conda activate dflash

cd ~/ml-stack/runtimes/lucebox-hub/server
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120 -DDFLASH27B_USER_CUDA_ARCHITECTURES=120 -DDFLASH27B_ENABLE_BSA=ON
make -j$(nproc)

Download the required drafter models (see table below).

Required models:

Model	Path	Purpose	Size
Qwen3.6-27B UD-Q5_K_XL	`models/base/Qwen3.6-27B-UD-Q5_K_XL.gguf`	Target model	~18 GB
DFlash Drafter	`models/drafter/model.safetensors`	Decode speculation	~3.3 GB
PFlash Drafter	`models/pflash_drafter/Qwen3-0.6B-BF16.gguf`	Prefill compression + KVFlash scoring	~1.1 GB

Quick start:

ml serve-dflash      # start
ml serve-dflash --stop    # stop
ml serve-dflash --restart # restart

Manual launch:

export DFLASH_FP_USE_BSA=1
export DFLASH_FP_ALPHA=0.70

# Required: conda dflash build env libraries (CUDA 13 runtime)
export LD_LIBRARY_PATH="/home/px5090/miniconda3/envs/dflash/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

source ~/ml-stack/configs/dflash/server.env

cd ~/ml-stack/runtimes/lucebox-hub/server/build
./dflash_server \
  ~/ml-stack/models/base/Qwen3.6-27B-UD-Q5_K_XL.gguf \
  --draft ~/ml-stack/models/drafter/model.safetensors \
  --prefill-drafter ~/ml-stack/models/pflash_drafter/Qwen3-0.6B-BF16.gguf \
  --ddtree --ddtree-budget 22 \
  --fa-window 4096 \
  --max-ctx 262144 \
  --kvflash 32768 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --prefill-compression auto \
  --prefill-threshold 32000 \
  --prefill-keep-ratio 0.05 \
  --default-max-tokens 1024 \
  --port 8080

Why these flags:

All values are set in configs/dflash/server.env — edit that file to adjust defaults.

Flag	Value	Reason
`--kvflash 32768`	32K pool	Holds most of a 60K context without eviction. `auto` caps at 16K (too small for 50K+ prompts).
`--cache-type-k q8_0` / `--cache-type-v q8_0`	q8_0	8-bit quantized KV cache. Faster decode than bf16, better quality than tq3_0. Uses ~3.2 GiB for the 32K pool.
`--prefill-compression auto`	`auto`	Activates at 32K+ tokens via the pflash drafter. Compresses a 96K prefill from 143s → 3.3s.
`--prefill-threshold 32000`	32000	PFlash only triggers above this token count. Below, the target runs natively (no overhead).
`--prefill-keep-ratio 0.05`	0.05	Keeps 5% of compressed tokens. Balances quality vs. size.
`--fa-window 4096`	4096	Sliding attention window. Drops system prompt from attention at long contexts.
`--ddtree-budget 22`	22	Tree verify budget. Stable on 5090.
`--default-max-tokens 1024`	1024	Response cap. Required for OpenWebUI compatibility.

Frontend proxy (for OpenWebUI):

Spec decode requires temperature=0 and top_p=1 (any other value disables the draft-verify alignment). The services/api-webui/proxy.env file forces these defaults:

# proxy.env
DEFAULT_MAX_TOKENS=1024
DEFAULT_TEMPERATURE=0
DEFAULT_TOP_P=1

Local GGUF Serving — llama.cpp

For lightweight local serving without Docker, use llama.cpp to run quantized GGUF models directly on your GPU. Models are first registered via ml add-model, then served with a single command.

Quick start:

ml serve-model <family> <variant>

For example:

ml serve-model qwen qwen3.6-27b-gguf

The script reads the model registry, resolves the .gguf file path, and launches llama-server from runtimes/llama.cpp/. Active state is tracked in llama/ so claude-local and other tools can auto-detect the running server.

VRAM safety: Before launching, serve-model checks if the vLLM Docker container is running and will warn you — running both simultaneously on a consumer GPU will exceed VRAM.

Why use llama.cpp vs. dflash:

	llama.cpp	dflash (KVFlash + PFlash)
Best for	Short prompts, interactive chat, lightweight setups	Long context (40K+)
Setup	Single command, no Docker	Multiple model files, env config
Speculative decode	Basic (mtp/draft)	DDTree with high accept rates
Prefill compression	None	PFlash (40x speedup at 96K)
KV flash/paging	None	32K pool with host RAM spillover

For workloads that build up long context over time, dflash is the better choice. For quick local serving or short conversations, llama.cpp is simpler to set up.

See Serving Models for serve-model flags and Claude Local for auto-detection details.

📦 Dependencies

See Requirements above for the full list.

Name		Name	Last commit message	Last commit date
cli		cli
configs		configs
datasets		datasets
engine		engine
envs		envs
models		models
outputs		outputs
runtimes		runtimes
scripts		scripts
services		services
unsloth_compiled_cache		unsloth_compiled_cache
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation