Local Inference Lab

Popular repositories Loading

1. rtx6kpro rtx6kpro Public

   RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink

   Python 1.2k 91

2. b12x b12x Public archive

   Archived: b12x development has moved to FlashInfer. See README for migration and PR-porting guidance.

   Python 269 69

3. llm-inference-bench llm-inference-bench Public

   LLM inference decode throughput benchmark with Rich TUI dashboard. Measures token generation speed across concurrency levels and context lengths. Supports SGLang and vLLM engines.

   Python 121 19

4. blackwell-llm-docker blackwell-llm-docker Public

   Docker images for LLM inference (SGLang + vLLM) on NVIDIA Blackwell GPUs (SM120, CUDA 13.2)

   Python 94 23

5. vllm vllm Public

   Forked from vllm-project/vllm

   A high-throughput and memory-efficient inference and serving engine for LLMs

   Python 42 36

6. quant-toolkit quant-toolkit Public

   Python 17 9