inference-engineering

Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

Rust + cuTile research prototype for paged latent-cache LLM decode attention, validated on an RTX 4060.

A practical handbook for software engineers to learn AI, Large Language Models (LLMs), and Inference Engineering—from fundamentals to production systems.

CPU-only conversational voice assistant with VAD-based speech segmentation, streaming LLM response generation overlapped with concurrent TTS synthesis via producer-consumer queues. Batch STT (Whisper). Fully instrumented with per-stage latency benchmarking.

it's me as a repository

LLM inference benchmarking dashboard: Python FastAPI backend with async orchestration, WebSocket live TTFT/TBT/throughput comparison across configs (512/128 to 4096/1024 tokens), Grafana + Docker Compose stack, GitHub Actions CI; 21/21 pytest passing.

Trace-Aware Serving Controller: eval-gated inference policy optimization

An interactive playground for learning inference engineering—explore LLM serving concepts, tune the stack, and graduate to production incidents.

Offline, evidence-first bottleneck hypotheses for LLM inference traces

Deterministic simulator for KV-cache admission, placement, movement, eviction, and recomputation policies

Evidence-gated benchmark, comparison and promotion control plane for LLM inference serving changes.

Deep Agents and SvelteKit harness for authoring verifier-gated Bonsai workflow packs.

Executable field guide and deterministic labs for LLM inference engineering across kernels, scheduling, KV cache, placement, and control.

Hands-on LLM inference labs on free T4 GPUs with vLLM: prefix caching, AWQ quantization, throughput knee, chunked prefill, KV-cache pressure. Every number reproducible.

TypeScript inference SDK for self-hosted LLM, ASR, TTS and embeddings. GPU lifecycle coordination, model handoffs and clients for vLLM, Whisper, Chatterbox, Qwen3 and Kokoro.

Interactive inference-engineering lessons and a bilingual AI mentor, built with Next.js and the Vercel AI SDK.

A hands-on lab for how AI models get served: GPU memory, batching, the scheduler, parallelism and fleet economics. Drag sliders instead of reading the book.