Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions _quarto-html.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,8 @@ website:
contents: "zh/ch12-llm-training-engineering/*.qmd"
- section: "Part 7: LLM Serving:vLLM 与 SGLang"
contents:
- section: "Chapter 19: SGLang:从请求调度到 Cache-Aware Serving"
contents: "zh/ch19-sglang-serving/*.qmd"
- section: "Chapter 18: vLLM:从 LLM 推理到高性能推理框架"
contents: "zh/ch18-vllm-introduction/*.qmd"
- id: en
style: floating
collapse-level: 1
Expand Down
4 changes: 2 additions & 2 deletions _quarto-typst-zh.yml
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,6 @@ book:
- "zh/ch12-llm-training-engineering/ch12.9-distributed-training.qmd"
- "zh/ch12-llm-training-engineering/ch12.10-large-model-checkpoint.qmd"
- "zh/ch12-llm-training-engineering/ch12.11-exercises.qmd"
- part: "Chapter 19: SGLang:从请求调度到 Cache-Aware Serving"
- part: "Chapter 18: vLLM:从 LLM 推理到高性能推理框架"
chapters:
- "zh/ch19-sglang-serving/ch19.1-from-vllm-to-sglang.qmd"
- "zh/ch18-vllm-introduction/ch18.1-vllm-overview.qmd"
9 changes: 9 additions & 0 deletions refs-article.bib
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,15 @@ @article{kingma2022VAE
url = {https://arxiv.org/abs/1312.6114},
year = {2022}
}
@article{kwon2023vLLM,
archiveprefix = {arXiv},
author = {Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
eprint = {2309.06180},
primaryclass = {cs.LG},
title = {Efficient Memory Management for Large Language Model Serving with PagedAttention},
url = {https://arxiv.org/abs/2309.06180},
year = {2023}
}
@article{li2022BLIP,
archiveprefix = {arXiv},
author = {Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},
Expand Down
4 changes: 2 additions & 2 deletions zh/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,6 @@
- 12.10 大模型 Checkpoint:模型、优化器与分布式状态如何恢复
- 12.11 Chapter 12 练习题:LLM 训练工程

## Chapter 19: SGLang:从请求调度到 Cache-Aware Serving
## Chapter 18: vLLM:从 LLM 推理到高性能推理框架

- 19.1 From vLLM to SGLang:相同的 Serving 问题,不同的设计
- 18.1 从 generate() 到 vLLM:为什么推理需要一个 Engine
339 changes: 339 additions & 0 deletions zh/ch18-vllm-introduction/ch18.1-vllm-overview.qmd

Large diffs are not rendered by default.

46 changes: 46 additions & 0 deletions zh/ch18-vllm-introduction/figures/ch18.1-req-lifecycle.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
config:
startOnLoad: false
theme: base
htmlLabels: false
fontFamily: Arial, Microsoft YaHei, Noto Serif CJK SC, sans-serif
themeVariables:
fontSize: 16px
primaryColor: "#e9f2fc"
primaryBorderColor: "#296bb7"
primaryTextColor: "#333333"
lineColor: "#0b0b0b"

themeCSS: |
.node text,
.nodeLabel,
.label text {
font-weight: 400;
font-style: normal;
}

.node rect,
.flowchart-link {
stroke-width: 1px;
}
---

sequenceDiagram
participant A as API Server
participant E as EngineCore
participant S as Scheduler
participant K as KVCacheManager
participant G as GPU Worker

A->>E: 发送 Token IDs (IPC)
E->>S: 加入请求
E->>S: schedule()
S->>K: 分配/检查 KV Cache
K-->>S: 缓存分配结果
S-->>E: SchedulerOutput
E->>G: 执行模型 (via Executor)
G-->>E: ModelRunnerOutput
E->>S: update_from_output()
E-->>A: 输出 token (IPC)
Note over E,G: 持续调度和执行直到请求完成
A-->>A: Detokenization
247 changes: 247 additions & 0 deletions zh/ch18-vllm-introduction/figures/ch18.1-req-lifecycle.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
299 changes: 299 additions & 0 deletions zh/ch18-vllm-introduction/figures/ch18.1-vllm-architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
111 changes: 111 additions & 0 deletions zh/ch18-vllm-introduction/figures/ch18.1-vllm-architecture.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
<mxfile host="Electron">
<diagram name="vLLM V1 Architecture" id="U81PoWHJHGe-g25c4hQW">
<mxGraphModel dx="1563" dy="1128" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
<mxCell id="hyPA-xEpt3381CLIv8I8-1" parent="1" style="rounded=1;whiteSpace=wrap;html=1;container=0;fillColor=#F0F6FF;strokeColor=#76A7EC;strokeWidth=2;arcSize=7;align=center;verticalAlign=top;spacingTop=14;fontSize=26;fontStyle=1;fontColor=#22375A;shadow=1;convertToSvg=1;svgWhiteSpace=wrap;" value="Offline" vertex="1">
<mxGeometry height="240" width="390" x="350" y="20" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-2" parent="1" style="rounded=1;whiteSpace=wrap;html=1;container=0;fillColor=#F0FAF4;strokeColor=#76B995;strokeWidth=2;arcSize=7;align=center;verticalAlign=top;spacingTop=14;fontSize=26;fontStyle=1;fontColor=#24533B;shadow=1;convertToSvg=1;svgWhiteSpace=wrap;" value="Online · API Server Process" vertex="1">
<mxGeometry height="240" width="390" x="770" y="20" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-3" parent="1" style="rounded=1;whiteSpace=wrap;html=1;container=0;fillColor=#FFF9EF;strokeColor=#E8B85B;strokeWidth=2;arcSize=6;align=center;verticalAlign=top;spacingTop=14;fontSize=18;fontStyle=1;fontColor=#694A1B;shadow=1;convertToSvg=1;svgWhiteSpace=wrap;" value="&lt;font style=&quot;font-size: 26px;&quot;&gt;EngineCore Process&lt;/font&gt;" vertex="1">
<mxGeometry height="380" width="635" x="442.5" y="410" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-4" parent="1" style="rounded=1;whiteSpace=wrap;html=1;container=0;fillColor=#FFF2F4;strokeColor=#E590A0;strokeWidth=2;arcSize=7;align=center;verticalAlign=top;spacingTop=14;fontSize=18;fontStyle=1;fontColor=#80394B;shadow=1;convertToSvg=1;svgWhiteSpace=wrap;" value="&lt;font style=&quot;font-size: 26px;&quot;&gt;GPU Worker Process&lt;/font&gt;" vertex="1">
<mxGeometry height="256" width="315" x="40" y="534" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-5" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E3EEFF;strokeColor=#6498DF;strokeWidth=1.8;fontColor=#1B2E4B;fontSize=22;shadow=1;arcSize=9;fontFamily=Helvetica;convertToSvg=1;svgWhiteSpace=wrap;" value="LLM.chat / LLM.generate" vertex="1">
<mxGeometry height="54" width="285" x="402" y="85" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-6" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E3EEFF;strokeColor=#6498DF;strokeWidth=1.8;fontColor=#1B2E4B;fontSize=22;shadow=1;arcSize=9;fontFamily=Helvetica;convertToSvg=1;svgWhiteSpace=wrap;" value="LLMEngine" vertex="1">
<mxGeometry height="52" width="135" x="477" y="175" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-7" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E2F6E9;strokeColor=#6AAF85;strokeWidth=1.8;fontColor=#1B4931;fontSize=22;shadow=1;arcSize=9;fontFamily=Helvetica;convertToSvg=1;svgWhiteSpace=wrap;" value="HTTP Request" vertex="1">
<mxGeometry height="54" width="195" x="867" y="85" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-8" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E2F6E9;strokeColor=#6AAF85;strokeWidth=1.8;fontColor=#1B4931;fontSize=22;shadow=1;arcSize=9;fontFamily=Helvetica;convertToSvg=1;svgWhiteSpace=wrap;" value="API Server / AsyncLLM" vertex="1">
<mxGeometry height="52" width="245" x="842" y="175" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-9" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#EFE8FF;strokeColor=#9280D5;strokeWidth=1.8;fontColor=#392D65;fontSize=24;fontStyle=1;shadow=1;arcSize=9;convertToSvg=1;svgWhiteSpace=wrap;" value="EngineCoreClient" vertex="1">
<mxGeometry height="58" width="240" x="640" y="310" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-10" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#FFEAC8;strokeColor=#E3AB47;strokeWidth=1.8;fontColor=#503913;fontSize=22;fontStyle=1;shadow=1;arcSize=9;convertToSvg=1;svgWhiteSpace=wrap;" value="EngineCore" vertex="1">
<mxGeometry height="54" width="201" x="659" y="480" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-11" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#FFEAC8;strokeColor=#E3AB47;strokeWidth=1.8;fontColor=#503913;fontSize=22;shadow=1;arcSize=9;convertToSvg=1;svgWhiteSpace=wrap;" value="Model Executor" vertex="1">
<mxGeometry height="56" width="190" x="493" y="602" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-12" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#FFEAC8;strokeColor=#E3AB47;strokeWidth=1.8;fontColor=#503913;fontSize=22;shadow=1;arcSize=9;convertToSvg=1;svgWhiteSpace=wrap;" value="Scheduler" vertex="1">
<mxGeometry height="56" width="165" x="844" y="602" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-13" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#FFEAC8;strokeColor=#E3AB47;strokeWidth=1.8;fontColor=#503913;fontSize=22;shadow=1;arcSize=9;convertToSvg=1;svgWhiteSpace=wrap;" value="KVCacheManager" vertex="1">
<mxGeometry height="56" width="201" x="826" y="706" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-14" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#FFE1E8;strokeColor=#DF8396;strokeWidth=1.8;fontColor=#713243;fontSize=22;shadow=1;arcSize=9;convertToSvg=1;svgWhiteSpace=wrap;" value="GPU Worker" vertex="1">
<mxGeometry height="55" width="165" x="115" y="612" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-15" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#FFE1E8;strokeColor=#DF8396;strokeWidth=1.8;fontColor=#713243;fontSize=22;shadow=1;arcSize=9;convertToSvg=1;svgWhiteSpace=wrap;" value="Model Runner" vertex="1">
<mxGeometry height="55" width="165" x="115" y="706.5" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-16" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-5" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;exitX=0.5;exitY=1;entryX=0.5;entryY=0;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-6">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-17" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-7" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;exitX=0.5;exitY=1;entryX=0.5;entryY=0;fontSize=12;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-8">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-18" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-6" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;exitX=0.5;exitY=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fontSize=12;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-9">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="544.4300000000001" y="339" />
</Array>
<mxPoint x="620" y="330" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-19" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-8" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;exitX=0.5;exitY=1;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fontSize=12;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-9">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="964.43" y="339" />
</Array>
<mxPoint x="1090" y="320" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-20" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-10" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;entryX=0.5;entryY=0;exitX=0.5;exitY=1;exitDx=0;exitDy=0;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-11">
<mxGeometry relative="1" as="geometry">
<Array as="points" />
<mxPoint x="730" y="534" as="sourcePoint" />
</mxGeometry>
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-21" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-10" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;entryX=0.5;entryY=0;exitX=0.5;exitY=1;exitDx=0;exitDy=0;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-12">
<mxGeometry relative="1" as="geometry">
<Array as="points" />
</mxGeometry>
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-22" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-12" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;exitX=0.5;exitY=1;entryX=0.5;entryY=0;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-13">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-23" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-11" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-4">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="588" y="710" />
<mxPoint x="388" y="710" />
<mxPoint x="388" y="470" />
<mxPoint x="197.53" y="470" />
</Array>
<mxPoint x="190" y="490" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-24" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-14" style="edgeStyle=orthogonalEdgeStyle;rounded=1;html=1;endArrow=block;endFill=1;strokeColor=#334155;strokeWidth=2;exitX=0.5;exitY=1;entryX=0.5;entryY=0;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-15">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="hyPA-xEpt3381CLIv8I8-25" edge="1" parent="1" source="hyPA-xEpt3381CLIv8I8-9" style="endArrow=block;html=1;rounded=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;strokeWidth=2;strokeColor=#334155;align=center;verticalAlign=middle;fontFamily=Helvetica;fontSize=12;fontColor=default;labelBackgroundColor=default;endFill=1;convertToSvg=1;" target="hyPA-xEpt3381CLIv8I8-3" value="">
<mxGeometry height="50" relative="1" width="50" as="geometry">
<mxPoint x="840" y="500" as="sourcePoint" />
<mxPoint x="890" y="450" as="targetPoint" />
</mxGeometry>
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
2 changes: 1 addition & 1 deletion zh/index-parts/part07.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ comments: false

listing:
contents:
- "../ch19-sglang-serving/*.qmd"
- "../ch18-vllm-introduction/*.qmd"
sort: title asc
type: table
fields: [title, author, date]
Expand Down
Loading