Chapter 17

17Serving Systems in Production

Draft pending — this page is an outline of planned content. Write it by creating content/serving-systems.md.

17.1What a serving stack must do

Planned: Throughput, latency (TTFT and TPOT), and cost as competing goals.

17.2The engines

Planned: vLLM, TensorRT-LLM, SGLang, TGI; what differentiates them.

17.3Multi-request scheduling

Planned: Prefix caching, prioritization, and fairness across users.

17.4Autoscaling and the economics

Planned: GPU utilization, cold starts, and the unit economics of tokens.