Chapter 15

15Making Inference Fast

Draft pending — this page is an outline of planned content. Write it by creating content/inference-optimization.md.

15.1Prefill vs decode

Planned: The two phases; compute-bound vs memory-bandwidth-bound; why tokens/sec is dominated by decode.

15.2The KV cache

Planned: What it stores and why it grows; the memory math per token.

15.3Batching and PagedAttention

Planned: Continuous batching and paged KV memory; the vLLM idea in one picture.

15.4Speculative decoding

Planned: A small draft model proposing tokens a big model verifies; the free-lunch intuition.

15.5FlashAttention

Planned: IO-aware attention; recomputation to stay in fast memory.