Chapter 24

24Evaluating Language Models

Draft pending — this page is an outline of planned content. Write it by creating content/evaluation.md.

24.1Why evaluation is the hard part

Planned: Open-ended output has no single ground truth; the metric problem.

24.2Benchmarks and their discontents

Planned: MMLU-style suites, contamination, saturation, and gaming.

24.3LLM-as-judge

Planned: Using a model to grade a model; bias, calibration, and when to trust it.

24.4Human eval and arenas

Planned: Pairwise human preference, Elo-style ranking, and their limits.

24.5Building evals for your own product

Planned: Task-specific eval sets; the flywheel of logging, labeling, and regression tests.