2Deep Learning, Just Enough

Everything in this book runs on one engine: a differentiable function, a loss that scores its output, and gradient descent turning the loss into small parameter updates. You have seen all of this before. The job of this chapter is to compress it into the specific facts the later chapters lean on — how representations arise, what the training loop actually does, which optimizer LLMs use and why, why giant models generalize at all, and why the field settled on bf16. For the full course, the standard reference is still the Goodfellow, Bengio, and Courville textbook (Goodfellow et al., 2016); this is the refresher.

2.1From features to representations

Classical machine learning splits the work in two: a human designs features, and a model learns weights over them. Deep learning removes the split. You hand the network raw input — pixels, bytes, token ids — and each layer re-describes the output of the previous one, so the features themselves are learned. The early layers of a language model learn things like "these characters form a word"; later layers learn things like "this clause is the object of that verb." Nobody designed those features, and nobody could have designed the millions of stranger ones between them.

A useful geometric picture is the manifold view. Raw inputs are points tangled together in a huge space: sentences with opposite meanings can differ by one token. Each layer applies a learned transformation that untangles the manifold a little, so that by the final layer, things that should be treated the same sit near each other. Prediction becomes easy geometry at the top because the layers below did the unfolding.

2.2The training loop

Every model in this book — from a toy MLP to a frontier LLM — is trained by the same five lines:

for batch in data:
    logits = model(batch.inputs)  # Forward pass.
    loss = cross_entropy(logits, batch.targets)  # How wrong were we?
    loss.backward()  # Backprop: compute every parameter's gradient.
    optimizer.step()  # Nudge every parameter downhill.
    optimizer.zero_grad()

The only non-obvious part is loss.backward(). Backpropagation (Rumelhart et al., 1986) is the chain rule from calculus, applied systematically from the loss backwards through every operation, reusing intermediate results so the whole thing costs about as much as the forward pass. Autograd frameworks are bookkeeping for that chain rule — they record what the forward pass did, then replay it in reverse. There is nothing else in the box: when Chapter 7 talks about a trillion-token pretraining run, it is this loop, run a few million times on a few thousand GPUs.

2.3Optimizers and learning rates

Plain stochastic gradient descent takes the gradient of the current batch and steps against it. Two refinements matter, and they are the two knobs worth understanding:

• Momentum keeps a running average of recent gradients and steps along that instead. Individual batches are noisy; the average points where the loss is consistently going down.
• Adaptive scaling gives each parameter its own effective step size, normalized by the recent magnitude of its gradients. A parameter whose gradients are habitually huge takes proportionally smaller steps, and vice versa.

Adam (Kingma & Ba, 2015) combines both, and AdamW (Loshchilov & Hutter, 2019) fixes a subtle interaction with weight decay. The subtlety: if you add decay as an L2 penalty in the loss, it enters the gradient and is then divided by Adam's per-parameter denominator, so parameters with large gradients get less effective decay — the opposite of uniform. AdamW decouples the two, subtracting a fixed fraction of each weight directly, independent of gradient scale. AdamW is the default optimizer for essentially every LLM you will meet in this book, and Chapter 7 covers the learning-rate schedule (warmup, then decay) that accompanies it.

The learning rate itself remains the single most important hyperparameter in deep learning. Too high and the loss diverges; too low and you waste compute crawling. Everything else in an optimizer exists to make one global learning rate workable across millions of very different parameters.

2.4Regularization and generalization

The classical story says a model with too many parameters will memorize its training set and fail on new data, so you constrain it: weight decay shrinks weights toward zero, and dropout (Srivastava et al., 2014) randomly silences activations during training so no unit can depend too much on another.

LLMs complicate the classical story twice. First, the puzzle: modern networks are so overparameterized that they can memorize completely random labels — yet the same architectures, trained on real data, generalize well (Zhang et al., 2017). Capacity alone does not decide generalization; what the optimizer finds first, and the structure in the data, matter more than the parameter count. Second, the practice: a pretraining run makes roughly one pass over its enormous corpus, so the model rarely sees any example twice — classic overfitting has little room to happen. Weight decay survives in LLM training more as a stabilizer than as an anti-memorization device, and dropout has largely disappeared from pretraining.

2.5Numerical precision

A number format spends its bits on two things: dynamic range (how large and small the exponent lets values get) and precision (how many mantissa bits distinguish nearby values). float32 has plenty of both but costs 4 bytes per number — memory and bandwidth you will learn to resent in Parts II and IV.

The two half-size formats split the budget differently:

fp16's narrow range is the problem: gradients in a large model span many orders of magnitude, and values silently underflow to zero or overflow to infinity. Training in fp16 requires loss scaling — multiplying the loss so gradients sit inside the representable window — and it is fragile machinery (Micikevicius et al., 2018). bf16 keeps float32's full exponent, sacrificing precision instead (Kalamkar et al., 2019). That trade wins because training is noisy anyway: stochastic gradients already jitter far more than the mantissa error, but a single overflow can kill a run.

Precision returns in Chapter 16 from the other side: at inference time, weights are static and the noise argument changes, which is why quantization can push below 8 bits per weight when training cannot.

That is the equipment. Next, the specific way LLMs turn text into the integers this machinery consumes.