5How Modern Architectures Differ

Chapter 4 built the transformer as published in 2017 (Vaswani et al., 2017). No frontier model ships that exact recipe anymore — and yet the changes are few, and every one of them fixes a specific pain: training instability, KV-cache memory, or compute per token. This chapter walks the modern substitutions one component at a time. By the end you can read a real model's config file line by line, which is exactly how the chapter closes. The recipe below (rotary positions, pre-norm RMSNorm, SwiGLU, grouped-query attention) crystallized in the Llama family (Touvron et al., 2023) and is now close to universal for open models.

5.1Positional information: RoPE

Attention is order-blind: shuffle the tokens and the \(QK^\top\) scores shuffle with them. The original transformer injected order by adding a position vector to each token embedding at the bottom of the stack. That works, but position becomes just another ingredient blended into the residual stream, and attention only sees it indirectly.

Rotary position embeddings (RoPE) (Su et al., 2024) move position into the attention scores themselves. Split each query and key vector into two-dimensional pairs, and rotate each pair by an angle proportional to the token's position, with each pair rotating at its own frequency. The payoff is in the dot product: the score between a query at position \(m\) and a key at position \(n\) depends only on the offset \(m-n\), not on the absolute positions. Relative position falls out of the geometry.

RoPE also concentrates the levers for long context: because position lives in rotation frequencies, stretching the frequency schedule is how a model trained at one context length is extended to a much longer one. The named methods — linear position interpolation, NTK-aware scaling, and YaRN (Peng et al., 2023) — differ in how carefully they remap those frequencies so that short-range resolution survives; the serving costs return in Chapter 15.

5.2Normalization: RMSNorm and pre-norm

Two separate upgrades hide under "normalization."

Where it sits: the 2017 transformer normalized after each sublayer (post-norm), which at large depth needs a delicate warmup to avoid divergence. Placing the normalization before each sublayer (pre-norm) keeps the residual stream itself untouched — an identity path from embedding to output — and gradients flow through deep stacks without drama (Xiong et al., 2020). Every modern LLM is pre-norm; the block in Chapter 4 already showed it.

What it computes: LayerNorm (Ba et al., 2016) centers activations (subtract the mean) and scales them (divide by the standard deviation). RMSNorm (Zhang & Sennrich, 2019) drops the centering and simply divides by the root-mean-square, with a learned gain. It is cheaper — one fewer reduction over the hidden dimension, and no bias parameters — and models train just as well. When an ablation says the cheap half of an operation was the only half that mattered, the expensive half disappears from the recipe.

A related trick you will see in recent models is QK-norm: normalizing the queries and keys before their dot product to keep attention logits from blowing up at large scale (Dehghani et al., 2023).

5.3Activations: SwiGLU

The original FFN is two matrices with a ReLU between them. Modern models use a gated unit instead: project the input up twice, let one projection gate the other elementwise, then project back down,

The gate lets the layer modulate how much of each feature passes, not just whether it is positive — multiplicative control instead of a hard one-sided switch. Across GLU variants, SwiGLU reliably wins by a small margin (Shazeer, 2020).

There is one bookkeeping consequence you will see in every config: SwiGLU has three matrices where the classic FFN has two, so to hold the parameter count fixed, the hidden width shrinks from \(4d\) to roughly \(\tfrac{2}{3} \cdot 4d\). That is why real models have oddly specific FFN sizes instead of a clean \(4d\).

5.4Attention variants: MQA and GQA

At inference time, every past token's keys and values are cached so they are not recomputed per step — the KV cache, whose memory math Chapter 15 and Appendix A work out. In full multi-head attention, every head stores its own keys and values, so the cache scales with the head count, and at long context it competes with the weights for memory and bandwidth.

The fix is to share. Multi-query attention (MQA) keeps all the query heads but gives them one shared key/value head (Shazeer, 2019) — a large cache reduction with a measurable quality cost. Grouped-query attention (GQA) interpolates: query heads are divided into groups, each group sharing one KV head (Ainslie et al., 2023). With, say, 32 query heads and 8 KV heads, the cache shrinks 4× while quality stays near full multi-head attention. GQA is the current default.

Sharing heads is not the only lever on the cache. Sliding-window (local) attention bounds it a different way: each token attends only to the last \(w\) tokens, so cost and cache stop growing once the window is full, often with a few global layers interleaved to preserve long-range reach (Jiang et al., 2023). And multi-head latent attention (MLA), the successor probe past GQA, compresses the keys and values into a small shared latent, shrinking the cache below what GQA achieves (DeepSeek-AI, 2024).

5.5Mixture of experts

Everything so far tweaks the dense recipe. Mixture of experts (MoE) changes the deal: replace each FFN with many parallel FFNs (experts) plus a small router that sends each token to its top one or two experts (Shazeer et al., 2017; Fedus et al., 2022). The model's total parameters multiply, but the compute per token only touches the chosen experts.

That splits a number you have been treating as one. A "Mixtral 8×7B" has ~47B total parameters (what you must hold in memory) but only ~13B active parameters per token (what you pay compute and, per Appendix A, bandwidth for). Scaling laws (Chapter 9) say capacity buys quality; MoE buys capacity without buying per-token compute.

The costs are real, and the routing details are where the interviews live. Each expert has a capacity; tokens routed past it are dropped, so the router must be trained to spread load evenly. Classic MoE adds an auxiliary load-balancing loss for this; because that loss tugs against quality, newer models (DeepSeek-V3) pursue auxiliary-loss-free balancing via learned routing biases instead (DeepSeek-AI, 2024). All experts must also fit in memory even though most idle per token, training is less stable than dense, and Chapter 8's parallelism grows a new dimension since experts can live on different devices.

5.6A tour of a current open model

Here is the config of a Llama-3-class 8B model (Grattafiori et al., 2024), trimmed to the load-bearing fields:

{
  "hidden_size": 4096,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "intermediate_size": 14336,
  "hidden_act": "silu",
  "rms_norm_eps": 1e-05,
  "rope_theta": 500000.0,
  "max_position_embeddings": 8192,
  "vocab_size": 128256
}

Read it with this chapter in hand. num_key_value_heads: 8 against 32 attention heads is GQA with groups of four — a 4× smaller KV cache. intermediate_size: 14336 is \(3.5d\), not \(4d\): the SwiGLU two-thirds bookkeeping, rounded to hardware-friendly multiples, with hidden_act: silu naming the Swish gate. rms_norm_eps tells you normalization is RMSNorm, and rope_theta: 500000 is the rotation base — raised well above the original \(10{,}000\) for longer context. The vocabulary of 128k tokens is Chapter 3's tradeoff, sized generously to shorten sequences. Every line is one of this chapter's sections, frozen into JSON.

The architecture is now current.

The next part of the book takes these frozen blueprints and asks how the weights inside them are actually learned.