3Tokenization

A language model never sees text. It sees a sequence of integers, produced by a tokenizer that chops text into pieces from a fixed vocabulary and numbers them. The tokenizer is a contract signed before training begins: the model's entire view of language — what it can compare, count, and complete — is mediated by these pieces. That makes tokenization the source of some of the strangest LLM behavior, and of very real serving costs, so it deserves a chapter before we build the transformer that consumes the tokens.

3.1Why not characters or words

The two obvious units both fail.

Words fail on the open-vocabulary problem. Any fixed word list will be missing tomorrow's product names, someone's typo, rare morphology ("untranslatable"), and most of every other language. Everything missing collapses into a single unknown-word token, and the model is blind to it.

Characters never miss, but they are expensive. A character carries little meaning on its own, so the model burns layers reassembling words before it can think about them — and the sequences are long. Attention cost grows with the square of sequence length (Chapter 4), so 5× more positions is far more than 5× more compute. The context window, a scarce resource, fills with spelling.

Subword tokenization is the negotiated middle: a vocabulary of tens of thousands of pieces, where common words are a single token and rare words are built from a few fragments. The knob is vocabulary size, and it is a genuine tradeoff: a bigger vocabulary shortens sequences but grows the embedding matrix, and its rarest tokens are each seen less often during training, so their representations are worse.

3.2Byte-pair encoding

The dominant algorithm, byte-pair encoding (BPE), was born as a compression trick (Gage, 1994) and adapted for translation models (Sennrich et al., 2016). Building the vocabulary is greedy and simple: start from single characters, count adjacent pairs across the corpus, merge the most frequent pair into a new token, and repeat until you reach the target vocabulary size.

A toy corpus makes it concrete. Take low low low lower lowest and start from characters:

The merges are the tokenizer. To tokenize new text at inference time, you replay the merge list in the order it was learned; lowest becomes lowe + s + t, even though the trainer never saw a vocabulary entry for it. Frequent whole words fused early; rare words decompose into learned fragments.

3.3Byte-level BPE and friends

Production tokenizers are variations on this theme:

• Byte-level BPE runs BPE over the 256 possible bytes rather than characters, so every string — any language, any emoji, any binary garbage — is representable and there is no unknown token, ever. GPT-2 introduced this (Radford et al., 2019) and the GPT lineage kept it.
• WordPiece is BPE with a different merge criterion (Schuster & Nakajima, 2012): instead of the most frequent pair, it merges the pair that most raises the corpus likelihood under a unigram language model, which scores a pair roughly by its frequency divided by the product of its parts' frequencies. That normalization favors pairs that co-occur more than chance rather than pairs that are merely common because their parts are. It is the tokenizer of BERT (Devlin et al., 2019).
• Unigram flips the direction: start from a large candidate vocabulary and prune it down, keeping the pieces that best explain the corpus under a probabilistic model (Kudo, 2018).
• SentencePiece is the widely used library that packages these algorithms, treating text as a raw stream so the pipeline is language-agnostic and exactly reversible (Kudo & Richardson, 2018). The trick is to escape each space as a visible meta-symbol (▁) that lives in the vocabulary, so detokenizing is just concatenation with ▁ mapped back to a space — no language-specific rules, and the reason a leading space usually binds to the following word as one token.

The differences matter less than the shared shape: a current open model ships a byte-level BPE or SentencePiece vocabulary of roughly 32k to 256k tokens, learned once from a corpus resembling its pretraining data, then frozen forever.

3.4Consequences of tokenization

A frozen, frequency-based view of text explains a family of famous quirks:

• Character blindness. Ask a model how many r's are in "strawberry" and it must answer from a token like straw+berry — it does not see letters. Spelling, rhyming, and counting tasks fail not because the model is stupid but because the evidence was destroyed before it arrived.
• Numbers. If 1234 happens to be one token and 1235 is two, nearby numbers have wildly different shapes. Modern tokenizers force digits into fixed-size groups — some splitting single digits, some grouping from the right so that place value stays aligned across different numbers — which measurably helps arithmetic. It is a tokenizer design choice visibly changing a "reasoning" ability.
• A non-English tax. Vocabularies learned from English-heavy corpora spend their short codes on English. The same sentence can cost several times more tokens in Thai or Telugu than in English (Petrov et al., 2023) — which means higher API cost, a smaller effective context window, and often worse quality, all before the model has done anything.
• Glitch tokens. A token that barely appeared in training data has an essentially untrained embedding, and feeding it to the model produces erratic behavior. Unusual byte sequences are also a place where filters and models can disagree about what text "says" — part of the injection surface Chapter 18 returns to.

One economic point to carry forward: the token is the billing unit. API prices, context limits, and the serving throughput of Part IV are all denominated in tokens, so a tokenizer that spends 30% more tokens on your traffic is a 30% cost increase with no quality upside. When Chapter 15 measures tokens per second, remember that how much text a token buys was decided here.

With text turned into integers, we can now build the machine that reads them.