HOW AN LLM THINKS

One sentence through GPT-2 small — its real numbers, precomputed. Act 01: the sentence is cut into pieces it has numbers for.

ACT 01 · TOKENS

A language model never sees letters or words. GPT-2 starts from the raw bytes and glues the most common pairs together, again and again, in the order it learned from 40 GB of web text. Common words come out whole. Rarer ones — Reykjavik — come out in pieces. Each piece has a number: one of 50,257.

FACT These splits are GPT-2's own byte-pair encoding, replayed merge by merge from its 50,000 merge rules, and checked against the reference tokenizer.
FACT A leading space is part of the token: ␣Paris and Paris are different tokens with different numbers.
THEORY Tokens explain some odd failures. GPT-2 sees ␣strawberry as one token, number 41236, and never its letters: to count the r's it would have to have learned the spelling as a fact.

01 TOKENS

sources

What is real. Every GPT-2 number on this page — token splits, embeddings, all 144 attention patterns, residual stream, MLP activations, logit lens, logits — was computed once, offline, from GPT-2 small's released weights, and is shipped as static files. The page runs no model and sends nothing anywhere. The 3D layout, colours and the flow of particles are illustration. The training table is from the papers below; estimates are marked.

Also: /ai (a network training live) · /ai-brain. Drag to turn, pinch or scroll to zoom, double-tap to reset, tap a token or a dot to pick it.

Radford et al., “Language Models are Unsupervised Multitask Learners”, OpenAI 2019 — GPT-2, WebText, the 50,257-token BPE.
GPT-2 small weights and tokenizer: OpenAI, Modified MIT licence (github.com/openai/gpt-2), via huggingface.co/openai-community/gpt2. All model numbers on this page were computed offline from them by scripts/extract-llm.py.
Sennrich, Haddow & Birch, “Neural Machine Translation of Rare Words with Subword Units”, ACL 2016 — byte-pair encoding.
Vaswani et al., “Attention Is All You Need”, NeurIPS 2017.
Elhage et al., “A Mathematical Framework for Transformer Circuits”, Anthropic 2021 — the residual stream.
Olsson et al., “In-context Learning and Induction Heads”, Anthropic 2022.
Wang et al., “Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small”, ICLR 2023.
McDougall et al., “Copy Suppression: Comprehensively Understanding an Attention Head”, 2023.
Xiao et al., “Efficient Streaming Language Models with Attention Sinks”, 2023.
Geva et al., “Transformer Feed-Forward Layers Are Key-Value Memories”, EMNLP 2021; Meng et al., “Locating and Editing Factual Associations in GPT”, NeurIPS 2022.
nostalgebraist, “interpreting GPT: the logit lens”, 2020.
Elhage et al., “Toy Models of Superposition”, Anthropic 2022.
Jain & Wallace, “Attention is not Explanation”, NAACL 2019.
Kaplan et al., “Scaling Laws for Neural Language Models”, 2020 — C ≈ 6ND.
Brown et al., “Language Models are Few-Shot Learners”, 2020 — GPT-3, Table D.1.
Hoffmann et al., “Training Compute-Optimal Large Language Models”, 2022 — Chinchilla.
Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models”, 2023.
Llama Team, Meta, “The Llama 3 Herd of Models”, 2024.
DeepSeek-AI, “DeepSeek-V3 Technical Report”, 2024.
Epoch AI, Notable AI Models database — GPT-4 compute estimate.