NLP and the Road to LLMs

1. The one-sentence intuition

A large language model (LLM) is autocomplete trained on a huge slice of the internet. Under the hood it does exactly one thing: read a sequence of tokens (numbered chunks of text) and predict the next one — then append it and repeat. Translation, code, chat, "reasoning" are all that single trick, scaled up. The SWE mental model: it's your IDE's autocomplete, except the "index" isn't a symbol table — it's a few hundred billion floating-point numbers tuned by gradient descent so that the most probable next token is usually a good one.

2. Why a software engineer needs this

This lesson is the bridge from "neural nets" to everything in the Agents and RAG pillars. Four concrete payoffs:

• Tokens are the unit you pay for and reason about. Pricing, latency, and the context window (how much text fits) are all measured in tokens — not characters or words. "Why did my 50-page PDF blow the budget?" is a token question.
• Embeddings are the entire foundation of RAG. Vector search, chunking, rerankers — all of it rests on "meaning as geometry," which starts here.
• Next-token prediction explains the model's personality. Hallucination, confident wrongness, and why a knob called temperature exists all fall out of "it's sampling from a probability distribution," not "it's looking up facts."
• Pretraining vs. post-training tells you what fine-tuning actually changes. When someone says "we did SFT then RLHF," you'll know which behaviors that touches.

Interviews silently assume you can answer: what is a token, what is an embedding, what did attention buy us over RNNs, and what do softmax/temperature do? Miss these and you sound like you've only used the API, never understood it.

3. Build it up from scratch

3.1 Step one: text → numbers (tokenization)

Neural nets eat vectors of numbers, not strings. So the first job is to chop text into pieces and map each piece to an integer. A token is one such piece; the fixed list of all possible pieces is the vocabulary; a token's index in that list is its token id. Tokenization is the lookup "hello" → 15339.

Why not just split on words? The vocabulary would be effectively infinite (every typo, name, snake_case identifier, and emoji is a new word), and any word you didn't see in training becomes an unknown blank — useless for code and proper nouns.

Why not characters? The vocabulary is tiny and there are no unknowns, but sequences get ~5× longer (every letter is a step), and a lone t carries almost no meaning, so the model wastes capacity relearning that t-h-e spells "the."

The winning compromise is subword tokenization, usually via Byte-Pair Encoding (BPE). Idea: start from individual bytes/characters, then repeatedly find the most frequent adjacent pair and merge it into a new token. Common chunks ("the", "ing", "tion") become single tokens; rare words gracefully fall back to smaller pieces.

Worked micro-example. Train on the words low, lower, newest, widest. Start as characters:

l o w   ·   l o w e r   ·   n e w e s t   ·   w i d e s t

The pair e s appears in both newest and widest (most frequent), so merge it → es. Now es t is the most frequent pair → merge → est. After a few merges, est is a single reusable token, so a brand-new word like slowest tokenizes as slow + est instead of seven characters — no "unknown" needed.

Practical facts worth carrying:

• Real vocabularies are ~30k–130k tokens (GPT-2 used 50,257). Modern LLMs use byte-level BPE, so any string — Unicode, source code, gibberish — is representable; nothing is ever truly out-of-vocabulary.
• A rule of thumb for English: 1 token ≈ 4 characters ≈ ¾ of a word. "tokenization" might split as token + ization; "Anthropic" as Anthrop + ic. Whitespace usually attaches to the following token.
• Tokenization is mechanical and fixed before training — no learning yet. Learning starts once these ids become vectors.

3.2 Meaning as geometry (word embeddings)

A token id like 15339 is just a label; 15340 isn't "one more" than it. We want each token to become a vector where distance and direction encode meaning. That vector is an embedding: a list of, say, 768 learned numbers per token. The model keeps an embedding matrix of shape [vocab_size, d] and "looks up" each token's row — a learnable lookup table.

The classic result (word2vec, 2013; GloVe from Stanford, 2014) is that if you train these vectors so words appearing in similar contexts get similar vectors, geometry starts to mirror semantics. Similarity is measured by cosine similarity — the angle between two vectors:

Here a · b is the dot product (multiply componentwise, sum), and ‖a‖ is the vector's length. The result runs from −1 (opposite) to 1 (identical direction); "cat" and "dog" land close, "cat" and "thermodynamics" far apart. Even directions carry meaning, giving the famous analogy:

The "royalty" and "gender" concepts became roughly consistent directions in the space. This is exactly the trick RAG reuses: embed a query and your documents, then retrieve by cosine similarity.

The fatal limitation: word2vec/GloVe give one vector per word, forever. "bank" gets a single embedding that must average river bank and savings bank. There's no way to bend the vector based on the surrounding sentence. Real language is contextual — and fixing that is the rest of this story.

3.3 Reading in order (RNNs) and their bottleneck

To use context you must process a sequence. The first deep approach was the Recurrent Neural Network (RNN): walk left to right, maintaining a hidden state h (a vector summarizing everything seen so far). At each token: h_new = f(h_old, embedding(token)). The LSTM (1997) is a fancier RNN with gates that decide what to keep or forget, which helped it remember longer.

RNNs read context, but they have three painful bottlenecks:

1. A single fixed-size hidden state must compress the entire history. Early tokens get blurred out by the time you reach token 500 — the model literally forgets the start of a long paragraph.
2. Inherently sequential. Token 500 can't be computed until tokens 1–499 are done, so you can't parallelize across the sequence — brutal on GPUs.
3. Vanishing gradients over long distances make learning relationships between far-apart words hard (covered in how models learn).

3.4 Attention: let every token look at every other token

The fix (Bahdanau 2015, then generalized) is attention: instead of cramming the past into one hidden state, let each token directly look at every other token and pull in what's relevant. For the word "it" in "the cat sat because it was tired," attention lets "it" reach back and weight "cat" heavily.

Mechanically, every token produces three vectors:

• a query q — "what am I looking for?"
• a key k — "what do I offer?"
• a value v — "what I'll actually contribute if you attend to me."

A token compares its query to every other token's key via dot product (high dot product = relevant), softmaxes those scores into weights that sum to 1, and takes the weighted average of the corresponding values. In one line:

Q, K, V are the stacked query/key/value vectors for all tokens; QKᵀ is the all-pairs relevance grid; dividing by √d (d = vector size) keeps the numbers from exploding; softmax turns each row into weights; multiplying by V is the weighted average. The key win: there's no fixed bottleneck and no sequential dependency — every pair interacts directly, and it all runs as parallel matrix multiplies. Now "bank" can finally become context-dependent: its output vector bends toward "river" or "money" based on its neighbors.

3.5 The Transformer (a teaser)

The Transformer (Vaswani et al., 2017, "Attention Is All You Need") threw out recurrence entirely and built the network from stacked blocks of attention + a small feed-forward network, with position information added back in (since attention alone is order-blind). Because it's parallel and scales beautifully on GPUs, it's the architecture behind every LLM today. The internals — multi-head attention, positional encodings, residual streams, layer norm — are the entire Transformers pillar. For now, hold this: a Transformer is a deep stack of "every token attends to every token, then thinks for a moment."

3.6 What a language model actually does

Put it together. Feed in a sequence of tokens; the Transformer outputs, for the next position, one raw score per vocabulary token. Those raw scores are logits — unnormalized "how likely is this the next token?" numbers, ranging over all reals. To turn ~100k logits into a probability distribution, apply softmax:

z_i is the logit for token i, e is exp (makes everything positive and amplifies gaps), the denominator normalizes so all p_i sum to 1, and T is temperature (ignore it / set T=1 for now). You now have P(next token | everything so far). You pick a token, append it, and run again — autoregressive generation.

The genius is the training signal. The "label" for any position is simply the token that actually came next in the real text — which is free, sitting right there in the corpus. No human annotation. This is self-supervised learning: the data labels itself. Run it over trillions of tokens and minimizing next-token cross-entropy loss forces the model to absorb grammar, facts, code, and reasoning patterns, because predicting the next token well requires understanding the previous ones.

Decoding — turning probabilities into text:

Temperature and top-p are the two knobs you'll actually turn in production. High temperature for brainstorming; low (or T=0) for extraction, classification, and anything you'll parse.

3.7 Pretraining vs. post-training, in one paragraph

What we just described — self-supervised next-token prediction on internet-scale text — is pretraining. It produces a base model: a brilliant autocomplete that will happily continue your prompt but doesn't reliably follow instructions or stay helpful/harmless. Post-training fixes that. Supervised fine-tuning (SFT) continues training on curated (instruction, ideal response) pairs so the model learns the assistant format. Then RLHF (Reinforcement Learning from Human Feedback) trains a reward model on human preference comparisons ("response A is better than B") and nudges the LLM toward higher-rewarded outputs. Same next-token machinery, just pointed at behavior instead of raw text. That's the whole job of the Fine-tuning pillar.

4. See it in code

The single most important transformation in the whole pipeline — logits → softmax → a sampled token — is a few lines of numpy. Everything before it (embeddings, attention, the Transformer stack) is "just" what produces the logits.

import numpy as np
 
# A language model's final layer emits one logit per vocabulary token:
# an unnormalized score for "is THIS the next token?". Toy 5-token vocab:
vocab  = ["the", "cat", "sat", "dog", "ran"]
logits = np.array([2.0, 1.0, 0.1, 1.5, -0.5])   # what the Transformer produced
 
def softmax(z, temperature=1.0):
    z = z / temperature          # temperature rescales the GAPS between logits
    z = z - z.max()              # subtract max: numerically safe, same result
    e = np.exp(z)                # exponentiate -> all positive, gaps amplified
    return e / e.sum()           # normalize -> a probability distribution (sums to 1)
 
probs = softmax(logits)
print(dict(zip(vocab, probs.round(3))))
# {'the': 0.453, 'cat': 0.167, 'sat': 0.068, 'dog': 0.275, 'ran': 0.037}
 
# Greedy decoding: always the argmax -> deterministic, can be repetitive.
print("greedy :", vocab[int(np.argmax(probs))])           # 'the'
 
# Sampling: draw in proportion to probability -> varied, "creative".
rng = np.random.default_rng(0)
print("sampled:", vocab[rng.choice(len(vocab), p=probs)])
 
# Temperature < 1 sharpens (more confident); > 1 flattens (more random).
print("T=0.5  :", softmax(logits, 0.5).round(3))   # mass piles onto 'the'
print("T=2.0  :", softmax(logits, 2.0).round(3))   # closer to uniform

Line by line: logits stands in for the model's output. softmax divides by temperature (so a small T exaggerates the lead of the top token, a large T evens everyone out), subtracts the max for numerical stability, exponentiates, and normalizes. argmax is greedy decoding; rng.choice(..., p=probs) is sampling — the reason two identical prompts can give different answers. Run it and watch T=0.5 concentrate probability on "the" while T=2.0 spreads it out. That's the entire intuition behind the temperature slider in any chat API.

5. Mental models & SWE analogies

• Tokenizer ≈ a lexer. It deterministically chops a string into a stream of known atoms (tokens) before the "compiler" (the model) runs. BPE just learns its token set from data instead of a hand-written grammar.
• Embedding ≈ a hash that preserves meaning. A normal hash scatters similar inputs randomly; an embedding does the opposite — similar meanings map to nearby vectors, which is exactly what makes vector search work.
• Attention ≈ a content-addressable cache / a soft JOIN. Each token issues a query and gets back a weighted blend of every other token's value, keyed by relevance — a differentiable lookup over the whole sequence instead of an exact key match.
• Logits ≈ pre-Response, softmax ≈ serialization to a probability JSON. Logits are the raw internal scores; softmax is the normalization step that makes them a well-formed distribution you can sample from.
• Pretraining ≈ a general-purpose stdlib; post-training ≈ your app code. Pretraining gives broad capability; SFT/RLHF specialize it into a helpful assistant without rewriting the foundation.

6. Common confusions

• "A token is a word." No — it's a subword. "unbelievable" can be 3 tokens; a space often rides along with the next token. Count tokens, not words, for cost and limits.
• "Embeddings are looked up from a fixed dictionary." Static word vectors (word2vec/GloVe) were. Inside an LLM, the output vector for a token is recomputed in context by attention — "bank" differs by sentence.
• "The model stores facts and retrieves them." It stores weights, not a database. It generates the statistically likely next token, which is why it can be fluently, confidently wrong (hallucinate). Grounding it is what RAG is for.
• "Higher temperature = smarter." Temperature only controls randomness, not capability. For parsing, extraction, or classification you usually want T=0.
• "Softmax outputs are calibrated confidence." They're a distribution, not a guarantee of correctness — a model can put 0.95 on a wrong token.
• "RLHF teaches the model new facts." Post-training mostly shapes behavior and format (be helpful, follow instructions); the knowledge came from pretraining.

7. Check yourself

8. Go deeper

• Stanford CS224n — NLP with Deep Learning (lectures + notes): the canonical path from word vectors → attention → Transformers. https://web.stanford.edu/class/cs224n/
• "Attention Is All You Need" (Vaswani et al., 2017) — read just the intro and Figure 1 without panic now. https://arxiv.org/abs/1706.03762
• Sennrich et al., subword units / BPE — the tokenization idea every LLM uses. https://arxiv.org/abs/1508.07909
• GloVe (Stanford NLP) — meaning-as-geometry, with downloadable vectors to play with. https://nlp.stanford.edu/projects/glove/
• Karpathy, "Intro to LLMs" — a 1-hour talk that ties tokenization, pretraining, and RLHF together. https://www.youtube.com/watch?v=zjkBMFhNj_g

Next: you now have the vocabulary — go open the Transformers pillar to see how attention is actually built, or jump to Building AI Agents to put a chat model to work. For the retrieval side of embeddings, head to RAG → chunking & embeddings.