Attention/MHA, RoPE, RMSNorm, activations, tokenization, sampling, and scaling laws — from first principles.
Every token quietly looks at every other token, decides who matters, and pulls in a weighted average — derive that from scratch, then make it fast.
Attention is a bag of vectors with no sense of order — positional encodings are the trick that smuggles 'where' back into a permutation-invariant machine, and the story of how RoPE beat everyone else.
The unglamorous plumbing — where you put the norm, whether you subtract the mean, and how you gate the MLP — is what actually lets a 100-layer transformer train without diverging.
The lossy compression layer between human text and the model — where 'strawberry' becomes three tokens the model can't spell and a glitch token can make GPT speak in tongues.
Every token a model emits is a deliberate bet placed on a reshaped probability distribution — this lesson teaches you to control that bet.
A single power law tells you how big to build, how long to train, and why every modern lab now over-trains a small model to win on inference.