Math For ML

1. The one-sentence intuition

ML runs on four pieces of math, and each is a thing you already use as a programmer: a vector is a list of numbers, a matrix is a pure function that maps one list to another, a gradient is the compiler-computed arrow telling you which way to nudge your parameters to reduce error, and a probability distribution is just normalized scores that sum to 1. You do not need to do this math by hand — autograd and BLAS do it — but you need to read it, the way you read a stack trace: fluently enough that "we take the dot product of the query and key vectors" lands as an obvious sentence, not a wall.

2. Why a software engineer needs this

Every downstream pillar quietly assumes this vocabulary:

• RAG is "embed text into vectors, then rank by dot-product similarity." If "dot product" is fuzzy, chunking and embeddings reads as hand-waving.
• Transformers are matrix multiplies plus softmax, end to end. Attention is literally softmax(QKᵀ/√d)·V — three operations from this lesson.
• Fine-tuning is gradient descent on a loss. Every word there — gradient, loss, cross-entropy, log-likelihood — is defined below.
• Evals report perplexity and log-loss; those are the probability section.

Interviewers won't ask you to prove a theorem. They'll say "why cosine similarity?" or "what does loss.backward() actually compute?" and expect a crisp, correct answer in 30 seconds. This lesson gives you exactly those answers and nothing you won't use.

3. Build it up from scratch

Three buckets: linear algebra (how data and models are represented), calculus (how models learn), probability (how we score predictions). We teach the real techniques — just the slice each one actually contributes.

3.1 Linear algebra — data and models are arrays

A vector is an ordered list of numbers, written x = [x₁, x₂, …, xₙ]. The subscript is just an index, like x[i]. Geometrically it's a point (or an arrow from the origin) in n-dimensional space. In ML, everything becomes a vector: an embedding is a vector of (say) 1024 numbers that represents the meaning of a word or document, learned so that similar meanings land at nearby points.

The dot product is the one operation to internalize. For two vectors a and b of the same length:

The Σ (capital sigma) means "sum over all i" — it's a for loop that adds up the elementwise products. Example: a = [1, 2, 3], b = [2, 0, 1] → a·b = 1·2 + 2·0 + 3·1 = 5.

Why it matters: the dot product measures alignment. It equals |a| |b| cos θ, where |a| is the length (or norm) of a — its magnitude, |a| = √(Σ aᵢ²) — and θ is the angle between the two vectors. So:

• vectors pointing the same way → large positive dot product,
• perpendicular (unrelated) → zero,
• opposite → negative.

Strip out the lengths and you get cosine similarity = a·b / (|a||b|), a pure "how aligned are these directions" score in [-1, 1]. That single formula is how vector search finds the chunk most relevant to your query: embed both, take the cosine, sort. Same operation is the heart of attention — "how much should token A attend to token B" is a dot product of their query/key vectors.

A matrix is a 2-D grid of numbers, W with entry Wᵢⱼ at row i, column j. The right mental model isn't "spreadsheet" — it's a function. A matrix is a linear transformation: feed it a vector, get back a vector, via matrix–vector multiply:

Read that as: output element i is the dot product of row i of W with the input x. A neural network layer is exactly this — y = Wx + b (the +b shifts the result, like an intercept) — so a layer is "apply this learned linear function, then a nonlinearity." The numbers in W are the learned parameters; training is the search for good ones.

Matrix–matrix multiply stacks this: (AB)ᵢⱼ = Σₖ Aᵢₖ Bₖⱼ — entry (i,j) is the dot product of row i of A with column j of B. This is the compute-bottleneck operation of all of deep learning; it's what GPUs are built to do fast and what "FLOPs" mostly counts. When someone says a model is "a stack of matmuls," they mean it almost literally.

3.2 Calculus — how a model improves

Training = "tweak the parameters to make the error smaller." Calculus tells you which way to tweak.

A derivative is a slope. For a one-input function f(x), the derivative f'(x) (also written df/dx) answers: if I nudge x up a tiny bit, how fast does f change, and in which direction? Positive slope → f rises as x rises; negative → f falls. Formally it's the limit of rise-over-run as the step shrinks to zero, but the operational meaning is just local sensitivity: Δf ≈ f'(x)·Δx.

Models have millions of inputs (parameters), not one. The gradient generalizes the derivative to many variables. For f(x₁,…,xₙ), the gradient is the vector of partial derivatives — each ∂f/∂xᵢ is "the slope in the xᵢ direction, holding the others fixed" (the curly ∂ just signals 'partial'):

∇ ("nabla") is the gradient operator. The key fact: the gradient points in the direction of steepest increase of f. So to decrease f (your error), step in the opposite direction. That's gradient descent:

η ("eta") is the learning rate — a small positive step size you choose (e.g. 0.01). The arrow ← means "update in place." Repeat until the error stops dropping. That one line is how every model in this atlas is trained.

Worked micro-example. Let f(x,y) = x² + y² (a bowl; minimum at the origin). Its gradient is ∇f = (2x, 2y). At the point (3, 4), ∇f = (6, 8). Step downhill with η = 0.1: new point = (3, 4) − 0.1·(6, 8) = (2.4, 3.2). Check: f went from 9+16 = 25 down to 5.76+10.24 = 16. One step, error fell. Iterate and you slide to the bottom.

The chain rule is the engine that makes this work for deep networks. A network is functions nested inside functions: loss(layer3(layer2(layer1(x)))). The chain rule says the derivative of a composition is the product of the local derivatives:

Concretely: z = (3x+1)². Let u = 3x+1, so z = u². Then dz/dx = (dz/du)(du/dx) = (2u)(3) = 6(3x+1). At x=1: 6·4 = 24.

Backpropagation ("backprop") is just the chain rule applied right-to-left across the whole network: do a forward pass to compute the loss, then walk backward multiplying local derivatives to get ∂loss/∂w for every parameter w at once. It's reverse-mode automatic differentiation. You never write it by hand — loss.backward() does it — but now you know exactly what that call computes: the gradient, via the chain rule.

3.3 Probability — scoring predictions

Models output scores; probability turns scores into calibrated predictions and gives us something principled to minimize.

A distribution assigns a probability to each possible outcome, with the rule that the probabilities are non-negative and sum to 1. "70% cat, 20% dog, 10% bird" is a distribution over 3 classes.

Softmax is how a model produces one. The network's raw output scores are called logits — unbounded real numbers, e.g. [1.0, 3.0, 1.0]. Softmax exponentiates and normalizes them into a distribution:

e^{z} is the exponential (always positive, so no negative "probabilities"); the denominator forces the outputs to sum to 1. For logits [1, 3, 1] you get [0.107, 0.787, 0.107]. Bigger logit → bigger share, and the gap is amplified.

Expectation is a probability-weighted average: E[X] = Σ x·p(x). It's the value you'd get on average over many draws; it shows up whenever we average a loss over data or over a model's outputs.

Conditional probability is P(A|B) — "probability of A given B is known" — defined as P(A,B)/P(B). A language model is one giant conditional: P(next token | all previous tokens). Bayes' rule flips a conditional around:

Read H as a hypothesis (e.g. "this email is spam") and D as observed data (the words). It says: posterior belief = (how well the hypothesis explains the data) × (prior belief), normalized. It's the backbone of probabilistic reasoning and the namesake of the Naive Bayes classifier.

Why log-likelihood and cross-entropy? Training a probabilistic model means maximum likelihood estimation (MLE): pick parameters that make the observed data most probable. The probability of the whole dataset (assuming independent examples) is a giant product ∏ᵢ p(xᵢ; θ). Products of thousands of small numbers underflow to zero and are painful to differentiate. So we take the log: log ∏ = Σ log, turning the product into a sum. Log is monotonic, so the maximizer is unchanged — but the math becomes a clean, stable sum of log p. We then flip the sign (optimizers minimize), giving negative log-likelihood, the thing we descend on.

For classification this is exactly cross-entropy loss. With a true label that's "100% the correct class" (a one-hot target) and model probabilities q, cross-entropy collapses to:

Worked example. True class = "cat," model says cat 0.7 → loss = −log(0.7) ≈ 0.357. A different run says cat 0.1 → loss = −log(0.1) ≈ 2.303. Confident and right → tiny loss; confident and wrong → big loss. That's why we minimize cross-entropy instead of accuracy: it's smooth and differentiable (so gradient descent has a slope to follow) and it rewards calibrated confidence, whereas accuracy is a flat step function with no gradient to learn from.

4. See it in code

A full classifier forward pass — matmul → softmax → cross-entropy — is just the math above:

import numpy as np
 
x = np.array([2.0, 1.0, 1.0])                 # input feature vector (length 3)
W = np.array([[ 1.0, -1.0,  0.0],             # learned weights: 3 classes (rows)
              [ 0.0,  2.0,  1.0],             #   x 3 features (cols)
              [ 1.0,  0.0, -1.0]])
 
logits = W @ x                                # matrix-vector multiply -> [1., 3., 1.]
                                              # each entry = dot(row_of_W, x)
 
probs = np.exp(logits) / np.exp(logits).sum() # softmax -> [0.107, 0.787, 0.107]
 
true_class = 1                                # the correct label is class #1
loss = -np.log(probs[true_class])             # cross-entropy = -log(0.787) ≈ 0.24
print(logits, probs, loss)

Line by line: W @ x (the @ is matmul) produces the logits, one dot product per class. np.exp(...)/...sum() is softmax, normalizing logits into a probability distribution. -np.log(probs[true_class]) is cross-entropy: because the model put 0.787 on the right class, the loss is low (≈0.24). Drop that to 0.1 and the loss jumps to ≈2.3 — the penalty for confident wrongness.

And the calculus half — the gradient via the chain rule — is one line with autograd:

import torch
 
x = torch.tensor(1.0, requires_grad=True)  # track gradients for x
z = (3 * x + 1) ** 2                        # forward pass builds the graph
z.backward()                                # backprop: chain rule, fills x.grad
print(x.grad)                               # tensor(24.) == 6*(3*1+1), as we derived

z.backward() walks the computation graph backward, multiplying local derivatives (d z/d u = 2u, d u/d x = 3) to land on dz/dx = 24 — exactly our hand-derived chain-rule answer. Scale this from one variable to a billion and you have how every model trains.

5. Mental models & SWE analogies

6. Common confusions

• "A matrix is a table of data." No — treat it as a function. The data flowing through is the vector; the matrix is the operation applied to it.
• "Dot product = element-wise multiply." Element-wise multiply keeps a vector; the dot product sums those products down to a single number (the similarity score).
• "The gradient points toward the minimum." It points toward the steepest increase. Gradient descent moves in the negative gradient direction.
• "Logits are probabilities." Logits are raw, unbounded scores. Only after softmax do they become probabilities in [0,1] that sum to 1.
• "Backprop is a separate algorithm from the chain rule." Backprop is the chain rule, applied systematically from the loss backward through the network.
• "We take logs to make numbers smaller." We take logs to turn an underflow-prone product into a stable sum — and because log is monotonic, the optimum is unchanged.
• "Bigger learning rate = faster training, always." Too large η overshoots and diverges; too small crawls. It's a tuned knob, not "more is better."

7. Check yourself

8. Go deeper

• Stanford CS229 — Linear Algebra review and Probability review: the exact two handouts ML courses assume; skimmable in an evening.
• 3Blue1Brown — Essence of Linear Algebra: the canonical visual intuition for "matrix = transformation." Pair with his Essence of Calculus for derivatives.
• CS231n — Backprop notes: the clearest from-scratch derivation of the chain rule as backprop, with circuit diagrams.
• Dive into Deep Learning — Mathematics appendix: runnable, ML-first treatment of every concept here.

Next: How models learn puts this gradient into the full training loop — loss surfaces, learning rates, SGD, and overfitting — then neural networks stacks the matmuls into a real model.