Scaling Laws: Kaplan, Chinchilla, and the Over-Training Era

1. Quick anchor

A scaling law is an empirical power law that predicts model loss from three numbers — parameters $N$, training tokens $D$, and the compute $C \approx 6ND$ that ties them together. The headline fact: for a fixed compute budget, there is one optimal $(N, D)$ pair, and Chinchilla showed that pair has $N$ and $D$ growing in lockstep — roughly 20 tokens per parameter at the optimum. Kaplan's 2020 laws got the shape right but the allocation wrong; they told labs to build huge, data-starved models (GPT-3: 175B params, 300B tokens), and Chinchilla proved a 70B model on 1.4T tokens beats them on the same compute. The twist that defines 2024-2026: labs now deliberately violate compute-optimality, training small models far past 20 tokens/param, because the budget that matters in production is inference, not training. Everything downstream — which model you pick, how long you train, whether you repeat data — falls out of this one curve.

2. Why interviewers probe this

• [IC5] — Can you reason quantitatively about a training run instead of cargo-culting "more data good"? Can you compute $C \approx 6ND$, explain the 20:1 ratio, and tell me why GPT-3 was undertrained? This separates engineers who tune knobs from those who understand the budget.
• [IC5] — Do you know the difference between training-optimal and deployment-optimal? An IC5 who says "Llama violates Chinchilla so Meta is wrong" fails; one who explains the inference-amortization argument passes.
• [IC6] — Can you make a defensible capital-allocation decision under a real constraint (finite unique data, fixed GPU-hours, a latency SLA)? Can you reason about epoch decay, the emergent-abilities debate, and when the smooth power law breaks? This is staff-level: you own the recipe, and a wrong call costs seven figures.
• [IC6] — Do you know the limits of the laws — that they predict loss, not capability; that they're fit on a specific data distribution; and that extrapolating them off-distribution is where expensive mistakes live?

3. Concept build-up

3.1 The object being fit: loss as a power law

Scaling laws fit a single scalar — cross-entropy loss $L$ — as a function of scale. The clean, separable form Chinchilla uses is:

Defining every symbol:

• $L$ — the per-token cross-entropy loss (nats), what you actually minimize.
• $N$ — number of model parameters.
• $D$ — number of training tokens.
• $E$ — the irreducible loss, the entropy of natural language itself; no model, however large, beats this floor.
• $A, B$ — fitted scale constants (units chosen so the terms have loss units).
• $\alpha, \beta$ — the exponents that govern how fast loss falls as you add parameters or data. Chinchilla fits $\alpha \approx 0.34$ and $\beta \approx 0.28$.

The two non-constant terms each say: doubling that resource multiplies its loss contribution by a fixed factor ($2^{-\alpha}$ or $2^{-\beta}$). Because $\alpha$ and $\beta$ are close, parameters and data reduce loss at similar rates — which is the mathematical seed of the "scale them together" conclusion. The Kaplan 2020 form was a simpler single-variable law $L(N) \approx (N_c / N)^{\alpha_N}$ that, crucially, used non-embedding parameter counts and a different learning-rate schedule; this is the technical root of why Kaplan and Chinchilla disagree.

3.2 The compute bridge: why $C \approx 6ND$

Compute, parameters, and data are not independent — they're chained by the cost of a forward+backward pass. For a dense transformer, one token costs about $2N$ FLOPs in the forward pass (one multiply-add per weight, counted as 2 FLOPs) and roughly $4N$ in the backward pass, totaling $\approx 6N$ FLOPs per token. Over $D$ tokens:

This is the load-bearing equation of the whole field. It means: fix $C$, and choosing $N$ forces $D = C / (6N)$. You cannot have both big and well-fed for free. Scaling laws are, at heart, a constrained optimization over this one line.

3.3 Kaplan vs Chinchilla: the same shape, the wrong allocation

Kaplan et al. (2020) concluded that under a fixed compute budget you should spend most of it on parameters: their fit implied $N_{\text{opt}} \propto C^{0.73}$ and only $D_{\text{opt}} \propto C^{0.27}$ — model size racing ahead of data. The industry took this literally. GPT-3 (175B params, 300B tokens) is the monument: under 2 tokens/param, wildly data-starved by the standard that came later.

Hoffmann et al. (2022) re-ran the experiment with 400+ models from 70M to 16B params, three independent methods, and a corrected learning-rate schedule (Kaplan's LR decay didn't reach the end of training, biasing his loss estimates). Their finding flipped the allocation: $N_{\text{opt}} \propto C^{0.50}$ and $D_{\text{opt}} \propto C^{0.50}$ — parameters and tokens scale equiproportionally. The 20-tokens-per-parameter rule is the practical summary (Chinchilla itself: 70B params, 1.4T tokens). The proof was Chinchilla (70B) beating Gopher (280B) — a 4× smaller model — on the same compute, by +7% on MMLU and across the benchmark suite, while being far cheaper to fine-tune and serve. The two reconciling technical points an interviewer wants: (1) Kaplan counted non-embedding parameters, distorting small-model fits; (2) his LR schedule under-trained the tail, making more data look less useful than it is.

3.4 The flat valley: why labs now over-train

Here is the subtlety that the 20:1 headline hides. The loss surface near the compute-optimal point is flat. Move to a model 2-3× smaller than optimal and feed it the extra compute as more tokens, and your training loss rises only slightly. Chinchilla-optimal minimizes loss per unit training FLOP — but it completely ignores inference.

Production economics invert the objective. A model is trained once and served billions of times. Inference cost scales with $N$ (≈$2N$ FLOPs per generated token), so a smaller model is permanently cheaper and lower-latency. The rational move: pick the smallest $N$ that hits your quality bar, then pour tokens in well past 20:1 to squeeze its loss down. This is exactly why Llama 3 8B trained on 15T tokens (≈1875 tokens/param, ~90× the Chinchilla ratio) and Llama 3.1 trained ~2 epochs on diverse corpora. The training run is "compute-suboptimal" on purpose — you knowingly burn extra GPU-hours up front to amortize a smaller, faster model across a deployment lifetime. The correct frame: Chinchilla optimizes a training budget; deployment optimizes total cost of ownership = training + (inference cost × expected query volume). When query volume is large, the optimum shifts hard toward smaller, over-trained models.

3.5 Data-constrained scaling: when you run out of text

The clean laws assume unique tokens are infinite. They aren't — high-quality web text is finite, and frontier runs now bump the ceiling. Muennighoff et al. (2023) studied what happens when $D$ exceeds the unique-token supply $D_{\text{unique}}$ and you must repeat data. Findings that matter:

• Repeating data has diminishing returns: the first repeat of a token is nearly as good as fresh data, but value decays roughly exponentially with epoch count.
• Past ~4 epochs, additional passes contribute almost nothing and eventually degrade performance (overfitting tax).
• They fit a modified law where repeated tokens have an effective count smaller than their raw count — a token seen $k$ times is worth less than $k$ fresh tokens, by a decaying factor.

The practical rule of thumb from this work: up to about 4 epochs, repeating data is a reasonable substitute for collecting more; beyond that, you're better off adding parameters, accepting a smaller model, or spending on data acquisition/curation. This is why most 2024-2026 production runs cap at 1-2 epochs on their best data and treat data quality as the binding constraint, not token count.

3.6 Emergent abilities: where the smooth law gets contested

Scaling laws predict loss, which falls smoothly. But Wei et al. (2022) catalogued emergent abilities — tasks (multi-step arithmetic, certain reasoning benchmarks) where accuracy is near-random until a scale threshold, then jumps sharply. If real, emergence means you cannot extrapolate capability from small models even when you can extrapolate loss. The live debate: a strong counter-argument holds that many "emergences" are artifacts of discontinuous metrics (exact-match accuracy) — swap to a smooth metric like per-token log-likelihood and the jump dissolves into a gradual curve. The synthesis emerging in 2024-2026 work frames grokking, double descent, and emergence as the same underlying phenomenon: a memorization-vs-generalization circuit competition, where a phase transition (U-shaped scaling on hard subtasks composing with inverted-U on easy ones) produces the apparent threshold. The honest interview position: loss scaling is robust and predictive; capability scaling is partly a measurement question and partly real phase-transition behavior, and you should not bet a roadmap on a single emergence threshold.

4. Minimal implementation

Fit a Chinchilla-style law from run data and solve for the compute-optimal split. This is the back-of-envelope every staff engineer should be able to run before approving a training budget.

import numpy as np
from scipy.optimize import curve_fit
 
# --- 1. Fit L(N, D) = E + A/N^alpha + B/D^beta from observed runs ---
# Each row: (params N, tokens D, measured final loss). Real runs, not toy.
runs = np.array([
    [70e6,   1.4e9,  3.40],
    [160e6,  3.2e9,  3.12],
    [410e6,  8.2e9,  2.88],
    [1.0e9,  20e9,   2.66],
    [2.8e9,  55e9,   2.49],
    [6.9e9,  140e9,  2.36],
])
N, D, L = runs[:, 0], runs[:, 1], runs[:, 2]
 
def loss_law(X, E, A, B, alpha, beta):
    n, d = X
    return E + A * n**(-alpha) + B * d**(-beta)
 
# Fit in a numerically sane range; bounds keep exponents in (0,1).
p0 = [1.7, 400.0, 400.0, 0.34, 0.28]
(E, A, B, alpha, beta), _ = curve_fit(
    loss_law, (N, D), L, p0=p0,
    bounds=([0, 0, 0, 0.05, 0.05], [4, 1e6, 1e6, 1.0, 1.0]),
    maxfev=100000,
)
print(f"E={E:.3f}  A={A:.1f}  B={B:.1f}  alpha={alpha:.3f}  beta={beta:.3f}")
 
# --- 2. Given a compute budget, solve the optimal N, D split ---
def compute_optimal(C, A, B, alpha, beta):
    """Minimize L over N,D subject to C = 6*N*D. Grid search on N."""
    Ns = np.logspace(8, 11, 4000)          # 0.1B .. 100B params
    Ds = C / (6.0 * Ns)                     # compute constraint
    losses = E + A * Ns**(-alpha) + B * Ds**(-beta)
    i = np.argmin(losses)
    return Ns[i], Ds[i], losses[i]
 
C = 1.2e21  # FLOPs
N_opt, D_opt, L_opt = compute_optimal(C, A, B, alpha, beta)
print(f"C={C:.1e} FLOPs -> N*={N_opt/1e9:.2f}B  D*={D_opt/1e9:.1f}B  "
      f"tokens/param={D_opt/N_opt:.1f}  loss={L_opt:.3f}")

What each block does. Block 1 fits the five constants of the separable law by nonlinear least squares; the bounds keep the exponents physically meaningful and the irreducible loss $E$ non-negative. Block 2 enforces the compute bridge $C = 6ND$ by parameterizing the whole feasible set with a single variable $N$ (since $D$ is then determined), evaluating loss along that line, and taking the minimum — exactly the constrained optimization of §3.2. The printed tokens/param should land near 20 when your fit recovers $\alpha \approx \beta$; if it doesn't, your fit (or your run data) is telling you the exponents diverge, which is itself a finding. The same compute_optimal function, swept over a range of $C$, traces the compute-optimal frontier you'd put in a planning doc.

5. Production tradeoffs

Cost. Training is a one-time $C \approx 6ND$; inference is recurring $\approx 2N$ FLOPs per output token times every query forever. The crossover math: extra training tokens to over-train cost $6N \cdot \Delta D$ once; the inference savings from a smaller $N$ accrue per query. Above a volume threshold (often millions of queries — easily reached for any deployed product) the over-trained small model wins on total cost of ownership.

Latency / quality. Smaller $N$ means lower per-token latency and higher throughput (more requests per GPU), the dominant serving lever. Over-training buys back the quality you'd lose from shrinking $N$ — up to a point of diminishing returns set by the $B/D^{\beta}$ term flattening.

Failure modes. (1) Extrapolating off-distribution — laws are fit on a fixed data mix; change the corpus (more code, more multilingual) and the constants move. (2) Data exhaustion — pushing tokens/param high enough that you exceed unique data and silently start repeating, paying the >4-epoch overfitting tax. (3) Confusing loss with capability — a lower loss does not guarantee an emergent skill cleared its threshold; eval on the actual task. (4) Stale $6ND$ — MoE and quantized training break the dense FLOP accounting; for MoE use active parameters per token, not total.

What changes at scale. At frontier compute the binding constraint stops being FLOPs and becomes unique high-quality tokens and memory bandwidth (see /inference). Allocation decisions shift from "how big a model" to "how do I get more good data" (synthetic data, curation, multi-epoch with quality filtering). MoE changes the curve: it decouples total parameters (capacity) from active parameters (per-token cost), letting you grow capacity while holding inference FLOPs roughly fixed — a different point on the cost/quality frontier than dense scaling laws describe.

6. How it's asked

7. Pitfalls & flashcards

• Quoting "20 tokens/param" as a law of nature. It's the compute-optimal ratio for one objective. Production over-trains 5-100× past it on purpose; quote the ratio and the inference-amortization caveat.
• Forgetting the constant 6. $C \approx 6ND$ (forward $2N$ + backward $4N$). People drop the factor and miscalculate budgets by 3×.
• Comparing Kaplan and Chinchilla numbers directly. Kaplan used non-embedding parameters and an under-decayed LR schedule; the exponents aren't apples-to-apples. The disagreement is methodological, not metaphysical.
• Assuming infinite data. Past ~4 epochs of repeated data, returns collapse and overfitting begins; effective token count is below raw count when you repeat.
• Treating loss as capability. Loss falls smoothly; some capabilities appear at thresholds (debated as real phase transitions vs metric artifacts). Eval the task, don't infer it from loss.
• Applying dense laws to MoE. Use active parameters per token for inference cost and the FLOP bridge; total parameters describe capacity, not per-token compute.

Flashcard. Chinchilla picks the cheapest training point ($\approx$20 tokens/param via $C\approx 6ND$); production deliberately over-trains a smaller model because inference is paid per query, forever — total cost of ownership, not training FLOPs, sets the real optimum.

8. Further reading

• Hoffmann et al., 2022 — Training Compute-Optimal Large Language Models (Chinchilla): the equiproportional result, the 20:1 ratio, and the Gopher comparison. arxiv.org/abs/2203.15556 · NeurIPS PDF
• Kaplan et al., 2020 — Scaling Laws for Neural Language Models: the original power laws and the parameter-heavy allocation that Chinchilla corrected. arxiv.org/abs/2001.08361
• Muennighoff et al., 2023 — Scaling Data-Constrained Language Models: the data-repetition law and the ~4-epoch cliff. arxiv.org/abs/2305.16264
• Wei et al., 2022 — Emergent Abilities of Large Language Models: the emergence catalogue and the threshold debate. arxiv.org/abs/2206.07682

Next: /inference — where the $2N$-per-token serving cost that drives over-training actually gets paid, and how to make it cheaper.