Positional Encodings: Why RoPE Won

1. Quick anchor

Self-attention treats its input as a set, not a sequence: shuffle the tokens and the math produces the same shuffled output, because attention scores are just dot products between content vectors with no notion of "before" or "after." Positional encoding is the mechanism that injects order back in. The field walked a path — fixed sinusoids (2017), learned absolute embeddings, then relative schemes, then the two that survived: RoPE, which rotates query and key vectors by an angle proportional to their position so the dot product depends only on the relative offset, and ALiBi, which adds a distance-proportional penalty straight to the attention scores. RoPE is the 2026 default (LLaMA, Qwen, Gemma, Mistral) because it encodes relative position with zero extra parameters and — critically — its frequencies can be rescaled at inference time to stretch an 8K-trained model to 128K with minimal fine-tuning. This lesson derives why, and shows you the position-interpolation and YaRN tricks that make long context actually work.

2. Why interviewers probe this

• IC5 signal: Can you derive permutation invariance from the attention formula, not just recite "attention has no order"? Do you understand absolute vs relative position and why the √d_k scaling and positional encoding are different concerns people often conflate? Can you implement RoPE correctly (the rotation, the per-frequency angles) without copying it?
• IC6 signal: Can you reason about context extension as an engineering decision — what changes in the KV cache, the attention entropy, the perplexity curve when you go from 8K to 128K? Do you know why RoPE extrapolates badly out-of-the-box but interpolates well, and can you compare PI vs NTK vs YaRN at the level of "which RoPE dimensions get scaled and by how much"? Can you connect positional choice to inference cost (KV cache, FlashAttention compatibility) and to failure modes like "lost in the middle"?

3. Concept build-up

3.1 The problem: attention can't see order

Start from scaled dot-product attention (Vaswani et al., 2017). For queries, keys, values $Q, K, V \in \mathbb{R}^{n \times d}$:

Every symbol: $Q$ is the stack of query vectors (one per position, dimension $d_k$), $K$ the keys, $V$ the values; $QK^\top$ is the $n \times n$ matrix of all pairwise dot products (the raw attention scores); $\sqrt{d_k}$ rescales so the dot products don't blow up and saturate softmax; softmax normalizes each row to a probability distribution over positions.

Now the key fact. Let $P$ be a permutation matrix (it reorders rows). Permute the input rows of $Q, K, V$ by $P$. Because the operation is built entirely from dot products and a row-wise softmax:

The output is just the same rows, reordered. The function commutes with permutation — it is permutation-equivariant, and if you also permute the output back, invariant to order. Concretely: with no positional signal, "the cat sat on the mat" and "mat the on sat cat the" produce identical token representations (up to reordering). The model literally cannot distinguish word order. That is fatal for language, where "dog bites man" ≠ "man bites dog."

So we must break the symmetry. Every positional scheme is a different answer to: where do we inject "where," and is it absolute or relative?

3.2 Absolute: sinusoids and learned embeddings

The original transformer added a fixed sinusoidal signal to the token embeddings before layer 1:

Here $pos$ is the integer position, $i$ indexes the embedding dimension pair, $d$ is the model dimension. Each dimension pair is a sinusoid of a different wavelength — short wavelengths in early dims, very long ones later, so the vector forms a smooth multi-frequency "clock" that uniquely stamps each position. The clever bit: $PE_{pos+k}$ is a linear function of $PE_{pos}$ (a rotation by $k$ steps in each frequency), so relative offsets are in principle recoverable. Zero learnable parameters.

The alternative — learned absolute embeddings (BERT, GPT-2) — just learns a lookup table of one vector per position up to a max length. Simple and slightly better in-distribution, but it has a hard wall: position 4097 has no embedding if you trained to 4096, and it cannot extrapolate at all.

Both share a deeper weakness: they bolt position onto the input once, then let it diffuse through the stack. The attention scores at layer 20 only see position through whatever survived 20 layers of mixing. And neither makes the score between tokens $m$ and $n$ depend cleanly on $m-n$. Language overwhelmingly cares about relative distance ("the adjective two words back"), so this is the wrong inductive bias.

3.3 RoPE: rotate the query and key

Rotary Position Embedding (Su et al., RoFormer, 2021) is the idea that won. Instead of adding to embeddings, it rotates the query and key vectors inside each attention layer by an angle proportional to position.

Group the head dimension $d$ into $d/2$ pairs. For each pair $i$, define a frequency $\theta_i = 10000^{-2i/d}$. A query at position $m$ gets each pair rotated by angle $m\theta_i$; a key at position $n$ by $n\theta_i$. In 2D, rotating $q$ by $m\theta$ and $k$ by $n\theta$ and taking their dot product gives:

where $R(\phi)$ is the 2×2 rotation by angle $\phi$. The two absolute rotations fuse into one rotation by the difference $(n-m)\theta$. This is the whole magic: after rotation, the attention score between positions $m$ and $n$ depends only on the relative offset $n-m$, never on the absolute indices. Stack this across all $d/2$ frequency pairs and you get a rich relative-position signal — high-frequency pairs resolve nearby tokens sharply, low-frequency pairs carry long-range information.

Why RoPE beat the field:

• Relative, but cheap. True relative encodings (Shaw et al.) added learned bias tensors that broke fast-attention kernels. RoPE gets relative position with zero extra parameters and as a pure elementwise rotation, so it's FlashAttention-compatible — you rotate q and k before the kernel, nothing else changes.
• Inner products preserved. Rotation is orthogonal: $\|R(\phi)q\| = \|q\|$. Position changes angle, not magnitude, so it doesn't corrupt the content signal.
• Decaying long-range dependency. Summing rotated pairs across frequencies makes the expected attention between far-apart tokens decay with distance — a sensible prior baked in for free.
• It's stretchable. Because position enters as a continuous angle $m\theta_i$, you can rescale $m$ or $\theta_i$ at inference to fit longer sequences (Section 3.5). Learned embeddings can't do this; sinusoids extrapolate poorly.

3.4 ALiBi: skip embeddings, bias the scores

ALiBi (Press et al., ICLR 2022) takes the opposite, blunter route: add nothing to the embeddings, and instead penalize attention by distance directly in the score matrix. For a query at $i$ and key at $j$:

where $\alpha_h$ is a fixed per-head slope (heads get a geometric series of slopes, e.g. $1/2, 1/4, 1/8, \dots$). The further away a key is, the bigger the linear penalty, so each head has a soft "attention horizon" — some heads see locally, some globally. No learned position parameters at all.

ALiBi's headline result is length extrapolation: train on $L=1024$, run at $2L=2048$, and perplexity stays on par with a model trained at 2048 — plus ~11% faster training and ~11% less memory than the sinusoidal baseline. The reason is intuitive: the bias is a smooth, unbounded function of distance, so positions beyond training still produce sensible (if increasingly penalized) scores. The limitation: beyond ~4× training length the linear penalty over-suppresses distant tokens and quality degrades; it never uses far context the way RoPE-extension methods can. ALiBi shows up in MPT and BLOOM; RoPE won the broader race because it preserves long-range information rather than discarding it.

3.5 Context extension: stretching RoPE to 128K

Here's the production reality. You pretrain with RoPE on, say, 8K tokens. Naively running at 128K extrapolates — positions $m$ now reach angles $m\theta_i$ far outside any the model saw during training, especially in the high-frequency pairs, whose angles wrap around many times. Attention goes haywire; perplexity explodes. The fix is to interpolate, not extrapolate: keep positions inside the trained angular range.

Position Interpolation (PI; Chen et al., 2023). Scale every position down by $s = L'/L$ (extended length / original length). Position 100,000 in a 128K window with $L=8\text{K}$ becomes $100000 \times 8/128 = 6250$ — comfortably in-range. Equivalently, divide all frequencies by $s$. PI is a one-line change and works, but it scales all dimensions uniformly, which over-compresses the high-frequency pairs that resolve adjacent tokens — you lose local resolution. Needs fine-tuning on ~1000 steps of long data.

NTK-aware / NTK-by-parts. The insight: don't scale frequencies uniformly. High-frequency dims (short wavelength) handle local relationships and already cycle many times within the training window — interpolating them hard destroys fine detail. Low-frequency dims (long wavelength) carry the long-range signal and must be interpolated to reach new positions. So scale per dimension: leave high frequencies nearly untouched, interpolate low frequencies aggressively.

YaRN (Peng et al., ICLR 2024) formalizes NTK-by-parts into the current default. It splits RoPE dimensions into three bands by wavelength: dimensions whose wavelength is much shorter than the context get no scaling (preserve local detail); dimensions whose wavelength is longer than the context get full PI-style interpolation; in between, a smooth ramp blends the two. YaRN adds one more trick — a temperature factor on the attention logits ($\sqrt{1/t}$ scaling) that re-sharpens the attention distribution, which otherwise flattens as sequences grow longer. Result: YaRN reaches the target context with fine-tuning on <0.1% of pretraining tokens, and Dynamic-YaRN achieves ~2× extension at inference with no fine-tuning by scaling dynamically with the current sequence length. This is why a model card says "8K native, 128K with YaRN" — same weights, frequencies rescaled.

The mental model: extrapolation asks the model to imagine angles it never saw; interpolation re-maps long sequences into angles it already understands. RoPE wins context extension precisely because position is a continuous, rescalable angle.

4. Minimal implementation

Real RoPE, the way it's applied in LLaMA-class models — precompute the per-position cos/sin tables, then rotate q and k before attention. This is runnable and production-shaped (the rotate_half trick is exactly what HF and vLLM use).

import torch
 
def build_rope_cache(seq_len: int, head_dim: int, base: float = 10000.0,
                     scale: float = 1.0, device="cpu"):
    """Precompute cos/sin tables. `scale` > 1 implements linear Position Interpolation
    (PI): positions are divided by `scale` to map a long sequence into the trained range."""
    assert head_dim % 2 == 0, "RoPE needs an even head dimension"
    # frequencies theta_i = base^(-2i/d), one per dimension-pair, repeated to full dim
    i = torch.arange(0, head_dim, 2, device=device).float()
    inv_freq = base ** (-i / head_dim)                       # [head_dim/2]
    pos = torch.arange(seq_len, device=device).float() / scale  # PI: shrink positions
    angles = torch.outer(pos, inv_freq)                      # [seq_len, head_dim/2]
    angles = torch.cat([angles, angles], dim=-1)             # [seq_len, head_dim]
    return angles.cos(), angles.sin()
 
def rotate_half(x):
    # split in half and rotate: [x1, x2] -> [-x2, x1]  (the 2D rotation, vectorized)
    x1, x2 = x.chunk(2, dim=-1)
    return torch.cat([-x2, x1], dim=-1)
 
def apply_rope(q, k, cos, sin):
    # q, k: [batch, heads, seq, head_dim]; cos/sin: [seq, head_dim] -> broadcast
    cos, sin = cos[None, None], sin[None, None]
    q_rot = q * cos + rotate_half(q) * sin
    k_rot = k * cos + rotate_half(k) * sin
    return q_rot, k_rot
 
# --- demo: confirm scores depend only on relative offset ---
torch.manual_seed(0)
B, H, T, D = 1, 1, 16, 8
q = torch.randn(B, H, T, D); k = torch.randn(B, H, T, D)
cos, sin = build_rope_cache(T, D)
q_r, k_r = apply_rope(q, k, cos, sin)
scores = (q_r @ k_r.transpose(-1, -2)) / D**0.5
 
# same query/key content at offsets (2,5) and (8,11) -> identical score (relative!)
qc = torch.randn(D); kc = torch.randn(D)
def score_at(m, n):
    qm = (qc * cos[m] + rotate_half(qc[None])[0] * sin[m])
    kn = (kc * cos[n] + rotate_half(kc[None])[0] * sin[n])
    return (qm @ kn) / D**0.5
print(score_at(2, 5).item(), score_at(8, 11).item())  # ~equal: offset 3 both times

The two printed scores match: identical content at the same relative offset yields the same attention score regardless of absolute position — exactly the property derived in 3.3. Flip scale to 16.0 and you've implemented PI for a 16× longer context; swap the uniform inv_freq shrink for a per-dimension band schedule and you've implemented YaRN. Note RoPE touches only q and k, never V, and adds zero parameters — which is why it slots cleanly in front of a FlashAttention kernel.

5. Production tradeoffs

Cost/latency. RoPE and ALiBi both add ~nothing to memory and don't inflate the KV cache (you cache the rotated keys; no separate position tensor). The dominant long-context cost is attention itself — $O(n^2)$ — which is why context extension pairs with FlashAttention-2/3. RoPE's elementwise rotation is a negligible fraction of attention FLOPs.

Quality / failure modes.

• Raw RoPE extrapolation collapses. Running past trained length without PI/YaRN spikes perplexity — high-frequency angles wrap into never-seen territory. Always interpolate.
• PI over-compresses local detail because it scales all dims uniformly; you can measure this as degraded performance on tasks needing precise short-range structure (code, math). YaRN's per-band scaling fixes it.
• Attention entropy flattens at long context — more tokens dilute the softmax; YaRN's temperature term re-sharpens it. Skip it and long-context recall suffers.
• ALiBi discards far context. Its monotone penalty means it can't strongly attend to a token 50K tokens back even if relevant — fine for streaming, bad for retrieval-style long-context.
• "Lost in the middle" persists across all schemes: even with correct RoPE+YaRN, models under-attend to mid-context tokens. Positional encoding enables long context; it doesn't guarantee uniform use of it (see /context-engineering).

What changes at scale. At 128K–1M context, the engineering shifts from "which encoding" to "how do I serve $n^2$ attention" — chunked prefill, KV-cache quantization, and ring/sequence-parallel attention dominate (see /inference). The positional choice is settled (RoPE+YaRN); the open work is the systems layer around it.

6. How it's asked

7. Pitfalls & flashcards

• Conflating √d_k scaling with positional encoding. The $\sqrt{d_k}$ divisor controls softmax saturation; positional encoding controls order. Different problems — don't blend them in an answer.
• Thinking RoPE adds parameters. It adds none — it's a fixed rotation applied to q and k. If you're caching a separate position tensor, you've done it wrong; cache the rotated keys.
• Running past trained length raw. Without PI/YaRN, RoPE extrapolation collapses. "It supports 128K" almost always means "with YaRN," not natively.
• Uniform interpolation everywhere. PI's one-size scaling kills local resolution; prefer per-frequency (NTK-by-parts/YaRN) for anything length-sensitive.
• Forgetting the temperature term. Long context flattens attention entropy; YaRN's logit temperature is doing real work, not a footnote.
• Assuming long context = used context. "Lost in the middle" is orthogonal to encoding; positional encoding enables the window, it doesn't guarantee uniform attention across it.

Flashcard. RoPE rotates q and k by position × frequency; because rotations compose, the score between positions m and n depends only on n − m — relative position, zero parameters, and (because position is a rescalable angle) stretchable to long context via PI/YaRN.

8. Further reading

• Vaswani et al., 2017 — Attention Is All You Need — sinusoidal baseline and the original attention formula (https://arxiv.org/abs/1706.03762).
• Su et al., 2021 — RoFormer: Rotary Position Embedding — the RoPE derivation (https://arxiv.org/abs/2104.09864).
• Press et al., 2022 — Train Short, Test Long: ALiBi — linear-bias extrapolation (https://arxiv.org/abs/2108.12409).
• Chen et al., 2023 — Extending Context Window via Position Interpolation (https://arxiv.org/abs/2306.15595).
• Peng et al., 2024 — YaRN: Efficient Context Window Extension — NTK-by-parts + temperature (https://arxiv.org/abs/2401.09836).

Next: /transformers/attention — how the scores you just learned to position get computed efficiently (FlashAttention), then /inference for serving long context at scale.