Positional encoding: derivation and role in Transformers

What's actually being asked

Explain why Transformers need positional encoding, derive the sinusoidal PE formula, and explain how it's combined with token embeddings.

Answer

Why positional encoding is needed

Self-attention treats the input as a set, not a sequence — Attention(Q,K,V) = softmax(QK^T/√d_k)V has no notion of order baked in; swapping two tokens' positions just permutes the rows of Q, K, V, and the attention computation itself doesn't know which position came first. But order obviously matters in language ("dog bites man" ≠ "man bites dog"). So position has to be injected explicitly — added to the embeddings before the first layer. That's what positional encoding does.

The sinusoidal formula

PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

• pos — the token's position in the sequence (0, 1, 2, ...).
• i — the index of the pair of dimensions within the embedding (dimension 2i gets a sine, 2i+1 gets the matching cosine).
• d_model — the embedding dimension.
• 10000^(2i/d_model) — a wavelength term: for small i (early dimensions), the exponent is near 0, so this term is close to 1, giving a short wavelength (fast-oscillating, sensitive to small position changes); for large i (later dimensions), the exponent approaches 1, giving a very long wavelength (slowly-oscillating, sensitive to large position changes). Stacking many dimensions across this geometrically increasing range of wavelengths gives every position a unique, smoothly-varying fingerprint across the embedding.

Why sine/cosine pairs specifically

A key property: PE(pos+k) can be expressed as a linear function of PE(pos) for any fixed offset k, because of the sine/cosine angle-addition identities — sin(a+b) and cos(a+b) expand into linear combinations of sin(a), cos(a), sin(b), cos(b). Concretely, for a frequency ω:

sin(ω(pos+k)) = sin(ωpos)cos(ωk) + cos(ωpos)sin(ωk) cos(ω(pos+k)) = cos(ωpos)cos(ωk) − sin(ωpos)sin(ωk)

Since cos(ωk) and sin(ωk) are just constants once k is fixed, both expressions are linear combinations of sin(ωpos) and cos(ωpos) — i.e. of PE(pos)'s own components. This means attention can, in principle, learn to pick up relative position (e.g. "the token 3 positions back") via a simple linear operation on the positional encodings, not just absolute position — useful since relative distance between words usually matters more than absolute index.

Combining with token embeddings

Positional encoding is added, element-wise, to the token embedding before the first encoder/decoder layer:

input_i = TokenEmbedding(token_i) + PE(i)

Both vectors have dimension d_model, so the addition is well-defined. It's a sum, not a concatenation — the model learns to disentangle "what word" from "what position" from one combined vector, which works in practice because the two signals occupy largely different, learnable patterns in the embedding space.

Learned vs sinusoidal

The original Transformer uses the fixed sinusoidal formula above — no extra parameters to learn, and it can extrapolate to sequence lengths longer than anything seen in training, since the formula is defined for any pos. Some later models instead use learned positional embeddings (a trainable lookup table per position) — simpler, but limited to the maximum sequence length seen during training.

Exam tip

If asked to "derive" the formula, don't just state it — show the angle-addition identity reasoning (why PE(pos+k) is a linear function of PE(pos)) and the wavelength intuition for why 10000^(2i/d_model) sits in the denominator. Those two points separate "memorized the formula" from "understands the formula."