Cross-attention: decoder using encoder context

What's actually being asked

Explain how cross-attention lets a Transformer decoder use encoder information while generating output, with the Query/Key/Value roles, illustrated with a machine-translation example.

Answer

The mechanism

Cross-attention uses the exact same formula as self-attention:

Attention(Q,K,V) = softmax(QK^T/√d_k) V

but with a crucial difference in where Q, K, V come from:

• Query (Q) comes from the decoder's own sequence — specifically, the output of the decoder's masked self-attention sub-block at that point.
• Key (K) and Value (V) both come from the encoder's final output — the contextual representations of the entire source sequence.

So instead of a sequence attending to itself (self-attention), the decoder's current state asks a question ("given what I've generated so far, what in the source sentence is relevant right now?") and the encoder's output answers it. This is literally how encoder information crosses over into the decoder — hence "cross"-attention.

Machine-translation walkthrough

Translating English "The cat sat on the mat" → French, where the decoder has already generated "Le chat" and is now deciding the next word.

1. The encoder has already processed the full English sentence and produced one contextual vector per English token: K, V = these 6 encoder output vectors ("The", "cat", "sat", "on", "the", "mat").
2. The decoder's masked self-attention looks at what's been generated so far ("Le chat") and produces a query vector Q representing "what should come next, given 'Le chat' so far."
3. Cross-attention computes softmax(QK^T/√d_k) — this scores the query against each of the 6 encoder vectors. This step should assign high attention weight to the encoder vector for "sat" (the next concept to translate — "s'est assis" in French), and lower weight to "The", "on", "the", "mat" at this point.
4. The weighted sum over V (the encoder's value vectors, weighted by those scores) gives the decoder the specific piece of source-sentence meaning it needs right now, which flows into the rest of the decoder layer (FFN, etc.) to help produce the next output token.

As generation proceeds word by word, the query changes each step, so the cross-attention weights shift to point at different source words — this is the mechanism that replaced the older RNN "attention over encoder states" idea, generalized into the Transformer's uniform attention formula.

Why this matters

Without cross-attention, the decoder would only have access to the source sentence through a single fixed vector (like the old seq2seq encoder bottleneck), forcing the whole sentence's meaning through one compressed point. Cross-attention instead lets the decoder look directly, and selectively, at every encoder position at every decoding step — directly solving the long-sequence information bottleneck that plagued RNN-based translation.

Exam tip

The one-line fact examiners check for explicitly: "Q from decoder, K and V from encoder" — stating this correctly (and not reversing it) is the single most-checked point on cross-attention questions. Pairing it with a concrete example (which source word should get high attention weight at which decoding step) turns a bookish answer into a strong one.