Text classification using a Transformer encoder

What's actually being asked

Walk through a complete Transformer-encoder-based text classifier (BERT-style), end to end: how raw text becomes tokens, how the encoder turns those into contextual vectors, how one vector represents the whole sentence, how a small head turns that into a class, and how it's trained and used.

Answer

Pipeline, at a glance

raw text → tokenizer → [CLS] + tokens + [SEP] → Transformer encoder (bidirectional self-attn, no mask) → pooled vector → classification head → softmax over classes

1. Tokenization

The input sentence is split into subword tokens using the model's tokenizer (WordPiece for BERT, BPE for RoBERTa-family models). Two special tokens are added: [CLS] at the start (reserved for the pooled/classification representation) and [SEP] at the end (marks sentence boundary, also used between sentence pairs for tasks like NLI). The tokenizer outputs input_ids (vocabulary indices) and an attention_mask (1 for real tokens, 0 for padding, so padded positions get -∞ added to their attention scores and contribute nothing).

2. Contextual encoding

Token + positional embeddings are fed through a stack of encoder blocks. Unlike the decoder, the encoder's self-attention is bidirectional — no causal mask — so every token attends to every other token in the sequence, both left and right context:

Q = XW_Q, K = XW_K, V = XW_V Attention(Q,K,V) = softmax(QK^T/√d_k) V

with multi-head attention MultiHead(Q,K,V) = Concat(head_1,...,head_h) W_O exactly as in the canonical formula. Each encoder block wraps this (and a feed-forward sub-layer) in residual connections + LayerNorm, same as the decoder's sub-layers. The output is one contextual vector per input token — each vector now encodes that word's meaning in context, not in isolation (so "bank" in "river bank" and "bank account" get different vectors).

3. Pooling / CLS representation

A classifier needs one fixed-size vector for the whole sentence, not one per token. Two standard approaches:

• CLS pooling (BERT's default): take the final hidden state at the [CLS] position — since [CLS] attended to every token, it has learned (via training) to summarize the whole sequence.
• Mean pooling: average the hidden states of all real (non-padding) tokens — sometimes used, often with sentence-embedding models.

Either way, the result is a single vector h_pool ∈ R^d.

4. Classification head

A small feed-forward layer on top: logits = W h_pool + b, where W ∈ R^{C×d} for C classes, followed by P(class) = softmax(logits). For binary sentiment this is a 2-way softmax (or a single sigmoid logit); for multi-class topic tagging, C classes.

5. Training objective

Cross-entropy loss between the predicted class distribution and the true one-hot label:

L = −Σ_c y_c log(ŷ_c)

where y_c is 1 for the correct class and 0 otherwise, ŷ_c is the predicted probability for class c. In practice the whole network — the pretrained encoder weights and the newly-added classification head — is fine-tuned jointly on the labelled dataset, with the pretrained weights as a strong starting point rather than training from scratch.

6. Prediction

At inference, the same forward pass runs (no labels needed), and the predicted class is argmax_c ŷ_c — the class with the highest softmax probability.

Worked example

Sentence: "The delivery was fast but the packaging was damaged." Sentiment classification, 2 classes.

1. Tokenize → [CLS] The delivery was fast but the packaging was damaged . [SEP]
2. Encoder produces one contextual vector per token, picking up that "fast" (positive) and "damaged" (negative) are in tension.
3. Pool the [CLS] vector → h_pool.
4. Head → logits [1.8, 2.3] (illustrative numbers) → softmax → [0.38, 0.62].
5. argmax → class 1 → predicted label: mixed/negative-leaning, depending on how the label set was defined.

Exam tip

Don't confuse this with the decoder: the encoder's self-attention has no causal mask — every position sees the full sentence, both directions, which is exactly why encoders are good at understanding/classification and decoders are good at generation. If asked to name the model family, say BERT (encoder-only, classification/understanding tasks) vs GPT (decoder-only, generation).