Stable Diffusion's text-to-image pipeline end to end

What's actually being asked

Walk through the full Stable Diffusion text-to-image pipeline end to end — text encoder, latent space, noise addition, U-Net denoising, conditioning, decoder — then weigh its pros and cons for a product-image generation use case.

Answer

The pipeline, stage by stage

1. Text encoder. The prompt ("red leather wallet on a white background") is tokenized and passed through a pretrained text encoder (CLIP's text encoder, in the original Stable Diffusion) to get a sequence of contextual embeddings — one vector per token, capturing meaning, not just which words were used.

2. Latent representation. Unlike plain DDPM, which diffuses in pixel space, Stable Diffusion first compresses the image into a much smaller latent using a pretrained VAE encoder (e.g. a 512×512×3 image → a 64×64×4 latent). All the diffusion happens in this compressed latent — this is what makes SD cheap enough to run on a single consumer GPU.

3. Noise addition (forward process). During training, the DDPM forward process is applied to the latent z_0 instead of raw pixels: z_t = √ᾱ_t z_0 + √(1−ᾱ_t) ε, ε ~ N(0,I).

4. U-Net-based denoising (reverse process). A U-Net is trained to predict the noise ε_θ(z_t, t, c) that was added, where c is the text conditioning from step 1. Loss: 𝔼[‖ε − ε_θ(z_t,t,c)‖²].

5. Conditioning mechanism. The text embeddings are injected into the U-Net via cross-attention layers inside its blocks — the U-Net's own feature maps act as the attention Query, text embeddings act as Key and Value: Attention(Q,K,V) = softmax(QK^T/√d_k) V. This is exactly how "red leather wallet" steers the image instead of the model generating an arbitrary product.

6. Decoder. Once the reverse process has denoised the latent back to a clean z0_hat (over many steps, e.g. 20–50 with a fast sampler), the VAE decoder upsamples it from latent space back to a full-resolution image.

Chain in one line: prompt → text encoder → text embeddings → [noisy latent → U-Net(cross-attn with text) → denoise] ×T → clean latent → VAE decoder → product image.

Why generate in latent space, not pixels

Diffusing directly in pixel space (plain DDPM) is expensive at high resolution — every one of the T steps processes the full image. Compressing to a 48–64× smaller latent first makes every diffusion step correspondingly cheaper, with the VAE handling fine pixel detail separately.

Advantages

• Much cheaper than pixel-space diffusion — runs on a single consumer GPU, useful for an e-commerce team generating many product variants on a budget.
• Flexible conditioning: the same cross-attention mechanism that takes text can take other signals (e.g. a reference product sketch via ControlNet), so the pipeline generalizes beyond pure text prompts.
• Strong sample diversity and quality, inherited from diffusion's stable training — useful for generating many distinct catalogue-style images from one description.

Limitations

• Still slow relative to a GAN at inference — even 20–50 steps is far more compute per image than a single GAN forward pass, which matters at e-commerce catalogue scale.
• Fine text rendering, exact counts, and precise spatial/layout relationships ("logo on the top-left corner") remain weak points — the text embedding doesn't encode that kind of structured, compositional detail cleanly.
• Quality is capped by the pretrained VAE's compression — any detail lost compressing to latent space is a ceiling the U-Net can't exceed, a real concern for product images where exact colour/texture fidelity matters.
• Prompt sensitivity: small rewordings can produce noticeably different images, making precise brand/product control harder than a deterministic rendering pipeline.

Exam tip

Examiners often check whether you know where the text actually enters the network — say explicitly that conditioning happens via cross-attention inside the U-Net (Q from image features, K/V from text embeddings), not by simply concatenating the text vector to the input, which is a common wrong answer.