DATASCI 447 Lecture 27: Escaping the Prior Hole — The Diffusion Paradigm

Administrative Stuff

THE STORY SO FAR

We’ve built a complete text-to-image pipeline:

• VQ-VAE: compress images into discrete codebook tokens (L19-20)
• GPT: autoregressive generation of token sequences (L24)
• CLIP: text embeddings as conditioning signal (L26)

THE VQ-VAE CEILING AND THE CONTINUOUS ALTERNATIVE

Discrete codebooks bottleneck generation in two ways:

1. Finite vocabulary for an infinite visual world. A codebook with 512 entries divides the continuous visual manifold into 512 bins. Fine details between bins are lost.

2. No interpolation between concepts. In a discrete space, there’s no “halfway between code 237 and code 238.” The space between codebook entries is empty.

Continuous VAEs solve this — they map images to a dense, low-dimensional manifold. No discretization, no codebook limits. With perceptual and adversarial losses (L19), reconstructions can be sharp and detailed.

• An infinite conceptual canvas. Smooth interpolation works naturally.

THE VQ-VAE CEILING AND THE CONTINUOUS ALTERNATIVE

THE KL TUG-OF-WAR

The fatal flaw of continuous VAEs: the KL penalty.

The VAE loss has two terms fighting each other:

\[\mathcal{L}_{\text{VAE}} = \underbrace{\text{Reconstruction}}_{\text{wants complex, detailed latents}} + \beta \cdot \underbrace{D_{KL}(q(\mathbf{z}|\mathbf{x}) \| \mathcal{N}(\mathbf{0}, \mathbf{I}))}_{\text{wants simple Gaussian latents}}\]

• Reconstruction wants the encoder to preserve every detail — spread the latents out, use all dimensions, be expressive.

• The KL term wants the encoder to collapse everything onto a unit Gaussian — simple, smooth, undifferentiated.

These goals are fundamentally in conflict.

• The encoder can’t simultaneously be expressive AND match a simple Gaussian. It compromises — and the compromise is ugly.

THE PRIOR HOLE

The true posterior \(q(\mathbf{z}|\mathbf{x})\) looks like Swiss cheese — dense regions where training data maps to, surrounded by empty gaps where nothing lives.

• During training, the encoder produces latents that cluster in specific regions. The KL penalty pushes them toward a Gaussian, but not hard enough to fill the entire space.

THE PRIOR HOLE

THE QUESTION

We want:

• Continuous latent spaces (no discretization bottleneck)
• Sharp, detailed reconstructions (perceptual + adversarial losses)
• A prior we can actually sample from (no holes)

The KL penalty was our tool for making the prior matchable. But it fights reconstruction quality.

THE PARADIGM SHIFT

Diffusion doesn’t assume a shape for the prior. It learns it.

THE CORE IDEA

Two processes:

Forward process (fixed, no learning): gradually add noise to data until it becomes pure Gaussian noise. This is the “walk to the Gaussian.”

Reverse process (learned): gradually remove noise from Gaussian noise until it becomes data. This is “retracing your steps.”

THE FORWARD PROCESS: SMEARING THE MANIFOLD

Visualize what the forward process does to the data distribution:

• Start: data lives on a concentrated, low-dimensional manifold
• Each step adds a tiny bit of Gaussian noise, inflating the manifold slightly
• After many steps, the structure is completely destroyed
• End: an isotropic Gaussian cloud — no trace of the original data

THE FORWARD PROCESS: SMEARING THE MANIFOLD

WHY WOULD THIS EVEN WORK?

Diffusion seems to come out of left field. Why would “add noise then learn to remove it” be a good generative model?

The idea comes from three independent intellectual threads that converged around 2020:

1. Physics — non-equilibrium thermodynamics.

A drop of ink in water spreads until it’s uniformly dissolved. This is thermodynamic diffusion — a structured state dissolving into maximum entropy equilibrium. The process is gradual, Markovian, and well-understood mathematically.

• Sohl-Dickstein et al. (2015) asked: if the forward process (structure \(\to\) noise) follows known physics, can we learn the reverse (noise \(\to\) structure)?

• This is why it’s called “diffusion.” It’s not a metaphor — the forward process is literally thermodynamic diffusion applied to data.

THREE THREADS CONVERGING

2. Statistics — score matching and Langevin dynamics.

The score function \(\nabla_\mathbf{x} \log p(\mathbf{x})\) is the gradient of the log-density at any point. Think of it as a vector field that points “uphill” toward higher-density regions of the data.

• If you know the score everywhere, you can generate samples by following the gradient uphill — this is Langevin dynamics, a random walk biased toward high-density regions.

• Song & Ermon (2019) showed: train a neural network to estimate the score at multiple noise levels.

The deep connection we’ll see later: the noise prediction \(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\) that diffusion learns IS the score function, up to a scaling factor. Predicting noise and estimating the gradient of the log-density are mathematically equivalent.

THREE THREADS CONVERGING

3. Representation learning — denoising autoencoders.

Vincent et al. (2008) showed that training a network to denoise corrupted inputs implicitly learns the data distribution. The optimal denoising function points toward the data manifold. You saw this in your own autoencoders early in the semester!

Diffusion extends this from one noise level to a continuum of noise levels — it’s a denoising autoencoder trained at every scale simultaneously.

THE FORWARD PROCESS: THE MATH

Define the forward process as a Markov chain — each step only depends on the previous step:

\[q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1 - \beta_t}\,\mathbf{x}_{t-1},\; \beta_t \mathbf{I})\]

• In plain English: take the image from the previous step, shrink it slightly (multiply by \(\sqrt{1 - \beta_t}\), which is just under 1), and add a small amount of random noise (scaled by \(\beta_t\), which is small).

THE FORWARD PROCESS: THE MATH

\[q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1 - \beta_t}\,\mathbf{x}_{t-1},\; \beta_t \mathbf{I})\]

\(\beta_t\) is the noise schedule — it controls how much noise we add at each step.

• Small early (gentle noising), larger later after a bunch of noise addition (more aggressive).

Typical values: \(\beta_1 = 0.0001\), \(\beta_T = 0.02\), linearly spaced. Tiny amounts of noise, applied 1000 times.

THE CLOSED FORM: WHY TRAINING IS PRACTICAL

The key trick: we don’t need to iterate \(T\) steps to get to timestep \(t\).

• Define: \(\alpha_t = 1 - \beta_t\) (the “signal retention” at step \(t\))

• Define: \(\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s\) (the cumulative signal survival rate — how much of the original image survives after \(t\) steps)

THE CLOSED FORM: WHY TRAINING IS PRACTICAL

In plain English: the noisy image at step \(t\) is a weighted mix of the clean image and pure noise.

• \(\sqrt{\bar{\alpha}_t}\) controls how much signal survives; \(\sqrt{1 - \bar{\alpha}_t}\) controls how much noise is added.

• The weights are designed so the total variance stays constant — we’re not making the image “louder,” we’re replacing signal with noise.

VISUALIZING THE NOISE SCHEDULE

THE REVERSE PROCESS: LEARNING TO DENOISE

The forward process is fixed and known. The reverse process is what we need to learn.

• Start at \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) (pure noise). Learn a model \(p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t)\) that removes one step of noise. Apply it \(T\) times to walk from pure noise back to clean data.

In plain English: given a noisy image, make it slightly less noisy. Do this 1000 times in a row, and you go from static to a photograph.

THE REVERSE PROCESS: LEARNING TO DENOISE

WHAT DOES THE NETWORK PREDICT?

Three equivalent framings:

1. Predict the clean image \(\mathbf{x}_0\) directly from \(\mathbf{x}_t\) — “show me the denoised result”
2. Predict the noise \(\boldsymbol{\epsilon}\) that was added to get from \(\mathbf{x}_0\) to \(\mathbf{x}_t\) — “show me the static”
3. Predict the score \(\nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t)\) — “show me the gradient toward the data”

All three are mathematically equivalent — you can derive any one from the others (because knowing any two of \(\mathbf{x}_0\), \(\mathbf{x}_t\), \(\boldsymbol{\epsilon}\) gives you the third). In practice, noise prediction (option 2) works best.

THE ELBO FOR DIFFUSION

Just like VAEs, we can derive a variational lower bound on \(\log p(\mathbf{x}_0)\). The ELBO decomposes into a sum of terms, one per timestep:

\[\log p(\mathbf{x}_0) \geq -\underbrace{L_T}_{\text{prior matching}} - \sum_{t=2}^{T} \underbrace{L_{t-1}}_{\text{denoising at scale } t} - \underbrace{L_0}_{\text{reconstruction}}\]

In words, this bound says: the quality of your generative model is determined by three things:

1. Does your noising process actually reach pure Gaussian noise by step \(T\)? (\(L_T\) — yes, by construction, so this term is basically zero.)

2. At each intermediate step, can your network correctly undo one step of noising? (\(L_{t-1}\) — this is where ALL the learning happens.)

3. Can you reconstruct the final clean image from the almost-clean version at \(t = 1\)? (\(L_0\) — yes, this is easy because \(t = 1\) is barely noisy.)

The hard work is entirely in the middle: learn to denoise at every scale.

THE PARALLEL TO VAES

\[\log p(\mathbf{x}_0) \geq -\underbrace{L_T}_{\text{prior matching}} - \sum_{t=2}^{T} \underbrace{L_{t-1}}_{\text{denoising at scale } t} - \underbrace{L_0}_{\text{reconstruction}}\]

This ELBO has the exact same structure as the VAE ELBO:

• VAE: one KL term (encoder vs prior) + one reconstruction term
• Diffusion: \(T\) KL terms (one per timestep) + one reconstruction term

The VAE asks the encoder to match \(\mathcal{N}(\mathbf{0}, \mathbf{I})\) in one giant step \(\to\) prior holes.

Diffusion distributes this work across \(T\) tiny steps \(\to\) each step is easy, no holes.

THE POSTERIOR IS TRACTABLE (GIVEN \(\mathbf{x}_0\))

A crucial fact: \(q(\mathbf{x}_{t-1} | \mathbf{x}_t, \mathbf{x}_0)\) — the true reverse step when you know the clean image — is a Gaussian with a known mean and variance.

Why?

• Because Bayes’ rule applied to two Gaussians gives a Gaussian. Both the forward step \(q(\mathbf{x}_t | \mathbf{x}_{t-1})\) and the jump \(q(\mathbf{x}_t | \mathbf{x}_0)\) are Gaussian, so the conditional reverse is too.

THE POSTERIOR IS TRACTABLE (GIVEN \(\mathbf{x}_0\))

The posterior mean:

\[\tilde{\mu}_t(\mathbf{x}_t, \mathbf{x}_0) = \frac{\sqrt{\bar{\alpha}_{t-1}}\,\beta_t}{1 - \bar{\alpha}_t}\,\mathbf{x}_0 + \frac{\sqrt{\alpha_t}(1 - \bar{\alpha}_{t-1})}{1 - \bar{\alpha}_t}\,\mathbf{x}_t\]

• the best guess for the slightly-less-noisy image \(\mathbf{x}_{t-1}\) is a weighted average of two things you know: the clean original \(\mathbf{x}_0\) and the current noisy version \(\mathbf{x}_t\).

• Early in the process (\(t\) small, image is mostly clean): lean heavily on \(\mathbf{x}_t\) — it’s already almost right.

• Late in the process (\(t\) large, image is mostly noise): lean more on \(\mathbf{x}_0\) — \(\mathbf{x}_t\) is garbage.

SUBSTITUTING THE REPARAMETERIZATION

Here’s the key move. We can’t use \(\mathbf{x}_0\) during generation because we don’t have it. But we know the relationship:

\[\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\,\boldsymbol{\epsilon}\]

SUBSTITUTING THE REPARAMETERIZATION

The model doesn’t know \(\boldsymbol{\epsilon}\) — so it predicts it. Replace \(\boldsymbol{\epsilon}\) with the network’s prediction \(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\):

\[\mu_\theta(\mathbf{x}_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}}\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right)\]

• This is why noise prediction is the natural target. It’s the one thing we don’t know, and knowing it lets us compute everything else — the reverse mean, the estimated clean image, the denoising step. One prediction unlocks the whole chain.

THE SIMPLIFICATION

After pages of algebra — grinding through per-timestep KL divergences between two Gaussians, substituting the reparameterized posterior, canceling schedule coefficients — the complex ELBO simplifies to something almost embarrassingly simple:

\[L_{\text{simple}} = \mathbb{E}_{t,\, \mathbf{x}_0,\, \boldsymbol{\epsilon}}\left[\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta\!\left(\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\,\boldsymbol{\epsilon},\; t\right)\right\|^2\right]\]

THE SIMPLIFICATION

1. Sample a clean image \(\mathbf{x}_0\) from the dataset
2. Sample a random timestep \(t \sim \text{Uniform}(1, T)\)
3. Sample random noise \(\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
4. Create the noisy image: \(\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\,\boldsymbol{\epsilon}\)
5. Ask the network to predict \(\boldsymbol{\epsilon}\) from \(\mathbf{x}_t\) and \(t\)
6. Loss = MSE between the true noise and the predicted noise

That’s it. The training recipe in one sentence: pick a random image, add random noise at a random scale, and train the network to predict what noise you added.

No adversarial training. No carefully balanced loss terms. No mode collapse to worry about. Just MSE on noise prediction. The simplicity is shocking given how good the results are.

THE TRAINING ALGORITHM

repeat:
    x_0 ~ dataset                                    # sample a clean image
    t ~ Uniform(1, T)                                # pick a random timestep
    ε ~ N(0, I)                                      # sample random noise
    x_t = sqrt(ᾱ_t) * x_0 + sqrt(1 - ᾱ_t) * ε      # create noisy image
    loss = || ε - ε_θ(x_t, t) ||²                    # predict the noise
    gradient step on loss
until converged

Compare this to the other training loops we’ve built this semester:

• VAE: reconstruction + KL balancing (finicky \(\beta\) tuning, posterior collapse)
• GAN: alternating generator/discriminator updates (mode collapse, training instability)
• GPT: next-token cross-entropy (stable but requires careful tokenization)

THE COMPASS METAPHOR

The network \(\boldsymbol{\epsilon}_\theta\) acts as a compass pointing back toward the clean data manifold.

• At any point in the noisy space, the predicted noise \(\hat{\boldsymbol{\epsilon}}\) tells you: “the clean data is in THIS direction, THIS far away.”

• Subtracting the predicted noise from the noisy image takes one step back toward the manifold.

THE COMPASS METAPHOR

WHY THIS SOLVES THE PRIOR HOLE

Compare:

VAE: learns the data distribution only at the surface of the manifold. The prior is assumed (\(\mathcal{N}(\mathbf{0}, \mathbf{I})\)), not learned. Holes in the prior \(\to\) bad samples.

Diffusion: learns the data distribution at EVERY noise scale. The model must correctly denoise from any level of corruption. This forces it to understand the density of the space everywhere.

WHY THIS SOLVES THE PRIOR HOLE

By training the model to remove noise at every scale, we explicitly force it to learn the density of the space everywhere. There are no more holes. Sampling works because the model has seen — and learned to navigate from — every part of the noisy space.

• This is denoising as explicit regularization.

• The “prior hole” problem is solved not by assuming a better prior, but by learning the data distribution so thoroughly that no holes can exist.

The forward process guarantees we reach \(\mathcal{N}(\mathbf{0}, \mathbf{I})\) at \(t = T\). The reverse process learns to undo every step. The result: a fully learned, nonparametric path from noise to data, with no assumptions about the shape of the data distribution.

THE SAMPLING ALGORITHM

Start with \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) — pure noise. Then iterate backward:

For \(t = T, T-1, \ldots, 1\):

\[\mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}}\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right) + \sigma_t \mathbf{z}\]

where \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) and \(\sigma_t = \sqrt{\beta_t}\) for DDPM.

• At \(t = 1\), omit the noise (\(\mathbf{z} = \mathbf{0}\)).

• a little randomness helps explore the space and avoid getting stuck

• After \(T\) steps, \(\mathbf{x}_0\) is a clean sample from the learned distribution.

DDIM: DETERMINISTIC SAMPLING

The DDPM sampling algorithm has a quirk: the fresh noise \(\sigma_t \mathbf{z}\) added at each step makes generation stochastic. The same starting noise \(\mathbf{x}_T\) produces a different image every time.

Song et al. (2021) noticed something remarkable: you can derive a family of reverse processes that all correspond to the same forward process and the same trained model. DDPM is just one member of this family — the fully stochastic one.

At the other extreme is DDIM (Denoising Diffusion Implicit Models): a fully deterministic sampler.

DDIM: DETERMINISTIC SAMPLING

The DDIM update rule:

\[\mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\underbrace{\left(\frac{\mathbf{x}_t - \sqrt{1 - \bar{\alpha}_t}\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)}{\sqrt{\bar{\alpha}_t}}\right)}_{\text{predicted } \mathbf{x}_0} + \sqrt{1 - \bar{\alpha}_{t-1}}\;\underbrace{\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)}_{\text{predicted noise direction}}\]

• Instead of “remove some noise and add fresh randomness,” DDIM says “estimate the clean image, then re-noise it to the level appropriate for step \(t-1\).” No fresh randomness at all.

• The predicted \(\mathbf{x}_0\) term is just the reparameterization we derived earlier — use the network’s noise prediction to estimate what the clean image looks like. Then blend that estimate with the predicted noise direction to land at the right noise level for the next step.

WHY DDIM MATTERS

DDIM gives us two things DDPM doesn’t:

1. Deterministic generation. Same starting noise \(\to\) same output image, every time. This means each point in the Gaussian noise space maps to a specific image. The noise becomes a “seed” — a coordinate system for the image manifold.

• This enables latent space arithmetic: interpolate between two noise vectors and you get a smooth interpolation between the corresponding images. DDPM can’t do this because the stochastic noise at each step breaks the correspondence.

WHY DDIM MATTERS

DDIM gives us two things DDPM doesn’t:

2. Fewer steps without retraining. DDPM needs all \(T\) steps — skip steps and quality degrades badly because the stochastic noise compounds errors. DDIM’s deterministic updates are much more tolerant of skipped steps.

• With DDIM, you can subsample the timesteps — use every 10th step, or every 20th — and still get good results. 50 steps instead of 1000, using the exact same trained model.

• DDPM is a wandering hike — random detours at every step, must take all 1000 steps. DDIM is a straight-line walk — no detours, can take shortcuts.

WHY WE CAN’T TAKE ONE BIG STEP

Why 1000 tiny steps instead of 1 big step?

• The noise prediction \(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\) is a local approximation. It estimates the direction to the manifold from the current noisy position.

• Taking one giant step from \(t = 1000\) to \(t = 0\) would be like following a compass bearing for 1000 miles without rechecking — the terrain curves, and you’d end up far from your destination.

• Each small step recalculates the local direction. After 1000 recalculations, you arrive at the manifold precisely.

THE DIMENSIONALITY CONSTRAINT

The network takes in a noisy image \(\mathbf{x}_t\) and outputs predicted noise \(\hat{\boldsymbol{\epsilon}}\). These must have exactly the same dimensions.

• Input: noisy image of shape \((C, H, W)\)
• Output: noise map of shape \((C, H, W)\)

This rules out standard architectures:

• Encoders (ResNet, ViT): reduce spatial resolution. A 224×224 input becomes a 7×7 feature map. Can’t produce a 224×224 noise prediction.
• Classifiers: output a single label, not a spatial map.
• Standard autoencoders: bottleneck loses spatial detail.

THE DIMENSIONALITY CONSTRAINT

We need an architecture that preserves spatial resolution exactly while processing global context.

• Picture in, picture out. Map an image to an image of the same size, where each pixel’s value is informed by the entire image.

• Guesses? We’ve seen it before.

THE U-NET

You’ve seen U-Nets before (segmentation). Same architecture, different purpose.

• Encoder path (downsampling): captures what’s in the image at multiple scales. “What semantic content is here?” - starts at more noise
• Decoder path (upsampling): restores spatial resolution. “Where exactly should each detail go?” - returns less noise!
• Skip connections: bypass the bottleneck to preserve fine spatial details. The encoder says “there are sharp edges at these coordinates” and the skip connection delivers that directly to the decoder.
• Timestep embedding: the network needs to know WHICH noise level it’s denoising from. Encode \(t\) as a sinusoidal embedding (same idea as positional encoding in transformers) and inject it into every layer.

THE U-NET

THE U-NET IN PYTORCH (CONCEPTUAL)

class DiffusionUNet(nn.Module):
    def __init__(self):
        # Downsampling blocks
        self.down1 = DownBlock(3, 64)     # (3, H, W) → (64, H/2, W/2)
        self.down2 = DownBlock(64, 128)   # → (128, H/4, W/4)
        self.down3 = DownBlock(128, 256)  # → (256, H/8, W/8)

        # Bottleneck
        self.mid = MidBlock(256)

        # Upsampling blocks (with skip connections)
        self.up3 = UpBlock(256 + 256, 128)  # concat skip from down3
        self.up2 = UpBlock(128 + 128, 64)   # concat skip from down2
        self.up1 = UpBlock(64 + 64, 3)      # concat skip from down1

        # Timestep embedding
        self.time_mlp = nn.Sequential(
            SinusoidalPosEmbed(256),
            nn.Linear(256, 256), nn.GELU(),
        )

    def forward(self, x_t, t):
        t_emb = self.time_mlp(t)  # timestep injected at every layer
        # ... down, mid, up with skip connections and t_emb
        return predicted_noise  # same shape as x_t

COMPARISON TO EVERYTHING ELSE

How does diffusion compare to every generative approach we’ve studied this semester?

COMPARISON TO EVERYTHING ELSE

So, what’s the weakness?

THE CLIFFHANGER

We have the perfect mathematical framework to step from pure noise to a clean image.