DATASCI 447 Lecture 27: Escaping the Prior Hole — The Diffusion Paradigm

Administrative Stuff

THE STORY SO FAR

We’ve built a complete text-to-image pipeline:

• VQ-VAE: compress images into discrete codebook tokens (L19-20)
• GPT: autoregressive generation of token sequences (L24)
• CLIP: text embeddings as conditioning signal (L26)

THE VQ-VAE CEILING AND THE CONTINUOUS ALTERNATIVE

Discrete codebooks bottleneck generation in two ways:

1. Finite vocabulary for an infinite visual world. A codebook with 512 entries divides the continuous visual manifold into 512 bins. Fine details between bins are lost.

2. No interpolation between concepts. In a discrete space, there’s no “halfway between code 237 and code 238.” The space between codebook entries is empty.

Continuous VAEs solve this — they map images to a dense, low-dimensional manifold. No discretization, no codebook limits. With perceptual and adversarial losses (L19), reconstructions can be sharp and detailed.

• An infinite conceptual canvas. Smooth interpolation works naturally.

THE VQ-VAE CEILING AND THE CONTINUOUS ALTERNATIVE

THE KL TUG-OF-WAR

The fatal flaw of continuous VAEs: the KL penalty.

The VAE loss has two terms fighting each other:

\[\mathcal{L}_{\text{VAE}} = \underbrace{\text{Reconstruction}}_{\text{wants complex, detailed latents}} + \beta \cdot \underbrace{D_{KL}(q(\mathbf{z}|\mathbf{x}) \| \mathcal{N}(\mathbf{0}, \mathbf{I}))}_{\text{wants simple Gaussian latents}}\]

• Reconstruction wants the encoder to preserve every detail — spread the latents out, use all dimensions, be expressive.

• The KL term wants the encoder to collapse everything onto a unit Gaussian — simple, smooth, undifferentiated.

These goals are fundamentally in conflict.

• The encoder can’t simultaneously be expressive AND match a simple Gaussian. It compromises — and the compromise is ugly.

THE PRIOR HOLE

The true posterior \(q(\mathbf{z}|\mathbf{x})\) looks like Swiss cheese — dense regions where training data maps to, surrounded by empty gaps where nothing lives.

• During training, the encoder produces latents that cluster in specific regions. The KL penalty pushes them toward a Gaussian, but not hard enough to fill the entire space.

THE PRIOR HOLE

THE QUESTION

We want:

• Continuous latent spaces (no discretization bottleneck)
• Sharp, detailed reconstructions (perceptual + adversarial losses)
• A prior we can actually sample from (no holes)

The KL penalty was our tool for making the prior matchable. But it fights reconstruction quality.

THE PARADIGM SHIFT

Diffusion doesn’t assume a shape for the prior. It learns it.

THE CORE IDEA

Two processes:

Forward process (fixed, no learning): gradually add noise to data until it becomes pure Gaussian noise. This is the “walk to the Gaussian.”

Reverse process (learned): gradually remove noise from Gaussian noise until it becomes data. This is “retracing your steps.”

THE FORWARD PROCESS: SMEARING THE MANIFOLD

Visualize what the forward process does to the data distribution:

• Start: data lives on a concentrated, low-dimensional manifold
• Each step adds a tiny bit of Gaussian noise, inflating the manifold slightly
• After many steps, the structure is completely destroyed
• End: an isotropic Gaussian cloud — no trace of the original data

THE FORWARD PROCESS: SMEARING THE MANIFOLD

WHY WOULD THIS EVEN WORK?

Diffusion seems to come out of left field. Why would “add noise then learn to remove it” be a good generative model?

The idea comes from three independent intellectual threads that converged around 2020:

1. Physics — non-equilibrium thermodynamics.

A drop of ink in water spreads until it’s uniformly dissolved. This is thermodynamic diffusion — a structured state dissolving into maximum entropy equilibrium. The process is gradual, Markovian, and well-understood mathematically.

• Sohl-Dickstein et al. (2015) asked: if the forward process (structure \(\to\) noise) follows known physics, can we learn the reverse (noise \(\to\) structure)?

• This is why it’s called “diffusion.” It’s not a metaphor — the forward process is literally thermodynamic diffusion applied to data.

THREE THREADS CONVERGING

2. Statistics — score matching and Langevin dynamics.

The score function \(\nabla_\mathbf{x} \log p(\mathbf{x})\) is the gradient of the log-density at any point. Think of it as a vector field that points “uphill” toward higher-density regions of the data.

• If you know the score everywhere, you can generate samples by following the gradient uphill — this is Langevin dynamics, a random walk biased toward high-density regions.

• Song & Ermon (2019) showed: train a neural network to estimate the score at multiple noise levels.

The deep connection we’ll see later: the noise prediction \(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\) that diffusion learns IS the score function, up to a scaling factor. Predicting noise and estimating the gradient of the log-density are mathematically equivalent.

THREE THREADS CONVERGING

3. Representation learning — denoising autoencoders.

Vincent et al. (2008) showed that training a network to denoise corrupted inputs implicitly learns the data distribution. The optimal denoising function points toward the data manifold. You saw this in your own autoencoders early in the semester!

Diffusion extends this from one noise level to a continuum of noise levels — it’s a denoising autoencoder trained at every scale simultaneously.

THE FORWARD PROCESS: THE MATH

Define the forward process as a Markov chain — each step only depends on the previous step:

\[q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1 - \beta_t}\,\mathbf{x}_{t-1},\; \beta_t \mathbf{I})\]

• In plain English: take the image from the previous step, shrink it slightly (multiply by \(\sqrt{1 - \beta_t}\), which is just under 1), and add a small amount of random noise (scaled by \(\beta_t\), which is small).

THE FORWARD PROCESS: THE MATH

\[q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1 - \beta_t}\,\mathbf{x}_{t-1},\; \beta_t \mathbf{I})\]

\(\beta_t\) is the noise schedule — it controls how much noise we add at each step.

• Small early (gentle noising), larger later after a bunch of noise addition (more aggressive).

Typical values: \(\beta_1 = 0.0001\), \(\beta_T = 0.02\), linearly spaced. Tiny amounts of noise, applied 1000 times.

THE CLOSED FORM: WHY TRAINING IS PRACTICAL

The key trick: we don’t need to iterate \(T\) steps to get to timestep \(t\).

• Define: \(\alpha_t = 1 - \beta_t\) (the “signal retention” at step \(t\))

• Define: \(\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s\) (the cumulative signal survival rate — how much of the original image survives after \(t\) steps)

THE CLOSED FORM: WHY TRAINING IS PRACTICAL

In plain English: the noisy image at step \(t\) is a weighted mix of the clean image and pure noise.

• \(\sqrt{\bar{\alpha}_t}\) controls how much signal survives; \(\sqrt{1 - \bar{\alpha}_t}\) controls how much noise is added.

• The weights are designed so the total variance stays constant — we’re not making the image “louder,” we’re replacing signal with noise.

VISUALIZING THE NOISE SCHEDULE

THE REVERSE PROCESS: LEARNING TO DENOISE

The forward process is fixed and known. The reverse process is what we need to learn.

• Start at \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) (pure noise). Learn a model \(p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t)\) that removes one step of noise. Apply it \(T\) times to walk from pure noise back to clean data.

In plain English: given a noisy image, make it slightly less noisy. Do this 1000 times in a row, and you go from static to a photograph.

THE REVERSE PROCESS: LEARNING TO DENOISE