DATASCI 447 Lecture 15: Generating New Images (a.k.a. it was all needed…)

Administrative Stuff

WHERE WE ARE

Last class - Detection and Segmentation

• What is the main idea behind single-shot object detection (e.g. YOLO)? How do we reuse the CNN backbone?

• What is the main idea behind semantic segmentation? How do we reuse the CNN backbone?

• How do we create learnable upsampling for images?

WHERE WE ARE

WHERE WE ARE

Compress the image via a CNN to learn what is in the image

Blow the image back up to learn where those features are in the image

WHERE WE ARE

All semester: discriminative models.

We modeled \(P(\mathbf y | \mathbf x)\) — given an image, predict a label.

• Logistic regression → neural networks → CNNs → ResNets → U-Nets

We got very good at this. Transfer learning, fine-tuning, LoRA, segmentation.

THE OTHER FORK

Remember Lecture 1 — the Nature Box.

The joint distribution \(P(\mathbf y , \mathbf x)\) generates all data. We immediately took the discriminative path: model \(P(\mathbf y| \mathbf x)\), throw away \(P(\mathbf x)\).

THE FROG PROBLEM

We trained logistic regression to distinguish frogs from planes. 88% accuracy.

Then we asked: “what image maximizes \(P(y = \text{ frog } | \mathbf x)\)?”

WHAT A GENERATIVE MODEL NEEDS

A generative model is going to be a model that can generate new data points from the learned distribution.

• The goal is to learn some representation of the input data that allows us to create new and coherent draws from the data distribution

Good frog vs. bad frog

WHAT A GENERATIVE MODEL NEEDS

WHAT A GENERATIVE MODEL NEEDS

WHAT A GENERATIVE MODEL NEEDS

Assume we have a collection of dog images. (Yes - we’re switching gears here because I already have a good corpus of high res dog images.)

Three requirements:

Density estimation — assign \(P(\mathbf x)\) to any input image.

• Given a learned distribution over inputs, I should be able to take any combination of pixels and compute the probability that it was drawn from the learned distribution.

• If learn about dogs, but give it a cat image, it should say “this is unlikely to be a dog.”

WHAT A GENERATIVE MODEL NEEDS

Assume we have a collection of dog images. (Yes - we’re switching gears here because I already have a good corpus of high res dog images.)

Three requirements:

Sampling — draw new \(\mathbf x \sim P(\mathbf x)\) that we’ve never seen before

• Reach into the learned distribution and sample a new dog

• Each draw should be a plausible dog image that fits the learned distribution.

• We should create a cat or random mess with very low probability

WHAT A GENERATIVE MODEL NEEDS

Assume we have a collection of dog images. (Yes - we’re switching gears here because I already have a good corpus of high res dog images.)

Three requirements:

Representation — learn a compact, structured description of what varies across images

• What does a dog look like? Can we represent ear differences and control generation for different kinds of ears?

• To what extent can we manipulate the representation to generate specific features? An effectively learned representation should allow me to interpolate between different kinds of dogs and generate new variations in a controllable way.

WHAT A GENERATIVE MODEL NEEDS

We have tools for requirement (3).

• How so?

The question: can those tools be extended to handle (1) and (2)?

PCA AS A LINEAR ENCODER-DECODER

You’ve already seen PCA (Lecture 6). Let’s reframe it as an architecture.

\[\text{Encoder: } \mathbf{z} = W^T \mathbf{x} \qquad \text{Decoder: } \hat{\mathbf{x}} = W \mathbf{z}\]

• Encoder: project \(\mathbf x\) onto the top k principal components → \(\mathbf z \in \mathbb{R}^k\)

• Bottleneck: \(\mathbf z\) has k ≪ P dimensions (the compressed code)

• Decoder: reconstruct \(\hat{\mathbf{x}}\) from \(\mathbf z\) (project back to the original space)

• Loss: find \(\mathbf W\) that minimizes \(\| \mathbf x - \mathbf W \mathbf W^T \mathbf x \|\)

PCA finds the linear encoder-decoder pair that minimizes reconstruction error.

THE LINEAR AUTOENCODER

THE LINEAR AUTOENCODER

Represented as a forward pass in a Pytorch model:

class LinearAutoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim):
        super(LinearAutoencoder, self).__init__()
        self.encoder = nn.Linear(input_dim, latent_dim, bias=False)  # W^T
        self.decoder = nn.Linear(latent_dim, input_dim, bias=False)  # W

    def forward(self, x):
        z = self.encoder(x)       # Encode: z = W^T x
        x_hat = self.decoder(z)   # Decode: x̂ = W z
        return x_hat

THE LINEAR AUTOENCODER

Key result (Baldi & Hornik, 1989): A linear autoencoder trained with MSE learns the same subspace as PCA. The weights converge to span the principal component subspace.

PCA is a linear autoencoder. Same algorithm, different notation.

THE LINEARITY TRAP

PCA / linear autoencoders can only capture linear relationships between features.

• Decomposition of pairwise covariance implies linearity (QTM 110)

Remember L1: the data manifold is a low-dimensional sheet curved through high-dimensional space. PCA can only find flat subspaces.

• Random pixels \(\neq\) dog.

• In fact, a very narrow highly correlated slice of pixel space is dog.

GOING NONLINEAR: THE DEEP AUTOENCODER

The fix: add nonlinear activations.

GOING NONLINEAR: THE DEEP AUTOENCODER

For an image:

Encoder:  x → [Conv → ReLU → Pool → Conv → ReLU → Pool → ...] → z
Decoder:  z → [ConvTranspose → ReLU → ConvTranspose → ReLU → ...] → x̂
Loss:     ‖x - x̂‖²

• The encoder learns curved manifold coordinates

• The decoder reconstructs from them

• Same encoder-decoder structure as U-Net — just without skip connections

We’ve extended PCA from flat subspaces to curved manifolds.

GOING NONLINEAR: THE DEEP AUTOENCODER

The U-Net bottleneck was a tensor — say, 512 × 4 × 4. It preserved spatial structure because segmentation needs to know where things are.

For generation, we want something different.

GOING NONLINEAR: THE DEEP AUTOENCODER

We flatten to a vector \(\mathbf z \in \mathbb R^K\). Why?

• Sampling: we need to draw \(\mathbf z\). It’s easy to define distributions over vectors (like \(N(0, I)\)). Defining meaningful distributions over spatially-structured tensors is much harder.

• Structure: a vector lets us enforce that each dimension captures an independent factor of variation — one dimension for ear shape, another for fur color. A tensor bakes in spatial correlations we don’t necessarily want in the latent space.

Destroy spatial structure on purpose.

• It forces the decoder to reconstruct space from a compact code — exactly the representation type we need for generative models!

WHAT DEEP AUTOENCODERS ACHIEVE

Deep autoencoders can be excellent at three things:

• Compression: represent 224×224×3 images (150k pixels) in ~1,024 dimensions

• Reconstruction: recover the original from the compressed code — often near pixel-perfect

• Representation: similar images get similar codes; the latent space is organized

THE EXPERIMENT

We trained a deep autoencoder on dogs. It reconstructs beautifully.

Now: sample a random \(\mathbf z\) — say, \(\mathbf z \sim N(0, I)\) — and decode it.

THE DISTRIBUTIONAL DIAGNOSIS

A deterministic encoder f: x → z maps each training image to a single point in latent space.

In distributional terms, this defines a Dirac delta posterior for each observation:

\[Q(\mathbf{z} | \mathbf{x}) = \delta(\mathbf{z} - f(\mathbf{x}))\]

• Why is this a problem? Think back to Lecture 2 on likelihood and implicit distributional assumptions.

THE DISTRIBUTIONAL DIAGNOSIS

A Dirac delta places all of its mass on exactly one point and zero mass everywhere else.

The encoder has said: “image x could only have come from z = f(x). Every other z-value is impossible.”

• Mass collapse, baby!

Our deep autoencoder is sooooo expressive that it can (and does) learn to perfectly reconstruct the training data!

• But, that’s not all we want…

WHY THIS KILLS GENERATION

The decoder \(f_{\theta}(\mathbf z)\) was only ever trained to produce outputs at the specific points \(\{f_{\theta}(\mathbf x_1), f_{\theta}(\mathbf x_2), ..., f_{\theta}(\mathbf x_n)\}\).

• For any \(\mathbf z\) that is not one of those points, the decoder has received zero training signal.

• Its behavior is undefined — it’s extrapolating into the void.

• OVERFITTING

WHY THIS KILLS GENERATION

Without constraint, we also allow the model to learn to circumvent the latent space all together!

• The model finds shortcuts around the latent space that bypass the need for meaningful latent codes.

• If the goal is to reconstruct 10k images as well as possible, why not just learn the identity function \(f_{\theta}(\mathbf{x}) = \mathbf{x}\)?

We can see this occurring in our example because the nearest neighbors don’t make a ton of sense.

• We’ve destroyed meaningful structure in the latent space.

THE PROBLEM IS MASS COLLAPSE (AGAIN)

We learned \(f(\mathbf{x}) \rightarrow \mathbf{z}\) with no constraint on what the latent space looks like.

Result: mass collapse. Isolated points. Voids everywhere.

HOW DO YOU REGULARIZE A LATENT SPACE?

In Ridge regression, we regularized \(\boldsymbol \beta\) by putting a prior on it (Lecture 3):

\[P(\boldsymbol{\beta}) = \mathcal{N}(\mathbf{0}, \lambda^{-1} I)\]

The prior penalized complex coefficients and pulled them toward zero. It prevented mass collapse by spreading probability away from the MLE.

CHOOSING THE PRIOR ON z

Key insight: z is latent — it can have whatever structure we want.

Unlike \(\boldsymbol \beta\) in regression (which lives in a space defined by the features), the latent space has no pre-existing meaning. We get to design it.

CHOOSING THE PRIOR ON z

CHOOSING THE PRIOR ON z

What do we want our latent space to look like?

• Similarities preserved — nearby images should map to nearby codes

• Independent dimensions — each latent dimension captures a different factor of variation

• Compactness — minimize uncovered space, keep everything tidy

CHOOSING THE PRIOR ON z

A prior that encodes all three properties:

\[P(\mathbf{z}) = \mathcal{N}(\mathbf{0}, I)\]

• Zero off-diagonals → dimensions are independent (no redundancy)

• Unit variance → each dimension uses the same scale (compact)

• Centered at origin → single, well-defined region of space

Simple. Elegant. And — as we’ll see — computationally convenient.

THE GENERATIVE STORY

Now write the full probabilistic model.

Step 1: Draw a latent code from the prior.

\[\mathbf{z} \sim P(\mathbf{z}) = \mathcal{N}(\mathbf{0}, I)\]

Step 2: Generate an observation via a nonlinear decoder.

\[P(\mathbf{x} | \mathbf{z}, \theta) = \mathcal{N}(\mathbf{x} \;|\; f_\theta(\mathbf{z}),\; \sigma^2 I)\]

where \(f_\theta\) is a neural network (the decoder) that maps z to image space.

• You’ve seen this before. Remember where?

THE GENERATIVE STORY

This is factor analysis (generative PCA) with a neural network instead of a linear map.

• Same structure as Bayesian regression from L3: prior on the hidden variable, likelihood conditioned on it. Here \(\mathbf z\) plays the role of \(\boldsymbol \beta\).

THE POSTERIOR ON z

Given an observed image x: what latent codes could have generated it?

Apply Bayes’ rule:

\[P(\mathbf{z} | \mathbf{x}, \theta) \propto P(\mathbf{x} | \mathbf{z}, \theta) \; P(\mathbf{z})\]

THE POSTERIOR ON z

Both the likelihood and the prior are Gaussian. If things were nice and conjugate, we would have a Gaussian posterior:

\[P(\mathbf{z} | \mathbf{x}, \theta) = \mathcal{N}(\mathbf{z} \;|\; \boldsymbol{\mu}_{\text{post}},\; \boldsymbol{\Sigma}_{\text{post}})\]

\[\boldsymbol{\Sigma}_{\text{post}}^{-1} = I + \frac{1}{\sigma^2} J_\theta^T J_\theta \qquad \boldsymbol{\mu}_{\text{post}} = \boldsymbol{\Sigma}_{\text{post}} \left( \frac{1}{\sigma^2} J_\theta^T (\mathbf{x} - f_\theta(\mathbf{z}_0)) \right)\]

• The posterior variance shrinks as we get more data. More iid data means more distributional coverage and less need to “regularize” the latent space.

• The posterior mean is a compromise between the prior (pulling toward 0) and the likelihood (pulling toward the z that best explains x).

• Note that \(J\) is the Jacobian! Change of variables trick.

• This is a method called the Laplace approximation and is the precursor to the variational method we’re getting to!

WHY THE POSTERIOR FIGHTS MASS COLLAPSE

This is the payoff.

Each observation x is no longer mapped to a single point — it’s associated with an entire distribution over z-values.

• The Gaussian posterior has full support on all of \(\mathbb R^k\) — every z has nonzero probability

• All posteriors share the same prior N(0, I) → they’re all concentrated in the same compact region

• The overlap between posteriors is substantial — the decoder sees training signal everywhere in this region

The prior is the regularizer. It prevents mass collapse by ensuring that each x spreads its probability across a region, and all those regions overlap.

WHY THE POSTERIOR FIGHTS MASS COLLAPSE

WHY THE POSTERIOR FIGHTS MASS COLLAPSE

WHY THE POSTERIOR FIGHTS MASS COLLAPSE

The Distributional Argument

Part 1 — The distributional fix:

Gaussian posteriors have full support on \(\mathbb R^K\). Every z has nonzero probability under every posterior. Holes are eliminated by construction — this is a mathematical guarantee of the family choice.

The Distributional Argument

Part 2 — The prior makes it practical:

The regularization against \(P(\mathbf{z}) = \mathcal{N}(0, \mathbf I)\) concentrates all posteriors in the same region, making their overlap substantial and the latent space navigable.

• We design the space to be smooth and continuous, so our prior structure induces a nice geometry.

The Distributional Argument

Part 3 — Mixtures of codes:

Any point in the latent space can be thought of as a mixture of the underlying codes and original images

• Just a coherent mixture since the latent code encodes the “dogness” of an image - more accurately, the correlation between the base features that are learned by our CNN backbone.

• Not a superimposition of features — the mixture is learned jointly since it is in the latent space that ensures that the relationships between features are preserved.

• Something like the halfway point between a golden retriever and a poodle is a golden doodle — not a dog with half a tail and half a snout

• Any code in \(\mathbb R^{K}\) will generate some version of a dog!

The Beginning of the Troubles

Unfortunately, our posterior on \(\mathbf z\) isn’t going to be multivariate Gaussian

• Likelihood: \(P(\mathbf{x} | \mathbf{z}, \theta) = \mathcal{N}(\mathbf{x}; f_\theta(\mathbf{z}), \sigma^2 \mathbf{I})\)

• Prior: \(P(\mathbf{z}) = \mathcal{N}(\mathbf{z}; 0, \mathbf{I})\)

Because the mean of our likelihood is a nonlinear function of \(\mathbf{z}\), the posterior on \(\mathbf z\) won’t add together nicely and lead to another multivariate normal.

The Beginning of the Troubles

So, given \(\mathbf X \in \mathbb R^{P \times P \times 3}\), we need to learn the following things:

1. \(f_{\theta}(\mathbf X)\): the decoder function that maps latent codes \(\mathbf z\) back to image space.

2. \(f_{\phi}(\mathbf X)\): the encoder function that maps images to latent codes.

Problem: WE DON’T KNOW Z!

• Too many unknowns.

THE PROBLEM: JOINTLY LEARNING z AND θ

Beautiful theory. But how do we actually learn this?

MARGINALIZE OVER z

By construction, \(P(\mathbf{z})\) is a proper probability distribution. So is our likelihood function \(P(\mathbf{x} | \mathbf{z}, \phi)\). So we can marginalize over z to get rid of it:

Find the marginal likelihood of the data:

\[P(\mathbf{x} | \phi) = \int_{\mathbf z} P(\mathbf{x} | \mathbf{z}, \phi) \; P(\mathbf{z}) \; d\mathbf{z}\]

Choose \(\phi\) that maximizes this value.

• Recognize this thing?

Quick Note: We’re going to assume that \(\theta\) is a direct function of \(\phi\) and \(\mathbf z\) - if I know two, then I know the third (or could learn it directly!)

MARGINALIZE OVER z

This integral.

WHY IT’S INTRACTABLE

\[P(\mathbf{x} | \phi) = \int P(\mathbf{x} | \mathbf{z}, \phi) \; P(\mathbf{z}) \; d\mathbf{z}\]

• \(P(\mathbf{x} | \mathbf{z}, \theta)\) is a neural network decoder — highly nonlinear

• z is high-dimensional (64 to 1,024 dimensions in practice)

• No closed form

WHY IT’S INTRACTABLE

• Grid methods: curse of dimensionality — the space is too empty (L1’s emptiness argument)

• Naive Monte Carlo: sample z₁, …, zₖ from P(z), average P(x | zᵢ, θ). But almost all random z’s produce images nothing like x → astronomically high variance

• Log makes it worse: \(\log \int \neq \int \log\) — can’t push the log inside. Integrals are continuous sums - \(\log a + b \neq \log a + \log b\).

CAN WE JUST USE BAYES’ RULE DIRECTLY?

Tempting thought. We wrote the posterior \(P(\mathbf{z} | \mathbf{x}, \phi) \propto P(\mathbf{x} | \mathbf{z}, \phi) P(\mathbf{z})\). Conjugacy gave us a nice Gaussian formula.

Can we just compute it?

THE KEY IDEA: APPROXIMATE THE POSTERIOR

We can’t compute the true posterior \(P(\mathbf{z} | \mathbf{x}, \phi)\).

So let’s approximate it.

THE KEY IDEA: APPROXIMATE THE POSTERIOR

We choose Q to be Gaussian:

\[Q_\phi(\mathbf{z} | \mathbf{x}) = \mathcal{N}(\mathbf{z} \;|\; \boldsymbol{\mu}_\phi(\mathbf{x}),\; \text{diag}(\boldsymbol{\sigma}^2_\phi(\mathbf{x})))\]

• Full support. Convenient.

• This is the encoder. The neural network that maps images to posterior parameters. Two heads that map \(\mathbf X\) to \(\boldsymbol{\mu}(\mathbf X)\) and \(\boldsymbol{\sigma}^2(\mathbf X)\).

• Defensible as \(N \to \infty\) due to Bayesian CLT - Bernstein-von Mises theorem.

DERIVING THE ELBO: STEP 1

Start from the log marginal likelihood — the thing we want to maximize but can’t compute:

\[\log P(\mathbf{x} | \phi) = \log P(\mathbf{x} | \phi) \cdot \underbrace{\int Q_\phi(\mathbf{z} | \mathbf{x}, \phi) \; d\mathbf{z}}_{= 1}\]

DERIVING THE ELBO: STEP 2

Now use Bayes’ rule: \(P(\mathbf{x} | \phi) = \frac{P(\mathbf{x} | \mathbf{z}, \theta) \; P(\mathbf{z})}{P(\mathbf{z} | \mathbf{x}, \phi)}\)

\[= \mathbb{E}_{Q}\left[\log \frac{P(\mathbf{x} | \mathbf{z}, \theta) \; P(\mathbf{z})}{P(\mathbf{z} | \mathbf{x}, \phi)}\right]\]

• Note that we define the decoder in terms of \(\theta\) - a different set of parameters - that have nothing to do with \(\mathbf z\) directly.

DERIVING THE ELBO: STEP 3

Multiply and divide by \(Q(\mathbf{z} | \mathbf{x}, \phi)\) inside the log:

\[\log P(\mathbf{x} | \phi) = \mathbb{E}_{Q}\left[\log \frac{P(\mathbf{x} | \mathbf{z}, \theta) \; P(\mathbf{z}) \; Q(\mathbf z | \mathbf x, \phi)}{P(\mathbf{z} | \mathbf{x}, \phi) \; Q(\mathbf z | \mathbf x, \phi)}\right]\]

Deconstruct into a particular form:

\[\log P(\mathbf{x} | \phi) = \mathbb{E}_{Q}\left[\log \frac{P(\mathbf{x} | \mathbf{z}, \theta) \; P(\mathbf{z})}{Q(\mathbf{z} | \mathbf{x}, \phi)}\right] + \mathbb{E}_{Q}\left[\log \frac{Q(\mathbf{z} | \mathbf{x}, \phi)}{P(\mathbf{z} | \mathbf{x}, \phi)}\right]\]

And deconstruct one more time using rules of logs and linearity of expectations:

\[\log P(\mathbf{x} | \phi) = \mathbb{E}_{Q}\left[\log P(\mathbf{x} | \mathbf{z}, \theta)\right] + \mathbb{E}_{Q}\left[\log \frac{P(\mathbf{z})}{Q(\mathbf{z} | \mathbf{x}, \phi)}\right] + \mathbb{E}_{Q}\left[\log \frac{Q(\mathbf{z} | \mathbf{x}, \phi)}{P(\mathbf{z} | \mathbf{x}, \phi)}\right]\]

DERIVING THE ELBO

\[ \mathbb{E}_{Q}\left[\log \frac{P(\mathbf{z})}{Q(\mathbf{z} | \mathbf{x}, \phi)}\right] \]

• Recognize this thing? Maybe it would be more familiar in the integral form?

\[ E_{Q}\left[ \log \frac{P(\mathbf z)}{Q(\mathbf{z} | \mathbf{x}, \phi)}\right] = \int Q(\mathbf{z} | \mathbf{x}, \phi) \; \log \frac{P(\mathbf z)}{Q(\mathbf{z} | \mathbf{x}, \phi)} \; d\mathbf{z} \]

DERIVING THE ELBO

\[ E_{Q}\left[ \log \frac{P(\mathbf z)}{Q(\mathbf{z} | \mathbf{x}, \phi)}\right] \]

This is the expected log ratio of the prior to the approximate posterior

DERIVING THE ELBO

THE THREE TERMS

\[\log P(\mathbf{x} | \phi) = \mathbb{E}_{Q}\left[\log P(\mathbf{x} | \mathbf{z}, \theta)\right] + \mathbb{E}_{Q}\left[\log \frac{P(\mathbf{z})}{Q(\mathbf{z} | \mathbf{x}, \phi)}\right] + \mathbb{E}_{Q}\left[\log \frac{Q(\mathbf{z} | \mathbf{x}, \phi)}{P(\mathbf{z} | \mathbf{x}, \phi)}\right]\]

\[\log P(\mathbf{x}) = \underbrace{\mathbb{E}_{Q}[\log P(\mathbf{x} | \mathbf{z})] - D_{KL}(Q(\mathbf{z}|\mathbf{x}) \| P(\mathbf{z}))}_{\text{ELBO: } \mathcal{L}(\theta, \phi; \mathbf{x})} + \underbrace{D_{KL}(Q(\mathbf{z}|\mathbf{x}) \| P(\mathbf{z}|\mathbf{x}))}_{\geq 0, \text{ unknowable}}\]

Term 1 — Reconstruction: \(\mathbb{E}_{Q}[\log P(\mathbf{x} | \mathbf{z})]\)

“How well does the decoder reconstruct x from z’s sampled according to Q?”

Term 2 — KL Regularizer: \(-D_{KL}(Q(\mathbf{z}|\mathbf{x}) \| P(\mathbf{z}))\)

“How close is the approximate posterior to the prior?” This is the term that enforces the shared geometry from Part 3.

Term 3 — Approximation Gap: \(D_{KL}(Q(\mathbf{z}|\mathbf{x}) \| P(\mathbf{z}|\mathbf{x}))\)

“How good is our approximation?” Always \(\ge\) 0 (Gibbs inequality). Can’t compute it — it requires the true posterior. Drop it.

THE ELBO

\[\log P(\mathbf{x}) = \underbrace{\mathbb{E}_{Q}[\log P(\mathbf{x} | \mathbf{z})] - D_{KL}(Q(\mathbf{z}|\mathbf{x}) \| P(\mathbf{z}))}_{\text{ELBO: } \mathcal{L}(\theta, \phi; \mathbf{x})} + \underbrace{D_{KL}(Q(\mathbf{z}|\mathbf{x}) \| P(\mathbf{z}|\mathbf{x}))}_{\geq 0, \text{ unknowable}}\]

Since the gap term \(\geq\) 0:

\[\text{ELBO} = \mathbb{E}_{Q}[\log P(\mathbf{x} | \mathbf{z})] - D_{KL}(Q(\mathbf{z}|\mathbf{x}) \| P(\mathbf{z})) \;\leq\; \log P(\mathbf{x})\]

THE ELBO IS RECONSTRUCTION VS. REGULARIZATION

\[\text{ELBO} = \underbrace{\mathbb{E}_{Q}[\log P(\mathbf{x} | \mathbf{z}, \theta)]}_{\text{Reconstruction}} - \underbrace{D_{KL}(Q(\mathbf{z}|\mathbf{x}, \phi) \| P(\mathbf{z}))}_{\text{Regularization}}\]

Look at this through the lens of every model we’ve built:

Reconstruction term → the encoder-decoder must still compress and reconstruct well. This is the autoencoder job. Push \(Q(\mathbf z | \mathbf x, \phi)\) to be informative about \(\mathbf x\).

KL term → the posteriors must stay close to \(\mathcal N(0, \mathbf I)\) - minimize the distance between the image posterior and the prior. Prevent mass collapse. Maintain shared geometry. Push \(Q(\mathbf z | \mathbf x , \phi)\) toward the prior.

A Quick History Lesson

Classical variational inference (pre-deep learning):

For each observation xᵢ, learn separate posterior parameters \((\mu_i, \sigma^2_i)\). N observations \(\to\) N separate optimization problems.

This was the standard approach for decades. No one did this…

A Quick History Lesson

Amortized inference:

Learn a single function \(Q_\phi(\mathbf{z} | \mathbf{x})\) — a neural network that takes any x and outputs \((\mu_\phi(\mathbf x), \sigma^2_\phi(\mathbf x))\).

One network replaces N optimizations.

WHY AMORTIZATION IS THE NATURAL APPROACH

Amortization seems obviously better. And it is.

• One neural network handles all observations — no per-observation optimization

• Generalizes to new x at test time — just run the encoder

• Scales to millions of observations without growing the parameter count

WHY AMORTIZATION IS THE NATURAL APPROACH

So why wasn’t this done from the start?

It required autodifferentiation frameworks like PyTorch to be practical. Classical VI was developed when computing gradients through a neural encoder was not feasible.

THE VAE IS AMORTIZED VI

Put it all together:

• Encoder network \(Q_\phi\): takes \(\mathbf x\), outputs \((\mu_\phi(\mathbf x), \sigma^2_\phi(\mathbf x))\). This is \(Q(\mathbf{z} | \mathbf{x}, \phi)\).

• Decoder network \(f_\theta\): takes \(\mathbf z \in \mathbb{R}^K\), outputs reconstructed image. This is \(P(\mathbf{x} | \mathbf{z}, \theta)\).

• Train both jointly by maximizing the ELBO (really minimizing the negative ELBO) with gradient descent.

THE LOGIC CHAIN

1. PCA is a linear encoder-decoder. Flat subspaces only.

2. Deep autoencoders extend PCA nonlinearly. Curved manifolds. Excellent reconstruction.

3. But deterministic encoders produce Dirac delta posteriors — zero measure between points. Generation fails.

4. Regularize the latent space with a prior — same logic as Ridge regression.

5. Gaussian posteriors have full support — holes eliminated by construction. The prior N(0, I) concentrates posteriors in a shared region.

6. Learning requires the marginal likelihood — the intractable integral from L4.

7. The ELBO gives us a computable lower bound. Reconstruction + KL regularizer.

8. Amortization parameterizes Q with a neural network encoder. That’s the VAE.

NEXT LECTURE

We turn the ELBO into a PyTorch training loop.

• The reparameterization trick — how to backprop through sampling

• The VAE architecture in code — encoder → (μ, log σ²) → sample → decoder

• Train it on dogs

• Sample from the prior and see coherent images — the payoff moment

• β-VAE and the reconstruction-regularization tradeoff — made visible