March 19, 2026
The VAE asks us to do a lot of things simultaneously:
The first four are the primary objective. Generation from random draws is somewhat secondary — it works because KL regularization creates a nice latent space, but it’s not what the loss function directly optimizes.
This is why reconstructions look decent but generations from \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) are blurry and samey.
What if we only cared about generation?
Start with an easy-to-sample distribution: \(\mathbf{z} \sim P(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})\) in \(\mathbb{R}^{256}\)
Learn a mapping \(g_\theta(\mathbf{z})\) such that when \(\mathbf{z} \sim P(\mathbf{z})\), the output \(g_\theta(\mathbf{z})\) is a coherent image of a dog.
This is just a decoder. A neural network that takes a random vector and outputs an image.
The idea tracks. But how do we judge quality?
Since \(P(\mathbf{z})\) is fixed (\(\mathcal{N}(\mathbf{0}, \mathbf{I})\), chosen by us), we need to find \(\theta\) such that \(g_\theta\) maps \(P(\mathbf{z})\) to \(P(\mathbf{x})\) — the distribution of all dog images in pixel space.
\(P(\mathbf{x})\) is a tiny, thin manifold in \(\mathbb{R}^{H \times W \times C}\).
Almost all of pixel space is garbage. The set of “images that look like dogs” is vanishingly small.
But it’s there. And we have samples from it — our training data.
If we just train \(g_\theta(\mathbf{z})\) to return the images in our training data, we get a deterministic autoencoder again.
The training set is not a fully characterized empirical distribution, even with billions of images — the curse of dimensionality means we’re always seeing a sparse sample of the true manifold.
We need to use \(\mathbf{X}\) to learn about \(P(\mathbf{x})\) without ever computing \(P(\mathbf{x})\) directly.
How?
Assume we have a CNN decoder parameterized by \(\theta\) that maps \(P(\mathbf{z})\) to the input space.
Goal: Make \(P(\mathbf{x})\) and \(Q(\mathbf{x})\) equivalent.
Formal equivalence:
\[\frac{P(\mathbf{x})}{Q(\mathbf{x})} = 1 \quad \forall \quad \mathbf{x}\]
The density ratio equals 1 everywhere.
Take the log:
\[\log \frac{P(\mathbf{x})}{Q(\mathbf{x})} = 0 \quad \iff \quad P = Q\]
The further from 0 the log density ratio is, the further the approximation is from the truth.
(KL divergence alarm bells should be ringing — we’ll come back to this.)
If I knew \(P(\mathbf{x})\), I could find \(\theta\) that minimizes the log density ratio everywhere.
I know \(Q(\mathbf{x})\) — it’s defined by my generator and the known prior \(P(\mathbf{z})\). \(\theta\) dictates its structure. Pick \(\theta\) to minimize the the expected log density ratio.
Note: There is a way to do this directly with distributions due to the Jacobian change of variable formula. We just won’t need to deal with this!
But, I don’t know \(P(\mathbf{x})\).
How can I assess the density ratio using only samples?
I have \(N\) samples from \(P(\mathbf{x})\) — the training images. Real.
Since I know \(P(\mathbf{z})\), I can take \(N\) samples from \(P(\mathbf{z})\) and transform them through \(g_\theta\) to get \(N\) samples from \(Q(\mathbf{x})\). Fake.
Our full collection: \(\{\mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_{2N}\}\)
Each image has \(H \times W \times C\) pixels.
Each image has a label:
\[y_i = \begin{cases} 1 & \text{real (generated by the world)} \\ 0 & \text{fake (generated by us)} \end{cases}\]