DATASCI 447 Lecture 18: Generative Adversarial Networks

Administrative Stuff

MAYBE VAEs ARE ASKING TOO MUCH

The VAE asks us to do a lot of things simultaneously:

• Map \(\mathbf{x} \to \mathbf{z}\) (encode)
• Map \(\mathbf{z} \to \hat{\mathbf{x}}\) (decode)
• Make \(Q(\mathbf{z}|\mathbf{x})\) close to \(\mathcal{N}(\mathbf{0}, \mathbf{I})\) (regularize)
• Make \(\hat{\mathbf{x}} \approx \mathbf{x}\) (reconstruct)
• Make random \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) decode to coherent images (generate)

A SIMPLER QUESTION

What if we only cared about generation?

A SIMPLER QUESTION

WHAT ARE WE TRYING TO LEARN?

Since \(P(\mathbf{z})\) is fixed (\(\mathcal{N}(\mathbf{0}, \mathbf{I})\), chosen by us), we need to find \(\theta\) such that \(g_\theta\) maps \(P(\mathbf{z})\) to \(P(\mathbf{x})\) — the distribution of all dog images in pixel space.

THE NAIVE APPROACH FAILS

If we just train \(g_\theta(\mathbf{z})\) to return the images in our training data, we get a deterministic autoencoder again.

• Each \(\mathbf{z}\) maps to a memorized image. No generation. No generalization.

THE SETUP

Assume we have a CNN decoder parameterized by \(\theta\) that maps \(P(\mathbf{z})\) to the input space.

• For notational ease: \(g_\theta(P(\mathbf{z})) = Q(\mathbf{x})\), our approximation to the true distribution.

THE LOG DENSITY RATIO

Take the log:

\[\log \frac{P(\mathbf{x})}{Q(\mathbf{x})} = 0 \quad \iff \quad P = Q\]

The further from 0 the log density ratio is, the further the approximation is from the truth.

• Logs are important because they make ratios symmetric about equality — a ratio of 2 and a ratio of 1/2 are equally bad, and \(\log 2 = -\log(1/2)\).

(KL divergence alarm bells should be ringing — we’ll come back to this.)

THE CORE PROBLEM

If I knew \(P(\mathbf{x})\), I could find \(\theta\) that minimizes the log density ratio everywhere.

• I know \(Q(\mathbf{x})\) — it’s defined by my generator and the known prior \(P(\mathbf{z})\). \(\theta\) dictates its structure. Pick \(\theta\) to minimize the the expected log density ratio.

• Note: There is a way to do this directly with distributions due to the Jacobian change of variable formula. We just won’t need to deal with this!

SAMPLES FROM BOTH DISTRIBUTIONS

I have \(N\) samples from \(P(\mathbf{x})\) — the training images. Real.

Since I know \(P(\mathbf{z})\), I can take \(N\) samples from \(P(\mathbf{z})\) and transform them through \(g_\theta\) to get \(N\) samples from \(Q(\mathbf{x})\). Fake.

SAMPLES FROM BOTH DISTRIBUTIONS

REWRITING THE DENSITY RATIO

Given our labeling scheme, we can rewrite the density ratio at any \(\mathbf{x}\):

\[\frac{P(\mathbf{x})}{Q(\mathbf{x})} = \frac{P(\mathbf{x} \,|\, y = 1)}{P(\mathbf{x} \,|\, y = 0)}\]

This is always true

• \(P(\mathbf{x})\) generates real images (\(y = 1\))

• \(Q(\mathbf{x})\) generates fake images (\(y = 0\)).

APPLYING BAYES’ RULE

Rewrite each conditional using Bayes’ rule:

\[\frac{P(\mathbf{x} \,|\, y=1)}{P(\mathbf{x} \,|\, y=0)} = \frac{P(y=1 \,|\, \mathbf{x}) \; P(\mathbf{x})}{P(y=1)} \cdot \frac{P(y=0)}{P(y=0 \,|\, \mathbf{x}) \; P(\mathbf{x})}\]

THE PRIOR DROPS OUT

Since we always draw \(N\) samples from \(P\) and \(N\) samples from \(Q\):

\[P(y = 0) = P(y = 1) = \frac{1}{2}\]

The prior ratio is 1. It drops out:

\[\frac{P(\mathbf{x})}{Q(\mathbf{x})} = \frac{P(y=1 \,|\, \mathbf{x})}{P(y=0 \,|\, \mathbf{x})}\]

THE PUNCHLINE

\[\frac{P(\mathbf{x})}{Q(\mathbf{x})} = \frac{P(y=1 \,|\, \mathbf{x})}{1 - P(y=1 \,|\, \mathbf{x})}\]

WHAT WE’VE SIDESTEPPED

Remember the VAE’s problem? We wanted \(P(\mathbf{x} \,|\, \theta) = \int P(\mathbf{x} \,|\, \mathbf{z}, \theta) P(\mathbf{z}) \, d\mathbf{z}\) and couldn’t compute the integral.

THE DISCRIMINATOR OBJECTIVE

If our goal is to find the classifier that best discriminates real from fake, we maximize the Bernoulli log-likelihood:

\[V(P, Q) = \max_\phi \; \mathbb{E}\left[y \log D_\phi(\mathbf{x}) + (1 - y) \log(1 - D_\phi(\mathbf{x}))\right]\]

where \(D_\phi(\mathbf{x}) = P(y = 1 \,|\, \mathbf{x})\) is a neural network classifier. \(\phi\) denotes the parameters for this classifier!

DISTRIBUTING THE EXPECTATION

The Bernoulli likelihood has two pieces — one active when \(y = 1\) (real) and one active when \(y = 0\) (fake).

\[V(P, Q) = \max_\phi \; \mathbb{E}\left[\log D_\phi(\mathbf{x}) \,|\, y = 1\right] + \mathbb{E}\left[\log(1 - D_\phi(\mathbf{x})) \,|\, y = 0\right]\]

FINDING THE OPTIMAL DISCRIMINATOR

Since \(P(\mathbf{x})\) and \(Q(\mathbf{x})\) have the same support, place both expectations under one integral:

\[V(P,Q) = \max_\phi \; \frac{1}{2} \int_{\mathbf{x}} P(\mathbf{x}) \log D_\phi(\mathbf{x}) + Q(\mathbf{x}) \log(1 - D_\phi(\mathbf{x})) \; d\mathbf{x}\]

THE INTEGRAND IS CONCAVE

For each \(\mathbf{x}\), we need to maximize:

\[P(\mathbf{x}) \log D(\mathbf{x}) + Q(\mathbf{x}) \log(1 - D(\mathbf{x}))\]

THE OPTIMAL CRITIC

\[D^*(\mathbf{x}) = \frac{P(\mathbf{x})}{P(\mathbf{x}) + Q(\mathbf{x})}\]

• \(P(y = 1 | \mathbf x) = \frac{P(\mathbf{x})}{P(\mathbf{x}) + Q(\mathbf{x})}\)

PLUGGING BACK IN

Since \(D^*(\mathbf{x}) = \frac{P}{P + Q}\), we also know \(1 - D^*(\mathbf{x}) = \frac{Q}{P + Q}\).

Plug both into the \(V(P, Q)\) formula:

\[V(P, Q) = \frac{1}{2}\mathbb{E}_{P(\mathbf{x})}\left[\log \frac{P(\mathbf{x})}{P(\mathbf{x}) + Q(\mathbf{x})}\right] + \frac{1}{2}\mathbb{E}_{Q(\mathbf{x})}\left[\log \frac{Q(\mathbf{x})}{P(\mathbf{x}) + Q(\mathbf{x})}\right]\]

• Note: \(\phi\) is gone. If we know the true distributions, the optimal discriminator and its loss are determined — no parameters to learn.

A FAMILIAR FRIEND

Multiply and divide inside each log by \(\frac{1}{2}\):

\[V(P, Q) = \frac{1}{2}\mathbb{E}_{P}\left[\log \frac{P(\mathbf{x})}{\frac{1}{2}(P(\mathbf{x}) + Q(\mathbf{x}))}\right] + \frac{1}{2}\mathbb{E}_{Q}\left[\log \frac{Q(\mathbf{x})}{\frac{1}{2}(P(\mathbf{x}) + Q(\mathbf{x}))}\right] - \log 2\]

Recognize these expectations?

A FAMILIAR FRIEND

A FAMILIAR FRIEND

A FAMILIAR FRIEND

A FAMILIAR FRIEND

THE JENSEN-SHANNON DIVERGENCE

\[V(P, Q) = \frac{1}{2}D_{KL}\left(P \;\Big\|\; \frac{P + Q}{2}\right) + \frac{1}{2}D_{KL}\left(Q \;\Big\|\; \frac{P + Q}{2}\right) - \log 2\]

This is the Jensen-Shannon divergence between \(P(\mathbf{x})\) and \(Q(\mathbf{x})\).

• Like KL, it measures how far two distributions are from one another.

• Unlike KL, it’s symmetric: \(\text{JSD}(P, Q) = \text{JSD}(Q, P)\).

• Just another distance to minimize

CONNECTING BACK TO THE DISCRIMINATOR

Let’s make this concrete.

We have two distributions — real images from \(P(\mathbf{x})\) and fake images from \(Q(\mathbf{x})\).

CONNECTING BACK TO THE DISCRIMINATOR

CONNECTING BACK TO THE DISCRIMINATOR

THE DISCRIMINATOR IS A JSD ESTIMATOR

So for a fixed generator \(g_\theta\):

1. Generate \(N\) fake images from \(g_\theta(P(\mathbf{z}))\)
2. Combine with \(N\) real images from the training set
3. Train a binary classifier to tell them apart
4. The resulting cross-entropy loss is the JSD between real and fake

THE GENERATOR’S GOAL

We want to find a generator function \(g_\theta\) that maps random draws from \(\mathcal{N}(\mathbf{0}, \mathbf{I})\) to images that are as similar as possible to real images from \(P(\mathbf{x})\).

TWO NETWORKS, TWO JOBS

Generator \(g_\theta\): takes a random vector \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) and produces an image of the same size as the training data (\(H \times W \times C\)). A neural network parameterized by \(\theta\).

THE MINIMAX OBJECTIVE

\[\min_\theta \max_\phi \; \frac{1}{2}\mathbb{E}_{P(\mathbf{x})}[\log D_\phi(\mathbf{x})] + \frac{1}{2}\mathbb{E}_{Q(\mathbf{z})}[\log(1 - D_\phi(g_\theta(\mathbf{z})))]\]

\(\phi\) wants to maximize this — find every difference between real and fake.

\(\theta\) wants to minimize this — make the fake distribution indistinguishable from real.

• Much like Scott Stapp and Fred Durst, they are adversaries

ALTERNATING TRAINING

We can’t optimize both simultaneously — each depends on the other.

• The generator reacts to the discriminator’s feedback and updates its parameters to produce more realistic images.
• The discriminator updates its parameters to better distinguish between real and fake images based on the new generator.

ALTERNATING TRAINING

Step 1 — Train the discriminator (generator frozen):

• Sample \(N\) real images from the training data
• Sample \(N\) noise vectors \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
• Generate \(N\) fake images using the current generator: \(\hat{\mathbf{x}} = g_\theta(\mathbf{z})\)
• Update \(\phi\) to minimize the cross-entropy of classifying real vs fake

ALTERNATING TRAINING

Step 2 — Train the generator (discriminator frozen):

• Sample \(N\) fresh noise vectors \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
• Generate \(N\) fake images: \(\hat{\mathbf{x}} = g_\theta(\mathbf{z})\)
• Pass them through the frozen discriminator
• Update \(\theta\) to maximize \(D_\phi(\hat{\mathbf{x}})\) — make the discriminator think the fakes are real

WHAT VICTORY LOOKS LIKE

At equilibrium:

• \(Q(\mathbf{x}) = P(\mathbf{x})\) — the fake distribution matches the real distribution
• \(D^*(\mathbf{x}) = \frac{1}{2}\) everywhere — the discriminator is at chance
• JSD \(= 0\) — the distributions are identical

THE LOSS FUNCTIONS

The minimax objective splits into two separate losses — one for each network.

Discriminator loss — maximize the ability to tell real from fake. Given N draws from the real distribution and N draws from the fake distribution under BCE loss:

\[\mathcal{L}_D(\phi) = -\frac{1}{2N}\left[ \sum_{i = 1}^N \log P(y = 1 | \mathbf x_i) + \sum_{i = 1}^N \log P(y = 0 | g_{\theta}(\mathbf z_i))\right]\]

where \(P(y = 1 | \mathbf{x}) = D_\phi(\mathbf{x})\) — the discriminator’s output.

Minimize \(\mathcal{L}_D\) — push \(D\) to score real images high and fake images low.

THE LOSS FUNCTIONS

The minimax objective splits into two separate losses — one for each network.

Generator loss — fool the discriminator. Try to maximize the probability that the discriminator thinks that the fake images are real:

\[\mathcal{L}_G(\theta) = -\frac{1}{N}\left[ \sum_{i = 1}^N \log P(y = 1 | g_\theta(\mathbf{z}_i))\right]\]

Minimize \(\mathcal{L}_G\) — push \(D\) to score fake images high.

HOW GRADIENTS FLOW

This is different than anything you’ve seen before - there are two separate optimization problems happening in sequence!

The generator and discriminator are separate networks. How does the generator learn from the discriminator’s judgment?

HOW GRADIENTS FLOW

During the generator step, the computation graph is:

\[\mathbf{z} \;\xrightarrow{g_\theta}\; \hat{\mathbf{x}} \;\xrightarrow{D_\phi}\; \text{score} \;\rightarrow\; \mathcal{L}_G\]

• The loss depends on the score. The score depends on \(\hat{\mathbf{x}}\). And \(\hat{\mathbf{x}}\) depends on \(\theta\).

• When we call loss.backward(), gradients flow backward through \(D_\phi\) (frozen — gradients pass through but don’t update \(\phi\)) into \(\hat{\mathbf{x}}\), then into \(g_\theta\).

• This is the same principle as LPIPS — a frozen network whose gradients flow through to update the generator.

HOW GRADIENTS FLOW — THE TWO STEPS

LET’S TAKE IT FOR A SPIN?

Normally, this is the point where we would implement the concepts we’ve discussed in a practical example.

• Generate some dogs.

THE ORIGINAL LOSS HAS A PROBLEM

In theory, the minimax game converges to an equilibrium state. In practice, the original BCE loss breaks down early in training.

At the start of training, the generator produces random noise. The discriminator easily classifies everything correctly with high confidence:

\[D(G(\mathbf{z})) \approx 0 \quad \text{for all fakes}\]

• The generator’s loss: \(\log(1 - D(G(\mathbf{z}))) \approx \log(1 - 0) = \log(1) = 0\)

• Gradient \(\approx\) 0. The generator is stuck exactly when it needs the most help.

WHY BCE FAILS

A DEEPER PROBLEM: NON-OVERLAPPING SUPPORTS

Early in training, the real and fake distributions may have no overlap in pixel space. Real dog images live on one manifold, random noise from an untrained generator lives somewhere completely different.

THE DEEPEST PROBLEM: MODE COLLAPSE

Much like many college students in the ChatGPT age, an optimizer given the opportunity to cheat and take the easy way out will do so.

• In QTM 110, all problem set solutions looked almost exactly the same! I wonder why…

THE DEEPEST PROBLEM: MODE COLLAPSE

THE DEEPEST PROBLEM: MODE COLLAPSE

WASSERSTEIN DISTANCE: A DIFFERENT MEASURE

Replace JSD with the Earth Mover’s Distance (Wasserstein-1 distance):

\[W(P, Q) = \inf_{\gamma \in \Pi(P, Q)} \mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim \gamma}\left[\|\mathbf{x} - \mathbf{y}\|\right]\]

• Intuition: if \(P\) and \(Q\) are piles of dirt, \(W(P, Q)\) is the minimum amount of work to reshape one pile into the other. “How much dirt do I need to move, and how far?”

• COMPLICATED LOSS BUT SIMPLE IMPLEMENTATION

WASSERSTEIN DISTANCE: A DIFFERENT MEASURE

Key property: \(W(P, Q)\) provides gradients everywhere, even when the distributions don’t overlap.

• JSD maxes out at \(\log 2\) when supports are disjoint — zero gradient.

Wasserstein distance grows linearly with the distance between disjoint distributions — always a gradient, always a direction to move. (Arjovsky et al., 2017)

WASSERSTEIN DISTANCE

When the real and fake distributions are far apart (early training), JSD is flat and useless. Wasserstein gives a clear gradient: “move in this direction to get closer.”

THE WASSERSTEIN GAN (WGAN)

The only architectural change: replace the sigmoid at the discriminator’s output with a linear activation (i.e., no activation at all).

\[D_{\text{BCE}}(\mathbf{x}) = \sigma(f_\phi(\mathbf{x})) \in [0, 1] \quad \longrightarrow \quad D_{\text{WGAN}}(\mathbf{x}) = f_\phi(\mathbf{x}) \in \mathbb{R}\]

The discriminator becomes a critic — it outputs a real-valued score instead of a probability. Positive = “looks real.” Negative = “looks fake.”

THE WASSERSTEIN GAN (WGAN)

The losses simplify accordingly — no log, no sigmoid, just raw scores:

Critic loss: maximize the gap between real and fake scores

\[\mathcal{L}_D = \mathbb{E}[D(G(\mathbf{z}))] - \mathbb{E}[D(\mathbf{x}_{\text{real}})]\]

Generator loss: maximize the score the critic assigns to fakes

\[\mathcal{L}_G = -\mathbb{E}[D(G(\mathbf{z}))]\]

Everything upstream of the final activation is identical — same convolutional layers, same feature extraction. We’re just removing the sigmoid bottleneck that squashes everything into \([0, 1]\) and causes the gradient saturation.

• Note that this also kills mode collapse since the critic can now provide a meaningful gradient even when the generator produces samples from a single mode.

THE LIPSCHITZ CONSTRAINT

There’s a catch. The Wasserstein distance is only valid if the critic is Lipschitz continuous — it can’t change too fast as a function of its input.

\[|D(\mathbf{x}_1) - D(\mathbf{x}_2)| \leq K \|\mathbf{x}_1 - \mathbf{x}_2\|\]

Without this constraint, the critic can assign \(+\infty\) to real and \(-\infty\) to fakes — the estimate blows up and training diverges.

HINGE LOSS: THE PRACTICAL SWEET SPOT

In practice, most modern GANs use hinge loss rather than pure Wasserstein:

\[\mathcal{L}_D = \mathbb{E}[\text{ReLU}(1 - D(\mathbf{x}_{\text{real}}))] + \mathbb{E}[\text{ReLU}(1 + D(\mathbf{x}_{\text{fake}}))]\]

\[\mathcal{L}_G = -\mathbb{E}[D(G(\mathbf{z}))]\]

The ReLU creates a margin: once D classifies correctly by a margin of 1, the gradient turns off. The discriminator stops getting stronger once it’s confident enough.

• This prevents D from dominating G — the exact problem that plagues both the original GAN and WGAN without careful tuning.

• The generator loss is the same as WGAN — no saturation, no log, always a gradient.

HINGE LOSS: IMPLEMENTATION

def discriminator_loss(real_logits, fake_logits):
    loss_real = torch.mean(F.relu(1.0 - real_logits))
    loss_fake = torch.mean(F.relu(1.0 + fake_logits))
    return loss_real + loss_fake
 
def generator_loss(fake_logits):
    return -torch.mean(fake_logits)

real_logits should be large positive \(\to\) 1 - large_positive is negative \(\to\) ReLU kills it \(\to\) no loss

fake_logits should be large negative \(\to\) 1 + large_negative is negative \(\to\) ReLU kills it \(\to\) no loss

Only contributes loss when D is wrong or not confident enough (margin < 1).

PATCHGAN: SPATIALLY LOCALIZED FEEDBACK

Instead of one real/fake score per image, output a grid of scores — one per image patch.

• A standard discriminator: “this whole image looks fake” \(\to\) one gradient signal for the entire image

• A PatchGAN: “the texture in the top-left looks fake, the fur in the center looks real, the background looks artificial” \(\to\) spatially localized feedback

class PatchDiscriminator(nn.Module):
    # Input: (B, 3, 128, 128)
    # Output: (B, 1, 8, 8) — 64 patch-level judgments

PATCHGAN ARCHITECTURE

class PatchDiscriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Conv2d(3, 64, 4, stride=2, padding=1),       # 128→64
            nn.LeakyReLU(0.2),
 
            nn.Conv2d(64, 128, 4, stride=2, padding=1),     # 64→32
            nn.GroupNorm(32, 128),
            nn.LeakyReLU(0.2),
 
            nn.Conv2d(128, 256, 4, stride=2, padding=1),    # 32→16
            nn.GroupNorm(32, 256),
            nn.LeakyReLU(0.2),
 
            nn.Conv2d(256, 1, 4, stride=1, padding=1),      # 16→15
        )
 
    def forward(self, x):
        return self.model(x)

LeakyReLU (not ReLU) — the discriminator needs gradients for “obviously fake” regions (negative activations). Standard ReLU would kill those gradients.

GroupNorm in the discriminator — same reasoning as the VAE. Per-image normalization, no batch dependence.

TRAINING TRICKS THAT MATTER

Spectral normalization on D: constrain how fast D can change — prevents arbitrarily sharp decisions

Higher LR for D: TTUR standard is 4x for the discriminator

Gradient clipping on G: occasional large gradients from D can destabilize G — clip_grad_norm_(G.parameters(), max_norm=10.0)

Differential Augmentation: apply random flips and crops to real and generated image before passing through the discriminator. This helps the discriminator not overfit to the training batches!

TRAINING TRICKS THAT MATTER

Even with all of these tricks, GANs are still notoriously unstable.

• Training is super picky

• Even sneezing in the wrong direction can destabilize training!

• We didn’t have that issue with the VAE…

• Our example today is perfect evidence of this

WHAT GAN LOSS CURVES LOOK LIKE

GAN losses don’t converge like VAE losses. They oscillate.

D_loss bounces around — when D improves, G catches up. When G improves, D adapts.

G_loss also bounces. Not monotonically decreasing.

WHAT GAN LOSS CURVES LOOK LIKE

WHY GANs DON’T BLUR

The discriminator only asks: “does this look real?”

It doesn’t ask: “does this match a specific target pixel-by-pixel?”

LET’S TAKE IT FOR A SPIN!

NOW, let’s actually generate some pets!

WHAT WE CAN’T DO

Modern responsive Gen AI architectures can’t use GANs alone!

No reconstruction: We can’t encode a real image. There’s no encoder.

• No way to match a picture of a dog to text that says “dog” (CLIP, coming later)

No structured latent space: Interpolating between \(\mathbf{z}\) vectors doesn’t produce meaningful transitions. No KL regularization.

• “What would a puppy look if the parents were a great dane and a chihuahua?”

WHAT WE CAN’T DO

Modern responsive Gen AI architectures can’t use GANs alone!

No editing: We can’t encode \(\to\) modify \(\to\) decode. One-way generation only.

• No drawing your crappy picture and having NanoBanana make it look good enough to sell on Etsy.

No easy conditioning: Requires conditional batch norm, projection discriminators — really expensive and not possible with current hardware

• You want a picture of a golden retriever? Here’s a corgi.

THE FUNDAMENTAL TRADEOFF

THE IDEA: ADD THE GAN TO THE VAE

What if the VAE’s decoder received a third gradient signal?