March 3, 2026
Last class - Detection and Segmentation
What is the main idea behind single-shot object detection (e.g. YOLO)? How do we reuse the CNN backbone?
What is the main idea behind semantic segmentation? How do we reuse the CNN backbone?
How do we create learnable upsampling for images?
Compress the image via a CNN to learn what is in the image
Blow the image back up to learn where those features are in the image
An important idea in deep learning: the encoder/decoder architecture
Encoder: compresses the input into a compact representation (bottleneck)
Decoder: reconstructs the output from the compact representation
The bottleneck is a low-dimensional code that preserves the essence of the original input!
All semester: discriminative models.
We modeled \(P(\mathbf y | \mathbf x)\) — given an image, predict a label.
We got very good at this. Transfer learning, fine-tuning, LoRA, segmentation.
But we’ve been ignoring half the picture.
Remember Lecture 1 — the Nature Box.
The joint distribution \(P(\mathbf y , \mathbf x)\) generates all data. We immediately took the discriminative path: model \(P(\mathbf y| \mathbf x)\), throw away \(P(\mathbf x)\).
Today we take the generative path: model \(P(\mathbf x)\) directly.
Not “given this image, is it a dog?” but “what does a dog look like?”
Not “classify this pixel” but “generate this pixel”
We trained logistic regression to distinguish frogs from planes. 88% accuracy.
Then we asked: “what image maximizes \(P(y = \text{ frog } | \mathbf x)\)?”
The result was garbage. Random noise that happened to trigger the frog detector.
Discriminative models know what isn’t a frog (via the decision boundary) but have no idea what a frog actually looks like.
To generate, we need \(P(\mathbf x)\) — the distribution of images.
A generative model is going to be a model that can generate new data points from the learned distribution.
Good frog vs. bad frog