DATASCI 447 Lecture 12: State of The Art CNNs for Any Task.

Administrative Stuff

WHERE WE LEFT OFF

Last time: how to encode domain knowledge about images directly into the network architecture.

We formalized three beliefs about images, built them into convolutional layers, assembled those layers into VGG, and saw the dramatic payoff over fully connected networks.

Today: what happens when we push deeper, how to handle symmetries the architecture misses, and why the representation — not the classification — is the real product of a CNN.

THREE BELIEFS ABOUT IMAGES

Before we saw any data, we encoded three beliefs into our architecture.

What architectural choices encode each belief?

THREE BELIEFS → THREE CONSTRAINTS

THREE BELIEFS → THREE CONSTRAINTS

A single convolutional layer encodes all three.

These aren’t engineering convenience — they follow from how the physical world produces images: light interacts locally, objects move, and parts compose into wholes.

ARCHITECTURE MATTERS

The CNN doesn’t have more capacity than the MLP — the parameter counts are comparable.

• So why does it work so much better?

ARCHITECTURE MATTERS

Think about what this eliminates.

An MLP connecting a 224×224 image to its first hidden layer has 38.5 million parameters — one for every pixel-to-unit connection. Most of these encode relationships between distant pixels that have nothing to do with each other.

REDUCE VARIANCE BY INDUCING BIAS

Same logic as Ridge regression:

More bias (structural constraints) → dramatically less variance (fewer effectively free parameters) → better generalization when the prior matches the data.

REDUCE VARIANCE BY INDUCING BIAS

When the prior is correct (images have local, translation-equivariant structure), the tradeoff is overwhelmingly favorable.

• MLP on CIFAR-10: ~45% test accuracy.

• CNN on CIFAR-10, same parameter budget: ~65% test accuracy.

WHEN THE PRIOR IS WRONG

When the prior is wrong, convolutions hurt.

• Tabular data has no spatial locality.

• Nobody uses CNNs on Kaggle tabular competitions. XGBoost wins there because it makes no spatial assumptions.

Architecture is a prior — match it to your data.

VGG: THE STANDARD TEMPLATE

VGG (Simonyan & Zisserman, 2014) is the cleanest expression of the three beliefs.

• Always use 3×3 filters.

• Double the channels each time you halve the spatial dimensions.

VGG: THE STANDARD TEMPLATE

The architecture is just this pattern repeated:

\[[\text{Conv}(3 \times 3) \to \text{BN} \to \text{ReLU}] \times B \to \text{MaxPool}(2 \times 2)\]

• Each block detects features at one spatial scale.

• Pooling compresses space.

• The next block operates on the compressed representation — detecting combinations of the previous features over a larger effective region.

WHY ALWAYS 3×3?

Why not use a single 7×7 filter to see a bigger region?

• Because two stacked 3×3 conv layers have the same effective receptive field as one 5×5 layer. Three stacked 3×3 layers match one 7×7.

But stacked 3×3 is strictly better:

• Fewer parameters: \(2 \times (3^2) = 18\) vs. \(5^2 = 25\) for equivalent receptive field (28% fewer)
• More nonlinearities: a ReLU between each layer → more expressive per parameter
• Better gradient flow: more BN layers to stabilize training

THE EFFECTIVE RECEPTIVE FIELD

For a 224×224 input:

By the final block, each unit “sees” nearly the entire image — built entirely from 9-parameter filters.

GLOBAL AVERAGE POOLING

After the final conv block: \(7 \times 7 \times 512 = 25{,}088\) values.

If we flatten this directly and connect to an FC layer with 512 hidden units:

\[25{,}088 \times 512 = 12.8 \text{ million parameters}\]

That’s more parameters than the entire conv backbone that built the features. Incredibly wasteful.

GLOBAL AVERAGE POOLING

Key idea: Each of those 512 feature maps already represents “how much of feature \(k\) is present in this image.”

We don’t need 25,088 numbers to summarize that — we need one number per feature map.

\[\text{GAP}: \mathbf{F}^{H \times W \times C} \longrightarrow \bar{\mathbf{f}}^{C \times 1}, \quad \text{where } \bar{f}_k = \frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W} F_{ijk}\]

The classifier becomes nn.Linear(512, K). For 10 classes: 5,120 parameters vs. 12.8 million.

• Just do this. It works really well!

THE FUNNEL: TRADING SPACE FOR SEMANTICS

Each pooling step halves the spatial dimensions while the number of channels grows:

• After Block 1 (112×112, 64 channels): Simple local features. “There’s a vertical edge here.” “This patch is brown.”
• After Block 3 (28×28, 256 channels): Combinations of features. “This region has a furry texture.” “There’s something eye-shaped here.”
• After Block 5 (7×7, 512 channels): Semantic features. “This is a cat face.” “This is striped fur.”

The funnel architecture creates the conditions for hierarchical feature construction. Training fills in the content in the optimal way.

VGG PERFORMANCE

We’ll now see VGG in action.

• We’ll look at accuracy and compute time.

THE TRADEOFF

Carefully constructed depth trumps width/dumb depth every time in terms of accuracy.

• But it costs a lot. Even with GPU acceleration, VGG takes substantially more wall-clock time than a shallow CNN or MLP.

• The convolution operation at each position is cheap (9 multiplies for a 3×3 filter). But we’re doing it at every spatial position, across every filter, at every layer.

• Is it worth it? Almost always yes since prediction is incredibly cheap after training.

• But we’ll soon see cases where we want to go even deeper — and that’s where things break.

FER2013: A HARDER PROBLEM

Pets (cat vs. dog) is a relatively easy classification task.

• The differences between cats and dogs are structurally obvious

FER2013 is much harder: 48×48 grayscale images of faces, 7 emotion categories (angry, disgust, fear, happy, sad, surprise, neutral).

FER2013: A HARDER PROBLEM

The differences between emotions are semantically subtle.

• The same face can look angry or disgusted based on small changes around the eyes and mouth.

• A slight downturn of the lips.

• A furrowing of the brow.

These are fine-grained, high-level features

• Detect not just “there are eyes here”

• “The eyes are narrowed in a specific way that suggests anger rather than concentration.”

VGG with a few blocks does okay. Can we do better with more depth?

ADDING DEPTH FOR DIFFICULT FEATURES

Since FER2013 images are relatively small (48×48)

• Add more convolution blocks before each max pool. More conv layers at each spatial scale means more feature compositions before we compress.

\[[\text{Conv}(3\times3) \to \text{BN} \to \text{ReLU}] \times 3 \to \text{MaxPool} \to \ldots\]

• More depth at each scale = more opportunity to detect the subtle feature combinations

ADDING DEPTH: WHAT HAPPENS?

Adding depth improves performance — up to a point.

• Training accuracy degrades

• Test accuracy degrades

THE DEGRADATION PROBLEM

This is NOT overfitting.

If it were overfitting, training accuracy would be high and test accuracy would be low. That’s not what we see.

The deeper model is worse on the training set.

• Deep is better

• But the network can’t find that solution.

• Optimization failure.

WHY DEPTH FAILS: THE GRADIENT CHAIN

Callback to L10: gradients are a product of Jacobians through every layer.

\[\frac{\partial \mathcal{L}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathbf{h}_D} \cdot \frac{\partial \mathbf{h}_D}{\partial \mathbf{h}_{D-1}} \cdot \frac{\partial \mathbf{h}_{D-1}}{\partial \mathbf{h}_{D-2}} \cdots \frac{\partial \mathbf{h}_1}{\partial \mathbf{x}}\]

• Even with BN keeping activations healthy, ill-conditioned optimization landscape.

• One small value ruins the whole thing!

WHY DEPTH FAILS: COMPOUNDING FACTORS

For FER2013, the problem is compounded:

• Small spatial resolution (48×48): fewer spatial positions means less redundancy in the gradient signal. Each gradient estimate is noisier.

• Subtle discriminative features: the differences between “angry” and “sad” live in tiny pixel-value variations around the eyes and mouth. Small very localized gradients in the first place!

THE RESIDUAL INSIGHT

Key idea (He et al., 2015):

Instead of asking a block to learn the desired mapping \(h(\mathbf{x})\), ask it to learn the residual — how to adjust the current representation.

\[h(\mathbf{x}) = f(\mathbf{x}) + \mathbf{x}\]

The network learns \(f(\mathbf{x}) = h(\mathbf{x}) - \mathbf{x}\): “what should I change?”

• Not “What is the next representation?”

THE RESIDUAL INSIGHT

We’re not really changing what we’re doing.

• The network still learns convolutions, still applies BN and ReLU, still builds hierarchical features. The representational capacity is essentially the same.

• Learn corrections rather than complete transformations.

THE RESIDUAL BLOCK

The residual block:

Input x ─────────────────────────── (+) ──→ ReLU ──→ Output
   │                                  ↑
   └──→ Conv → BN → ReLU → Conv → BN ─┘
         (the residual branch f(x))

The skip connection is an identity shortcut — no learnable parameters. The block’s output is \(f(\mathbf{x}) + \mathbf{x}\).

THE RESIDUAL BLOCK: ORDERING MATTERS

The residual path should always start with a conv and end with a batch norm

• If the residual branch started with ReLU, the output \(f(\mathbf{x})\) could only be non-negative (ReLU kills negatives).

• Same deal if we end with ReLU!

A face-recognition network might need to suppress a “smiling” feature when it detects “furrowed brow” — that requires a negative residual. If we constrain residuals to be positive, we lose half the adjustment space.

WHY SKIP CONNECTIONS FIX GRADIENTS: THE DIRECT ARGUMENT

\[ h = f(\mathbf{x}) + \mathbf{x} \]

The gradient through a residual block:

\[\frac{\partial h}{\partial \mathbf{x}} = \frac{\partial f(\mathbf{x})}{\partial \mathbf{x}} + \mathbf{I}\]

• The identity term \(\mathbf{I}\) means gradients always have a direct path backward, regardless of what \(f\) does.

• Where have you seen this idea before?

This is the “gradient highway” — information flows backward through the skip connections unimpeded by the transformations in the main path.

THE DEEPER ARGUMENT: FROM MULTIPLICATION TO ADDITION

But the \(+ \mathbf{I}\) argument only tells us about one block. What happens when we stack many blocks?

Let’s trace the math carefully. This is where the real insight lives.

PLAIN NETWORKS: NESTED FUNCTIONS

In a plain network, the forward pass is a nested composition:

\[y = f_3(f_2(f_1(\mathbf{x})))\]

The gradient is a product of Jacobians (chain rule):

\[\frac{\partial y}{\partial \mathbf{x}} = \frac{\partial f_3}{\partial f_2} \cdot \frac{\partial f_2}{\partial f_1} \cdot \frac{\partial f_1}{\partial \mathbf{x}}\]

If any single factor is small, the entire product vanishes.

RESIDUAL NETWORKS: A DIFFERENT STRUCTURE

Now with residual connections, each block computes:

\[h_l = f_l(h_{l-1}) + h_{l-1}\]

Let’s trace a 3-block network carefully. Write \(h_0 = \mathbf{x}\) (input).

EXPANDING THE RESIDUAL NETWORK

Block 1:

\[h_1 = f_1(h_0) + h_0\]

Block 2:

\[h_2 = f_2(h_1) + h_1 = f_2(h_1) + f_1(h_0) + h_0\]

Block 3:

\[h_3 = f_3(h_2) + h_2 = f_3(h_2) + f_2(h_1) + f_1(h_0) + h_0\]

The output is no longer a nested composition — it’s the input plus a sum of residuals.

THE GRADIENT BECOMES A SUM

Now differentiate \(h_3\) with respect to \(\mathbf{x}\):

\[\frac{\partial h_3}{\partial \mathbf{x}} = \frac{\partial f_3}{\partial h_2}\frac{\partial h_2}{\partial \mathbf{x}} + \frac{\partial h_2}{\partial \mathbf{x}}\]

Since \(\frac{\partial h_2}{\partial \mathbf{x}} = \frac{\partial f_2}{\partial h_1}\frac{\partial h_1}{\partial \mathbf{x}} + \frac{\partial h_1}{\partial \mathbf{x}}\) and \(\frac{\partial h_1}{\partial \mathbf{x}} = \frac{\partial f_1}{\partial \mathbf{x}} + \mathbf{I}\), we can expand:

\[\frac{\partial h_3}{\partial \mathbf{x}} = \left(1 + \frac{\partial f_3}{\partial h_2}\right)\left(1 + \frac{\partial f_2}{\partial h_1}\right)\left(1 + \frac{\partial f_1}{\partial \mathbf{x}}\right)\]

EXPANDING: ALL POSSIBLE PATHS

Expand that product:

\[\frac{\partial h_3}{\partial \mathbf{x}} = \underbrace{1}_{\text{skip all}} + \underbrace{\frac{\partial f_1}{\partial \mathbf{x}}}_{\text{block 1 only}} + \underbrace{\frac{\partial f_2}{\partial h_1}}_{\text{block 2 only}} + \underbrace{\frac{\partial f_3}{\partial h_2}}_{\text{block 3 only}}\]

\[+ \underbrace{\frac{\partial f_2}{\partial h_1}\frac{\partial f_1}{\partial \mathbf{x}}}_{\text{blocks 1\&2}} + \underbrace{\frac{\partial f_3}{\partial h_2}\frac{\partial f_1}{\partial \mathbf{x}}}_{\text{blocks 1\&3}} + \underbrace{\frac{\partial f_3}{\partial h_2}\frac{\partial f_2}{\partial h_1}}_{\text{blocks 2\&3}}\]

\[+ \underbrace{\frac{\partial f_3}{\partial h_2}\frac{\partial f_2}{\partial h_1}\frac{\partial f_1}{\partial \mathbf{x}}}_{\text{all blocks}}\]

That’s \(2^3 = 8\) terms — one for every combination of “take the skip” or “take the residual branch” at each block.

THE KEY SHIFT: EVERY SUBSET IS A PATH

Each of the 8 terms corresponds to a path through the network:

The gradient is the sum over all \(2^D\) paths that respect ordering.

FROM MULTIPLICATION TO ADDITION: THE KEY SHIFT

FROM MULTIPLICATION TO ADDITION: WHY THIS MATTERS

The \(1\) in each factor \((1 + \frac{\partial f_l}{\partial h_{l-1}})\) is the skip connection!

• Even if the residual gradients \(\frac{\partial f_l}{\partial h}\) are tiny, the short paths (including the direct path through all skips (gradient = 1)) always carry signal.

• For \(D = 50\) blocks: \(2^{50} \approx 10^{15}\) paths. The chance of all of them vanishing simultaneously is essentially zero.

• This is why ResNets can be 50, 100, 152 layers deep. The gradient doesn’t have to survive a single long chain — it takes all possible shortcuts.

THE RESNET ARCHITECTURE

Same funnel as VGG — spatial dims halve, channels double — but built from residual blocks instead of plain conv layers.

Global average pooling replaces the FC head (same improvement we discussed earlier).

All trainable. Error keeps improving with depth — the degradation problem is solved.

RESULTS: DEPTH WORKS AGAIN

The skip connection doesn’t add meaningful capacity — it enables optimization of the capacity that was already there.

• The plain network had the representational power - just couldn’t find it

• Skip connections make the good solutions findable.

ResNet SGD landscape is much smoother and much more easy to work with!

(Look at notebook for performance info)

THREE WAYS TO UNDERSTAND SKIP CONNECTIONS

(1) Gradient flow: The identity highway prevents vanishing gradients regardless of depth. Gradients always have a direct path of length zero.

(2) Implicit ensemble: A ResNet with \(D\) blocks has \(2^D\) paths of varying length. It behaves like an ensemble of shallow and deep networks. Sorta like a built in dropout in the convolutional backbone.

(3) Residual learning: If the current representation is already good, the block only needs to learn small corrections. Most blocks in a trained ResNet do learn small residuals — the skip carries most of the information. The block is an editor, not an author.

WHY DEPTH IS IMPORTANT: THE FACE EXAMPLE

Consider detecting “a face.”

• A face = two eyes above a nose above a mouth.

• An eye = a circular outline with a dark center.

• A circular outline = curved edges arranged in a ring.

• Curved edges = specific combinations of oriented edge fragments.

WHY DEPTH WORKS

A deep network mirrors this hierarchy:

Layer 1: detects edge fragments (shared across all features).

Layer 2: combines edges into curves and corners (reused for eyes, noses, mouths).

Layer 3: combines curves into parts — eyes, noses, mouths.

Layer 4: combines parts into faces.

Each layer reuses the features below it. An edge detector trained in layer 1 helps detect both eyes and mouths. The total parameter count grows polynomially with the complexity of the target function.

THE INVARIANCE GAP

Recall from earlier: conv layers give us translation equivariance for free.

• Weight sharing = same filter applied everywhere

But conv layers are NOT equivariant to rotation, scale, or color shifts.

• A rotated cat produces a different feature map

• A darker photo of the same dog produces different activations

The architecture doesn’t know that these are the same animal.

THE INVARIANCE GAP

Think about what this means for FER2013.

• A face tilted slightly to the left and the same face straight-on express the same emotion — but the pixel patterns around the eyes and mouth shift in ways that have nothing to do with the emotional content.

• A face photographed under bright studio lighting and the same face in a dim room look completely different at the pixel level. But the expression is identical.

The label and semantic meaning of the image hasn’t changed. The pixels have.

How do we get invariance to these transformations?

DATA AUGMENTATION: THE IDEA

Apply random transformations that preserve semantics during training.

Each time the network sees a training image, randomly:

• Flip it
• Crop it
• Rotate it
• Adjust brightness/contrast

The meaning doesn’t change!

• The model never sees the exact same image twice. Effectively, we’re multiplying our dataset size by the number of augmentation combinations.

DATA AUGMENTATION: THE IDEA

Why does FER2013 need this so badly?

• ~28,000 training images at 48×48 grayscale for a semantically hard task!

• Without augmentation: the network memorizes and misses the nuanced variations in expression.

• With augmentation: the same face appears flipped, slightly rotated, slightly cropped. The only thing consistent across all versions is the expression. The network is forced to learn what actually matters. (In theory)

COMMON AUGMENTATIONS

Horizontal flip: The most basic augmentation. An angry face facing left and an angry face facing right express the same emotion. (Note: we don’t vertically flip — upside-down faces are rare in practice.)

Random crop + resize: Take a random sub-region and resize it back to the original dimensions. Forces the network to recognize the expression even when parts of the face are slightly cut off. Also provides slight scale variation.

Small rotation: ±15° or so. Real photos aren’t perfectly level. Faces aren’t always perfectly upright.

Color jitter: Random changes to brightness and contrast. (For grayscale FER2013, this is brightness/contrast only. For RGB datasets like our cat vs. dog data, we also jitter saturation and hue.)

COMMON AUGMENTATIONS

The key constraint: every augmentation must preserve the label.

• Horizontal flip preserves “angry” ✓

• Random crop preserves “angry” (as long as the key facial features are visible) ✓

• Brightness jitter preserves “angry” ✓

But:

• rotating a “6” by 180° turns it into a “9” ✗

• flipping a “b” horizontally turns it into a “d” ✗

The choice of augmentations is domain knowledge.

AUGMENTATION IN PYTORCH

from torchvision import transforms

train_transform = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomResizedCrop(48, scale=(0.85, 1.0)),
    transforms.RandomRotation(15),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.ToTensor(), #Normal
    transforms.Normalize(mean=[0.5], std=[0.5]), #Normal
])

AUGMENTATION IN PYTORCH

Critical detail: augmentation is applied only during training.

• At test time, do deterministic operations (normalize, center, etc.)

• We’re evaluating the model’s ability to generalize to unseen data as the world produces it.

This is a 5-line code change in the data pipeline. No changes to the model, no changes to the training loop. Massive impact.

REFRAMING: WHAT DOES A CNN ACTUALLY PRODUCE?

The standard view: a CNN takes an image and outputs a class label.

• Cat photo → CNN → “cat” (probability 0.94)

This is correct but incomplete. It misses the most important thing the CNN does.

REFRAMING: WHAT DOES A CNN ACTUALLY PRODUCE?

The better view: the real output of a CNN is the representation in the penultimate layer — the vector right before the final classifier.

• Cat photo → Conv backbone → [0.82, -0.14, 1.37, …, 0.56] → Linear → “cat”

• That 512-dimensional vector (after GAP) is a summary of the image in terms of learned visual concepts.

• A 512 number representation of the image, not the class label

THE FEATURE TRANSFORM

The conv backbone learns a nonlinear feature transform:

\[\phi: \mathbb{R}^{H \times W \times 3} \to \mathbb{R}^{d}\]

that maps raw pixels into a space where classes become linearly separable.

For VGG/ResNet with global average pooling: \(d = 512\).

• This 512-dimensional vector summarizes the entire image.

• Each dimension corresponds to “how much of learned concept \(k\) is present in this image.”

One dimension might encode “amount of fur texture.” Another might encode “presence of pointed ears.” Another might encode “orange coloring.” The network figured out which concepts matter — we never told it.

REVERSE ML?

Here’s the deeper connection to the entire course:

All of classical ML — logistic regression, SVM, random forests — works well when given good features. The hard part was always feature engineering. Practitioners spent months hand-crafting features for each domain.

CNNs automate feature engineering. The conv backbone learns \(\phi\) — the feature transform — end-to-end from data. The classification head is just logistic regression on top.

This is why deep learning took over computer vision.

• Not better classifiers (like XGBoost)
• The features got better.

WHAT DID OUR CAT VS. DOG BACKBONE LEARN?

Our ResNet was trained on 7,500 cat and dog images at 224×224. It achieved ~93% test accuracy with VGG, and 96% with ResNet + augmentation.

Along the way, it learned to detect:

• Edges and gradients (layer 1)

• Textures (layers 2–3)

• Parts (layers 3–4)

• Compositions (layer 5) — “pointy ears + narrow face = cat face”, “floppy ears + wide snout = dog face”

THE EXPERIMENT: TRANSFER TO CIFAR-10

Here’s the payoff of thinking about CNNs as representation learners.

Take our ResNet trained on cat vs. dog (7,500 images, 224×224, binary classification with augmentation).

• Remove the classification head. Keep only the conv backbone — our learned \(\phi\).

THE EXPERIMENT: TRANSFER TO CIFAR-10

CIFAR-10: 10 classes — airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck.

• Our backbone has never seen an airplane. It has never seen a frog. It was trained entirely on cats and dogs.

• There’s also a practical problem: our backbone expects 224×224 inputs, but CIFAR-10 images are 32×32.

THE EXPERIMENT: UPSCALING CIFAR-10

Solution: upscale the CIFAR-10 images from 32×32 to 224×224.

cifar_transfer_transform = transforms.Compose([
    transforms.Resize(224),       ## 32×32 → 224×224
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

• Yes, we’re blowing up a tiny image to 7× its original size. The upscaled image looks blocky and blurry to us.

• But the conv backbone doesn’t care about that…

THE EXPERIMENT: FROZEN BACKBONE + NEW HEAD

Freeze the entire cat vs. dog backbone. No gradients flow through it — we treat it as a fixed feature extractor.

• Train only a new linear classification head: nn.Linear(512, 10).

\[\mathbf{x}^{\text{CIFAR}} \xrightarrow{\text{resize to 224}} \mathbf{x}' \xrightarrow{\text{frozen } \phi_{\text{CatDog}}} \mathbf{z} \in \mathbb{R}^{512} \xrightarrow{\text{new Linear}} \hat{y} \in \mathbb{R}^{10}\]

The backbone extracts features. The new head learns to classify using those features. That’s it.

WHY DOES THIS WORK?

Think about what the cat vs. dog backbone learned — and what transfers.

Edges and gradients — every image has edges. A vertical edge in a cat photo = vertical edge in an airplane photo.

Textures and patterns — fur texture is specific to animals, rough textures appear everywhere, trucks have smooth metal, frogs have bumpy skin. Nothing unique about textures!

Shape primitives — curved edges, sharp corners, elongated regions. Cat ears and airplane tails are different shapes, but the ability to detect shapes partially transfers.

“Cat face” and “dog face” detectors — these are specific to the training task. Get rid of it and use the new features we learned for any task!

THE UNIVERSAL FEATURE HIERARCHY

This is the key insight:

• Early-layer features are universal. They’re determined by the physics of images, not by the classification task.

• Middle-layer features are broadly useful. Textures, corners, and simple shapes appear across many visual domains.

• Late-layer features are task-specific. “Cat face detector” doesn’t help with airplanes.

The further you go from the input, the more task-specific the features become.

THE UNIVERSAL FEATURE HIERARCHY

This has a direct implication:

If someone trained a CNN on a massive, diverse dataset — say, 1.2 million images across 1,000 categories covering animals, vehicles, food, scenes, objects — the features would be even more universal.

WHAT THIS MEANS

If you need to classify images, the question isn’t “how do I build a classifier?”

The question is “where do I get a good representation?”

• Option 1: Learn \(\phi\) from scratch on your data. Requires tens of thousands of images, a lot of compute, and a lot of patience. This is what we’ve been doing all lecture.

• Option 2: Use a \(\phi\) that someone else already learned on a massive dataset. Reuse their backbone. Train only a small head on your task. This works surprisingly well!

That’s transfer learning — and it’s the entire subject of next lecture.

NEXT LECTURE: TRANSFER LEARNING AND EFFICIENT ADAPTATION

If someone already learned a good \(\phi\) on a massive dataset, can we reuse it?

Three adaptation modes:

• Feature extraction — freeze the backbone, train only a new head. The simplest approach. (We just did this with our cat vs. dog backbone.)

• Fine-tuning — unfreeze some layers, train with a small learning rate. The backbone adjusts to your domain.

• LoRA — learn a small, low-rank correction to the pretrained weights. The efficient approach for large models.

Nobody trains CNNs from scratch for regression, classification, object detection, etc. in 2026! Next lecture, we’ll see why.