DATASCI 447 Lecture 13: Transfer Learning or Why I Learned to Stop Worrying and Love the Pre-Trained Model

Kevin McAlister

February 24, 2026

Administrative Stuff

WHERE WE LEFT OFF

Last time: ResNets solved the depth problem, augmentation filled the invariance gap, and we reframed CNNs as representation learners.

  • How? Give me the main ideas.

We took a cat vs. dog backbone and transferred it to CIFAR-10 — features trained on two classes helped classify ten.

Today: we go deeper on why this works and exploit it fully with pretrained backbones from ImageNet.

THE CNN AS AN EMBEDDING MACHINE

Last lecture’s key reframe: a trained CNN backbone is an embedding function.

\[\phi: \mathbb{R}^{H \times W \times 3} \to \mathbb{R}^{d}\]

  • It takes a 224×224×3 image (150,528 numbers with no useful structure) and maps it to a 512-dimensional vector.

  • Similar images land near each other. Different images land far apart.

THE CNN AS AN EMBEDDING MACHINE

A CNN backbone maps images to vectors where “Persian cat” is near “Siamese cat” and far from “Boeing 747.”

  • The backbone is an image embedding model.

  • The classification head is just a linear classifier on top of the embeddings — the least interesting part.

WHAT MAKES A GOOD EMBEDDING?

A good embedding has three properties:

  1. Images that share visual concepts land near each other

  2. Images that differ in visually meaningful ways land far apart

  3. Downstream tasks become simple linear operations on the embedding

THE LIMITATION OF OUR CAT VS. DOG EMBEDDINGS

Our cat vs. dog backbone learned good general features — edges, textures, shapes.

But its later layers were tuned specifically for distinguishing cats from dogs.

  • “Pointy ears” is a highly informative dimension.

  • “Has wings” is not — the network never needed to detect wings.

  • “Has wheels” is not — the network never saw a car.

Transfer to CIFAR-10 worked because low-level features are universal. But the embedding wasn’t optimized for broader visual understanding.

WHAT IF WE TRAINED ON EVERYTHING?

If we trained on a massive, diverse dataset — thousands of categories spanning animals, vehicles, food, scenes, tools, plants — the embedding would need to capture a much richer vocabulary:

  • “Has wings” would matter (birds vs. dogs)

  • “Has wheels” would matter (trucks vs. horses)

  • “Is shiny” would matter (metal vs. fur)

  • “Has text on it” would matter (street signs vs. landscapes)

The resulting embedding would be a general-purpose visual representation — useful for almost any downstream image task.

THE UNIVERSAL FEATURE HIERARCHY

Early-layer features are universal. Edge detectors, color gradients, Gabor-like filters. These are determined by image physics, not by the task. Every CNN trained on natural images converges to similar early features — they look like primate visual cortex (V1).

Middle-layer features are broadly useful. Textures, corners, simple shapes. These appear across many visual domains.

Late-layer features are task-specific. “Cat face detector” doesn’t help with airplanes. “Wheel detector” doesn’t help with flowers.

The further you go from the input, the more task-specific the features become.

THE UNIVERSALITY CLAIM

The early-layer features of a CNN are determined by the statistical regularities of natural images, not by the downstream task.

  • Edges, color gradients, and Gabor-like filters emerge because all natural images share the same local structure

  • They’re produced by the same physics — light reflecting off surfaces, boundaries between objects and backgrounds

This is why transfer worked at all. This is why a cat vs. dog backbone has features useful for recognizing airplanes.

  • But a backbone trained on just cats and dogs has a narrow vocabulary in its later layers. We want something broader.

FROM CAT VS. DOG TO IMAGENET

Our cat vs. dog backbone: 15,000 images, 2 classes, narrow domain.

  • Decent transfer to CIFAR-10 but limited by the narrow training distribution.

What we want: a backbone trained on a dataset so broad and diverse that the resulting embeddings capture most of what matters about visual content.

That backbone exists. The dataset it was trained on is called ImageNet.

IMAGENET: THE FEATURE FACTORY

ImageNet (ILSVRC): 1.2 million training images, 1,000 categories.

Categories span a huge range: hundreds of dog breeds, hundreds of bird species, vehicles, instruments, food, scenes, household objects.

IMAGENET: THE SCALE

Think about what training on this forces the network to learn.

  • It can’t get by with “pointy ears vs. floppy ears” — it needs to distinguish 120 dog breeds from each other, AND from 1,000 other things.

  • It needs edges, textures, material properties, shapes, parts, spatial relationships, color patterns — a comprehensive visual vocabulary.

The resulting embedding isn’t tuned for any single task. It’s tuned for visual understanding generally.

IMAGENET: THE COMPETITION

The ImageNet Large Scale Visual Recognition Challenge drove the deep learning revolution in computer vision.

  • 2012: AlexNet (8 layers) — first deep CNN to win, cut error from 25.8% to 16.4%

  • 2014: VGG (19 layers) — our design template, 7.3% error

  • 2015: ResNet (152 layers) — 3.6% error. Better than human performance on this task.

IMAGENET: TOP-5 PREDICTIONS