DATASCI 447 Lecture 13: Transfer Learning or Why I Learned to Stop Worrying and Love the Pre-Trained Model

Administrative Stuff

WHERE WE LEFT OFF

Last time: ResNets solved the depth problem, augmentation filled the invariance gap, and we reframed CNNs as representation learners.

• How? Give me the main ideas.

We took a cat vs. dog backbone and transferred it to CIFAR-10 — features trained on two classes helped classify ten.

Today: we go deeper on why this works and exploit it fully with pretrained backbones from ImageNet.

THE CNN AS AN EMBEDDING MACHINE

Last lecture’s key reframe: a trained CNN backbone is an embedding function.

\[\phi: \mathbb{R}^{H \times W \times 3} \to \mathbb{R}^{d}\]

• It takes a 224×224×3 image (150,528 numbers with no useful structure) and maps it to a 512-dimensional vector.

• Similar images land near each other. Different images land far apart.

THE CNN AS AN EMBEDDING MACHINE

A CNN backbone maps images to vectors where “Persian cat” is near “Siamese cat” and far from “Boeing 747.”

• The backbone is an image embedding model.

• The classification head is just a linear classifier on top of the embeddings — the least interesting part.

WHAT MAKES A GOOD EMBEDDING?

A good embedding has three properties:

1. Images that share visual concepts land near each other

2. Images that differ in visually meaningful ways land far apart

3. Downstream tasks become simple linear operations on the embedding

THE LIMITATION OF OUR CAT VS. DOG EMBEDDINGS

Our cat vs. dog backbone learned good general features — edges, textures, shapes.

But its later layers were tuned specifically for distinguishing cats from dogs.

• “Pointy ears” is a highly informative dimension.

• “Has wings” is not — the network never needed to detect wings.

• “Has wheels” is not — the network never saw a car.

WHAT IF WE TRAINED ON EVERYTHING?

If we trained on a massive, diverse dataset — thousands of categories spanning animals, vehicles, food, scenes, tools, plants — the embedding would need to capture a much richer vocabulary:

• “Has wings” would matter (birds vs. dogs)

• “Has wheels” would matter (trucks vs. horses)

• “Is shiny” would matter (metal vs. fur)

• “Has text on it” would matter (street signs vs. landscapes)

THE UNIVERSAL FEATURE HIERARCHY

Early-layer features are universal. Edge detectors, color gradients, Gabor-like filters. These are determined by image physics, not by the task. Every CNN trained on natural images converges to similar early features — they look like primate visual cortex (V1).

Middle-layer features are broadly useful. Textures, corners, simple shapes. These appear across many visual domains.

Late-layer features are task-specific. “Cat face detector” doesn’t help with airplanes. “Wheel detector” doesn’t help with flowers.

The further you go from the input, the more task-specific the features become.

THE UNIVERSALITY CLAIM

The early-layer features of a CNN are determined by the statistical regularities of natural images, not by the downstream task.

• Edges, color gradients, and Gabor-like filters emerge because all natural images share the same local structure

• They’re produced by the same physics — light reflecting off surfaces, boundaries between objects and backgrounds

This is why transfer worked at all. This is why a cat vs. dog backbone has features useful for recognizing airplanes.

• But a backbone trained on just cats and dogs has a narrow vocabulary in its later layers. We want something broader.

FROM CAT VS. DOG TO IMAGENET

Our cat vs. dog backbone: 15,000 images, 2 classes, narrow domain.

• Decent transfer to CIFAR-10 but limited by the narrow training distribution.

IMAGENET: THE FEATURE FACTORY

ImageNet (ILSVRC): 1.2 million training images, 1,000 categories.

Categories span a huge range: hundreds of dog breeds, hundreds of bird species, vehicles, instruments, food, scenes, household objects.

IMAGENET: THE SCALE

Think about what training on this forces the network to learn.

• It can’t get by with “pointy ears vs. floppy ears” — it needs to distinguish 120 dog breeds from each other, AND from 1,000 other things.

• It needs edges, textures, material properties, shapes, parts, spatial relationships, color patterns — a comprehensive visual vocabulary.

The resulting embedding isn’t tuned for any single task. It’s tuned for visual understanding generally.

IMAGENET: THE COMPETITION

The ImageNet Large Scale Visual Recognition Challenge drove the deep learning revolution in computer vision.

• 2012: AlexNet (8 layers) — first deep CNN to win, cut error from 25.8% to 16.4%

• 2014: VGG (19 layers) — our design template, 7.3% error

• 2015: ResNet (152 layers) — 3.6% error. Better than human performance on this task.

IMAGENET: TOP-5 PREDICTIONS

IMAGENET: TOP-5 PREDICTIONS

A trained ImageNet model doesn’t just get the right answer — its top-5 guesses reveal what the embedding “sees.”

• Even the “wrong” guesses are visually sensible. A grille looks like a convertible. A cherry looks like a grape.

• The embedding has learned a perceptual similarity space — not just classification boundaries.

PRETRAINED MODELS: THE LIBRARY

PRETRAINED MODELS: THE LIBRARY

PyTorch’s torchvision.models gives you access to dozens of pretrained backbones — all trained on ImageNet, all free to use.

• Each model was trained for days or weeks on clusters of GPUs with 1.2 million images.

• You get all of that investment — all of those learned features — in a single line of code.

TRANSFER LEARNING: THE IDEA

Bottom layers = universal feature detectors (edges, textures).

Top layers = task-specific classifiers.

Strategy: keep the bottom, replace the top.

MODE 1: FEATURE EXTRACTION (FROZEN BACKBONE)

Simplest approach: freeze all pretrained weights, train only the new head.

• The backbone is a fixed feature extractor — no gradients flow through it.

• This is exactly what we did last lecture with our cat vs. dog backbone on CIFAR-10 — but now with a much better backbone.

MODE 1: FEATURE EXTRACTION (FROZEN BACKBONE)

FEATURE EXTRACTION: THE EVIDENCE

The Razavian et al. (2014) result was stunning:

Take an ImageNet-trained CNN. Freeze it. Extract features. Train an SVM on top.

• Beat the prior state of the art on Objects, Scenes, Birds, Flowers, Human Attributes, Object Attributes

• No task-specific architecture. No task-specific training of the backbone. Just good embeddings + a simple classifier.

The title of the paper: “CNN Features Off-the-Shelf: An Astounding Baseline for Recognition”

They weren’t kidding.

FEATURE EXTRACTION IN PYTORCH

import torchvision.models as models

# Load pretrained ResNet-50 (trained on ImageNet)
model = models.resnet50(pretrained=True)

# Freeze ALL parameters
for param in model.parameters():
    param.requires_grad = False

# Replace the classification head
model.fc = nn.Linear(2048, num_classes)

That’s it. Three lines. The full training pipeline is in the notebook — but the setup really is this simple.

FEATURE EXTRACTION IN PYTORCH

For our datasets:

# Cat vs. Dog (binary)
model.fc = nn.Linear(2048, 2)

# FER2013 (7 emotions)  
model.fc = nn.Linear(2048, 7)

# CIFAR-10 (10 classes)
model.fc = nn.Linear(2048, 10)

# Chest X-Ray Pneumonia (binary)
model.fc = nn.Linear(2048, 2)

Same backbone. Different heads. Each head has only \(2048 \times K + K\) parameters.

ImageNet Rules

1. Number of Channels: All images must have 3 channels (RGB).
2. Input Size: All images must be resized to a standard size depending on the backbone (e.g., 224×224 for ResNet).
3. Normalization: Images must be normalized using the ImageNet statistics:
   • Mean: [0.485, 0.456, 0.406]
   • Std: [0.229, 0.224, 0.225]

transforms.Compose([
    transforms.Resize(224),                    
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

HANDLING GRAYSCALE IMAGES

What about FER2013 (grayscale, 48×48) or chest X-rays (grayscale)?

The ImageNet backbone expects RGB — 3 input channels. Grayscale is 1 channel. The shapes don’t match.

Solution: repeat the grayscale channel three times.

transforms.Compose([
    transforms.Grayscale(num_output_channels=3),  # 1ch → 3ch
    transforms.Resize(224),                        # 48×48 → 224×224
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

We’ll use this exact transform in the notebook for both FER2013 and Chest X-Ray.

HANDLING GRAYSCALE IMAGES

Why does this work?

• The first-layer filters are looking for local spatial patterns — edges, gradients, textures.

• Those patterns exist in the grayscale image just the same. An edge is an edge whether it’s in one channel or three.

• Some first-layer filters are tuned to color contrasts (red vs. green). Those will see no useful signal since R=G=B everywhere. They’ll produce near-zero activations and the network learns to ignore them.

• The spatial filters carry the load — and those are the majority of what layer 1 learns.

This reinforces the point: early features are universal. Edge detectors don’t care about color channels.

FEATURE EXTRACTION + MLP

We don’t have to stop at a linear head.

Once we have the 2,048-dim embedding from the frozen backbone, we can feed it into a small MLP:

model.fc = nn.Sequential(
    nn.Linear(2048, 256),
    nn.ReLU(),
    nn.Dropout(0.3),
    nn.Linear(256, num_classes)
)

• This gives the classifier more capacity to learn nonlinear decision boundaries in the embedding space.

• It trains in minutes because we’re only training a tiny network on 2,048-dim inputs — no convolutions, no backprop through the backbone.

FEATURE EXTRACTION: RESULTS

Let’s go to the notebook and see how frozen ImageNet features perform across our datasets.

What to watch for:

• Training time. We’re only training a tiny head — this should be fast.

• Where it works great. Cat vs. Dog and CIFAR-10 are natural images, similar to ImageNet.

• Where it plateaus. FER2013 and Chest X-Rays are far from ImageNet’s domain. The frozen features may not have the right vocabulary.

That gap motivates the next section: fine-tuning.

THE “PRIOR” INTERPRETATION

The Bayesian framing:

• The pretrained weights are the prior. They encode everything the network learned about visual structure from 1.2 million images.

• The new classification head is the likelihood — it maps the prior beliefs about image content to the specific task.

WHEN TO USE FEATURE EXTRACTION

Feature extraction works best when:

• Your target dataset is small (hundreds to low thousands of images, weak likelihood offset by strong realistic prior!)

• Your target domain is similar to ImageNet (natural photos of objects, animals, scenes)

• You need a quick baseline — frozen feature extraction trains in minutes, not hours

• You want to compare many approaches rapidly

WHEN FEATURE EXTRACTION ISN’T ENOUGH

Feature extraction works great when the target domain is similar to ImageNet — natural photos of objects, animals, scenes.

But what about domains that look nothing like ImageNet?

• Medical imaging (X-rays, histopathology slides)

• Satellite imagery

• Industrial defect inspection

• Microscopy

WHEN FEATURE EXTRACTION ISN’T ENOUGH

The low-level features (edges, textures) still transfer. But the higher-level features (“dog face,” “sports car”) are useless for these domains.

We need to adapt the backbone.

MODE 2: FINE-TUNING

MODE 2: FINE-TUNING

Unfreeze some or all of the backbone layers. Train with a small learning rate.

• The pretrained weights are already near a good solution — large steps would destroy the useful features we’re trying to keep.

• Think of it as: the backbone starts with a good embedding and we gently adjust it to better suit our specific domain.

FINE-TUNING: THE INTUITION

Imagine you hired a professional photographer who’s spent 20 years shooting wildlife.

• Feature extraction = you hand them a new camera and say “shoot portraits of people, but don’t change anything about how you work.” They’ll do okay — composition skills transfer. But they won’t adjust their instincts for human faces.

• Fine-tuning = you say “shoot portraits, and feel free to gently adapt your technique.” They keep their 20 years of fundamentals but adjust framing, lighting, and focus for the new subject.

• Training from scratch = you hire someone who has never held a camera. Good luck.

HOW DEEP TO FINE-TUNE?

HOW DEEP TO FINE-TUNE?

You usually don’t need to unfreeze the whole backbone.

• Early layers (edges, textures) — universal. Leave them frozen.

• Later layers (object parts, compositions) — this is where ImageNet-specific features live. Unfreeze these.

• New head — learning from scratch. Always unfrozen.

HOW DEEP TO FINE-TUNE?

A useful mental model:

HOW DEEP TO FINE-TUNE?

Chest X-rays = far from ImageNet, moderate data → unfreeze later stages.

Cat vs. Dog = similar to ImageNet, decent data → frozen backbone is probably enough, but fine-tuning gives a small boost.

DIFFERENTIAL LEARNING RATES

Use different learning rates for different parts of the network.

optimizer = optim.Adam([
    {'params': model.layer1.parameters(), 'lr': 1e-6},  # early: tiny
    {'params': model.layer2.parameters(), 'lr': 1e-6},  # early: tiny
    {'params': model.layer3.parameters(), 'lr': 1e-5},  # mid: small
    {'params': model.layer4.parameters(), 'lr': 1e-5},  # late: small
    {'params': model.fc.parameters(),     'lr': 1e-3},  # head: normal
])

• Early features need tiny adjustments at most.

• The head needs to learn its task from scratch.

WHY SMALL LEARNING RATES?

WHY SMALL LEARNING RATES?

The pretrained weights represent a good region of the loss landscape — the result of training on 1.2 million images.

• A large learning rate would catapult the weights out of this region.

• The features that took weeks to learn would be destroyed in a few gradient steps.

Fine-tuning is like nudging a well-placed boulder. You want to roll it slightly downhill, not launch it off a cliff.

FINE-TUNING

Let’s go to the notebook.

We’ll fine-tune an ImageNet-pretrained ResNet:

• ~5,800 grayscale X-ray images, 2 classes (Normal vs. Pneumonia)

• Visually very different from ImageNet — no color, no objects, completely different texture vocabulary

• But edges, gradients, and intensity patterns still apply

FINE-TUNING: WHAT TO WATCH FOR

Pay attention to:

• Frozen vs. fine-tuned accuracy. The gap tells you how much the domain differs from ImageNet.

• Training time. Fine-tuning is slower than frozen feature extraction — we’re backpropagating through more layers.

• Overfitting risk. With only ~5,800 images and millions of unfrozen parameters, we need augmentation and early stopping. The pretrained initialization helps enormously — but it’s not a free pass.

YOU’VE ALREADY USED FINE-TUNING

Every student in this room has used a fine-tuned model. Every time you use ChatGPT, Claude, or any LLM chatbot:

• The base model (GPT, Claude’s base) = “pretrained backbone” — trained on massive text data to learn general language representations.

• Fine-tuning adapts it to follow instructions, be helpful, refuse harmful requests.

THE SAME IDEA, DIFFERENT DOMAINS

Transfer learning is not a vision trick. It’s a representation learning principle that applies everywhere.

THE COST PROBLEM

Full fine-tuning of a ResNet-50: update all ~25 million parameters.

• Store gradients for all of them.

• Adam stores 2 extra copies of every parameter (momentum + variance).

• Total memory: ~3× the model size just for optimizer state.

For ResNet-50, this is manageable (but requires essentially top-end consumer compute resources)

THE COST PROBLEM

But the trend is toward larger models:

• ViT-Large: 300M+ parameters

• LLMs: billions of parameters

• Multi-task deployment: 10 tasks = 10 full copies of the model

Is there a way to adapt a pretrained model without updating all the weights?

THE KEY OBSERVATION

When you fine-tune a large pretrained model, measure the weight updates:

\[\Delta W = W_{\text{fine-tuned}} - W_{\text{pretrained}}\]

Empirical finding: \(\Delta W\) is low-rank.

• Most of the “movement” in weight space during fine-tuning lives in a low-dimensional subspace.

• The model doesn’t need to change all 25M parameters independently — the changes are highly structured and redundant.

This connects to intrinsic dimensionality: the effective number of free parameters is much smaller than the actual count.

LoRA: LOW-RANK ADAPTATION

Instead of updating the full weight matrix \(W\), decompose the update:

\[W' = W + BA\]

where \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times d}\), and \(r \ll d\).

• The original \(W\) stays frozen.

• We only train \(B\) and \(A\) — two small matrices whose product is a rank-\(r\) update.

LoRA

LoRA

LoRA

The adapter is a “bypass” that adds a small correction to the frozen pretrained computation.

• \(W\) does the heavy lifting (frozen, no gradients)

• \(BA\) learns a small task-specific adjustment (trainable)

• Output = \(Wx + BAx\)

LoRA: INITIALIZATION

Key detail: \(B\) is initialized to zero.

\[W' = W + BA = W + 0 \cdot A = W\]

Training starts exactly at the pretrained model — no disruption.

The adapter gradually learns to add corrections as training progresses. If the pretrained model is already good enough for some layers, the adapter stays near zero for those layers.

• Compare this to full fine-tuning, where every gradient step immediately moves all weights away from the pretrained solution.

LoRA: THE LINEAR ALGEBRA

This is a low-rank prior on the weight update.

Connection to PCA: just as PCA says “most of the variance lives in a few directions,” LoRA says “most of the useful adaptation lives in a rank-\(r\) subspace.”

The rank \(r\) is a hyperparameter:

• Higher \(r\) = more capacity to adapt, more parameters

• Lower \(r\) = stronger prior, fewer parameters

• Typical values: \(r = 4, 8, 16\)

• Even \(r = 4\) works surprisingly well — the useful adaptation really is low-dimensional.

LoRA AT INFERENCE: ZERO EXTRA COST

At inference time, we can merge:

\[W' = W + BA\]

This is just a matrix addition — done once.

• The resulting \(W'\) is a regular weight matrix with no special structure.

• The model runs at exactly the same speed as the original.

• No extra latency, no extra memory at inference.

PRACTICAL COMPARISON

MULTI-TASK DEPLOYMENT: THE KILLER ADVANTAGE

One frozen backbone + tiny adapters per task.

• ResNet-50 backbone: 25M parameters (stored once)

• Each LoRA adapter: ~100K–500K parameters

LoRA IN THE NOTEBOOK

We’ll apply LoRA to:

1. Chest X-Ray Pneumonia — compare against frozen feature extraction and full fine-tuning on the same task

2. HAM10000 Skin Lesion Classification — 10,000 dermoscopic images, 7 classes of skin lesions (including melanoma)

• For the skin lesion dataset, 100% accuracy isn’t the goal — this is a screening tool where efficiency matters. LoRA’s speed and parameter efficiency make it the practical choice.

• We’ll compare all three approaches side by side in the notebook.

THE MODERN APPLIED VISION WORKFLOW

To summarize everything so far:

1. Take a pretrained backbone (ImageNet ResNet, etc.)

2. Choose adaptation strategy (frozen / fine-tuned / LoRA)

3. Add augmentation

4. Train with early stopping

BUT WHAT ABOUT SPATIAL TASKS?

Everything so far classifies whole images — one label per image.

But many real tasks require spatial outputs:

• Object detection: where are the objects and what are they? → bounding boxes + labels

• Semantic segmentation: which pixels belong to each class? → per-pixel labels

BUT WHAT ABOUT SPATIAL TASKS?

OBJECT DETECTION: THE KEY INSIGHT

The naive approach: slide a classifier window across the image at multiple scales.

• Computationally insane. Millions of windows per image.

OBJECT DETECTION: THE KEY INSIGHT

The key insight: the CNN backbone already computes a spatial grid of features.

• The \(7 \times 7 \times 512\) feature map before GAP tells us where features are, not just that they’re present.

• For classification, we averaged this away with GAP. For detection, we keep the spatial information and add a detection head on top.

YOLO: YOU ONLY LOOK ONCE

YOLO: YOU ONLY LOOK ONCE

YOLO takes a pretrained CNN backbone and replaces the single classification head with two heads:

1. Classification head — what’s in each grid cell?

2. Regression head — where exactly is the bounding box? (x, y, width, height)

YOLO: TWO HEADS, ONE BACKBONE

YOLO: TWO HEADS, ONE BACKBONE

• The backbone is the same ResNet/VGG we’ve been studying all block.

• Transfer learning applies directly — the backbone is pretrained on ImageNet.

• The detection heads are trained on detection data (e.g., COCO with bounding box annotations).

• Fast enough for real-time video — which is why YOLO is the standard for deployed detection systems.

YOLO: TWO HEADS, ONE BACKBONE

THE SEGMENTATION PROBLEM

Detection gives us boxes. But sometimes we need pixel-level precision:

• “Which exact pixels belong to the cat?”

This is semantic segmentation — a per-pixel classification task.

THE SEGMENTATION PROBLEM

CNN pipeline:

• Ingest image (1024 x 1024) → pretrained backbone → spatial compression → GAP (7 x 7 x 512)

Goal: Say which pixels belong to which class

• We’ve destroyed all of the pixel level information at the embedding bottleneck!

• Figure out how to go from embedding and return to 1024 x 1024 pixel space with new labels…

THE SEGMENTATION PROBLEM

The problem: our CNN backbone compresses \(224 \times 224\) down to \(7 \times 7\) through the funnel.

• We need to get back to \(224 \times 224\) for a per-pixel output.

• How do we reverse the funnel?

WHY SEGMENTATION NEEDS MORE THAN A HEAD

For classification and detection, the pretrained backbone does the heavy lifting.

• Classification: backbone → GAP → small head. Done.

• Detection: backbone → keep spatial grid → two small heads. Done.

For segmentation, we need an entire decoder that reverses the backbone’s spatial compression.

• Transposed convolutions or upsampling to go from \(7 \times 7\) back to \(224 \times 224\)

• Skip connections from encoder to decoder to recover spatial detail

PREVIEW: U-NET AND THE ENCODER-DECODER

Next lecture: the U-Net architecture.

• Encoder: our familiar CNN backbone — compresses to a low-resolution, high-semantic bottleneck

• Decoder: reverses the process — upsamples back to full resolution

• Skip connections between encoder and decoder at matching resolutions pass the spatial detail that the bottleneck discards

PREVIEW: U-NET AND THE ENCODER-DECODER

PREVIEW: WHY U-NET MATTERS BEYOND SEGMENTATION

The encoder-decoder structure will appear again when we get to generative models.

• In U-Net for segmentation: the bottleneck is a compressed feature representation.

• In a VAE (Variational Autoencoder): the bottleneck becomes a probabilistic latent space — not just compressed features, but a distribution we can sample from.

• In Stable Diffusion: the U-Net architecture is used between every denoising step to refine the generated image - a different bottleneck is created for each step to ensure high quality generated images.

NEXT LECTURE

Semantic segmentation and the U-Net encoder-decoder architecture.

Then: what happens when the decoder has no input image, and instead generates from a latent code?

• Can we use this structure to generate images?