DATASCI 447 Lecture 14: SPATIAL OUTPUTS — From ‘What’ to ‘Where’ to ‘Every Pixel’

Kevin McAlister

February 26, 2026

Administrative Stuff

Where We Stand

Questions:

  • Why does a CNN outperform every other method for image classification/regression tasks?

  • What is meant by “the universal hierarchy”?

  • Why does transfer learning against ImageNet work?

  • When might ImageNet features not be useful?

  • How do we fine tune a pre-trained model?

  • What is LoRA? Main gist?

Where We Stand

Last time: the transfer learning toolkit.

  • Frozen backbone + head: fast, works when target domain ≈ ImageNet
  • Fine-tuning: unfreeze later layers, small LR, for domain-shifted tasks
  • LoRA: low-rank adapters, near full fine-tuning quality at a fraction of the cost

Where We Stand

All of these answer the question “what is in this image?” with a single label.

Today: what if we need to know where? And what if we need to know for every pixel?

THE MODERN APPLIED VISION WORKFLOW

To summarize everything so far:

  1. Take a pretrained backbone (ImageNet ResNet, etc.)

  2. Choose adaptation strategy (frozen / fine-tuned / LoRA)

  3. Add augmentation

  4. Train with early stopping

This covers the vast majority of practical image classification and regression tasks.

Nobody trains from scratch in 2026 unless they’re doing research or working on a fundamentally new modality.

BUT WHAT ABOUT SPATIAL TASKS?

Everything so far classifies whole images — one label per image.

But many real tasks require spatial outputs:

  • Object detection: where are the objects and what are they? → bounding boxes + labels

  • Semantic segmentation: which pixels belong to each class? → per-pixel labels

BUT WHAT ABOUT SPATIAL TASKS?

OBJECT DETECTION: THE KEY INSIGHT

The naive approach: slide a classifier window across the image at multiple scales.

  • Computationally insane. Millions of windows per image.

OBJECT DETECTION: THE KEY INSIGHT

The key insight: the CNN backbone already computes a spatial grid of features.

  • The \(7 \times 7 \times 512\) feature map before GAP tells us where features are, not just that they’re present.

  • For classification, we averaged this away with GAP. For detection, we keep the spatial information and add a detection head on top.

YOLO: YOU ONLY LOOK ONCE

YOLO: YOU ONLY LOOK ONCE

YOLO takes a pretrained CNN backbone and replaces the single classification head with two heads:

  1. Classification head — what’s in each grid cell?

  2. Regression head — where exactly is the bounding box? (x, y, width, height)

One forward pass through the backbone + both heads = full detection.

That’s why it’s called “You Only Look Once” — detection as a single regression problem, not a pipeline.

YOLO: TWO HEADS, ONE BACKBONE