DATASCI 447 Lecture 14: SPATIAL OUTPUTS — From ‘What’ to ‘Where’ to ‘Every Pixel’

Administrative Stuff

Where We Stand

Questions:

• Why does a CNN outperform every other method for image classification/regression tasks?

• What is meant by “the universal hierarchy”?

• Why does transfer learning against ImageNet work?

• When might ImageNet features not be useful?

• How do we fine tune a pre-trained model?

• What is LoRA? Main gist?

Where We Stand

Last time: the transfer learning toolkit.

• Frozen backbone + head: fast, works when target domain ≈ ImageNet
• Fine-tuning: unfreeze later layers, small LR, for domain-shifted tasks
• LoRA: low-rank adapters, near full fine-tuning quality at a fraction of the cost

Where We Stand

All of these answer the question “what is in this image?” with a single label.

Today: what if we need to know where? And what if we need to know for every pixel?

THE MODERN APPLIED VISION WORKFLOW

To summarize everything so far:

1. Take a pretrained backbone (ImageNet ResNet, etc.)

2. Choose adaptation strategy (frozen / fine-tuned / LoRA)

3. Add augmentation

4. Train with early stopping

BUT WHAT ABOUT SPATIAL TASKS?

Everything so far classifies whole images — one label per image.

But many real tasks require spatial outputs:

• Object detection: where are the objects and what are they? → bounding boxes + labels

• Semantic segmentation: which pixels belong to each class? → per-pixel labels

BUT WHAT ABOUT SPATIAL TASKS?

OBJECT DETECTION: THE KEY INSIGHT

The naive approach: slide a classifier window across the image at multiple scales.

• Computationally insane. Millions of windows per image.

OBJECT DETECTION: THE KEY INSIGHT

The key insight: the CNN backbone already computes a spatial grid of features.

• The \(7 \times 7 \times 512\) feature map before GAP tells us where features are, not just that they’re present.

• For classification, we averaged this away with GAP. For detection, we keep the spatial information and add a detection head on top.

YOLO: YOU ONLY LOOK ONCE

YOLO: YOU ONLY LOOK ONCE

YOLO takes a pretrained CNN backbone and replaces the single classification head with two heads:

1. Classification head — what’s in each grid cell?

2. Regression head — where exactly is the bounding box? (x, y, width, height)

YOLO: TWO HEADS, ONE BACKBONE

YOLO: TWO HEADS, ONE BACKBONE

• The backbone is the same ResNet/VGG we’ve been studying all block.

• Transfer learning applies directly — the backbone is pretrained on ImageNet.

• The detection heads are trained on detection data (e.g., COCO with bounding box annotations).

• Fast enough for real-time video — which is why YOLO is the standard for deployed detection systems.

YOLO: TWO HEADS, ONE BACKBONE

YOLO TRAINING

Some notes:

1. YOLO is more of an idea than a single method these days

• A company called Ultralytics “owns” YOLO implementations and requires you to use their package to implement newer versions

• Not a terrible thing, but we like full open-source alternatives

• SSD (Single Shot MultiBox Detector) and FCOS (Fully Convolutional One-Stage Object Detection) are good torchvision contained alternatives

YOLO TRAINING

2. Object detection requires very specific training data

• Annotated images with bounding boxes for each object

YOLO TRAINING

3. YOLO fine-tuning can be a little picky

• Be prepared to experiment with different hyperparameters and training strategies.

• Data augmentation is really important here since we need to learn that dogs can appear in various poses, lighting conditions, and backgrounds.

YOLO TRAINING

4. YOLO loss needs to be altered

• Most area in an image is not a thing to be detected!

• Placing a box slightly off will lead to higher loss than if we placed no box at all since incorrect placement gives double penalization

• Placing small boxes within big boxes has higher loss due to the same double penalization effect.

• We want boxes to be in the correct place and size

YOLO TRAINING

Intersection over Union: the overlap between predicted and ground truth bounding boxes.

\[\text{IoU} = \frac{\text{Area of Overlap}}{\text{Area of Union}}\]

• Perfect detection → IoU = 1
• No overlap → IoU = 0
• Typically threshold at 0.5 for a “correct” detection

YOLO TRAINING

THE SEGMENTATION PROBLEM

Detection gives us boxes. But sometimes we need pixel-level precision:

• “Which exact pixels belong to the cat?”

This is semantic segmentation — a per-pixel classification task.

THE SEGMENTATION PROBLEM

Create a mapping from an input image (3 channel RGB) to a segmentation mask (1 channel, class labels).

THE SEGMENTATION PROBLEM

CNN pipeline:

• Ingest image (1024 x 1024) → pretrained backbone → spatial compression → GAP (7 x 7 x 512)

Goal: Say which pixels belong to which class

• This isn’t just classification anymore. Need to know both when and where w.r.t. the original pixel space.

• BUT, we’ve destroyed all of the pixel level information at the embedding bottleneck!

THE SEGMENTATION PROBLEM

THE SEGMENTATION PROBLEM

Goal: Say which pixels belong to which class

• CNN tells us what is in this image (dog, cat, bird, Fred Durst, etc.)

• We’ve destroyed all of the pixel level location information at the embedding bottleneck!

• Figure out how to go from embedding and return to pixel space with new labels…

The Naive Approach

Classify each pixel independently using a sliding window CNN.

• For a 224×224 image: ~50,000 separate classifications. Each one runs the full backbone.

The Core Idea: Encode, Then Decode

Encode: compress the image into a low-resolution, high-semantic feature representation (the usual CNN backbone). The bottleneck “knows” what is in the image but has lost where things are.

Decode: expand back to full resolution, translating the semantic knowledge into per-pixel predictions.

The Encoder-Decoder Concept

This is a general principle, not specific to segmentation.

Encoder: complex high-dimensional input → lower-dimensional latent representation capturing the “essence.”

Decoder: latent representation → desired output format.

The encoder-decoder is the nonlinear, learned version of the same compress-then-reconstruct idea.

The Decoder’s Challenge

The encoder is our familiar CNN — conv layers with pooling that shrink spatial dimensions. We know how to go from 224×224 to 7×7.

The decoder needs to reverse this: go from 7×7 back to 224×224.

• How do we “un-pool” and “un-convolve”?

General Note: For semantic segmentation, we want to stop the CNN backbone prior to GAP since GAP destroys all “Where?” information.

Simple Upsampling Options

Nearest neighbor: repeat each pixel in a 2×2 block. Cheap. Blocky.

Bilinear interpolation: weighted average of neighbors. Smoother, but no learnable parameters.

Both are fixed operations — the network can’t learn how to upsample. We want something smarter.

Transposed Convolution: The Idea

Regular convolution: each input pixel contributes to a smaller output via a learned filter.

Transposed convolution: each input pixel contributes to a larger output via the same filter mechanism.

Transposed Convolution: The Idea

Stride 2 (on the output matrix):

\[ \left[\begin{array}{cc}\color{blue}0 & 1 \\2 & 3 \end{array}\right] \circledast^{-1} \left[\begin{array}{cc}\color{blue}0 & \color{blue}1 \\\color{blue}2 & \color{blue}3 \end{array}\right] \]

\[ \left[\begin{array}{cccc}\color{blue}0 & \color{blue}0 & & \\\color{blue}0 & \color{blue}0 & & \\ & & & \\ & & & \end{array}\right] \]

Transposed Convolution: The Idea

Stride 2 (on the output matrix):

\[ \left[\begin{array}{cc} 0 & \color{blue}1 \\2 & 3 \end{array}\right] \circledast^{-1} \left[\begin{array}{cc}\color{blue}0 & \color{blue}1 \\\color{blue}2 & \color{blue}3 \end{array}\right] \]

\[ \left[\begin{array}{cccc}0 & 0 & \color{blue}0 & \color{blue}1 \\0 & 0 & \color{blue}2 & \color{blue}3 \\ & & & \\ & & & \end{array}\right] \]

Transposed Convolution: The Idea

Stride 2 (on the output matrix):

\[ \left[\begin{array}{cc} 0 & 1 \\\color{blue}2 & 3 \end{array}\right] \circledast^{-1} \left[\begin{array}{cc}\color{blue}0 & \color{blue}1 \\\color{blue}2 & \color{blue}3 \end{array}\right] \]

\[ \left[\begin{array}{cccc}0 & 0 & 0 & 1 \\0 & 0 & 2 & 3 \\ \color{blue}0& \color{blue}2 & & \\ \color{blue}4& \color{blue}6& & \end{array}\right] \]

Transposed Convolution: The Idea

Stride 2 (on the output matrix):

\[ \left[\begin{array}{cc} 0 & 1 \\2 & \color{blue}3 \end{array}\right] \circledast^{-1} \left[\begin{array}{cc}\color{blue}0 & \color{blue}1 \\\color{blue}2 & \color{blue}3 \end{array}\right] \]

\[ \left[\begin{array}{cccc}0 & 0 & 0 & 1 \\0 & 0 & 2 & 3 \\ 0& 2 &\color{blue}0 &\color{blue}3 \\ 4& 6&\color{blue}6 &\color{blue}9 \end{array}\right] \]

Transposed Convolution: The Idea

Transposed convolution:

• Passes the input over the filter elementwise

• Preserves the scale in the corresponding block of the output matrix

• Striding is w.r.t. the output matrix not the input matrix

• Results in a larger output than input.

Transposed Convolution: The Key Insight

Transposed convolution is a learnable upsampling.

• Same shared weight concept as before

• Choose the number of upsampling filters per layer.

It’s linear interpolation where the interpolation weights are learned via backpropagation.

• Start with a low-res feature map.

• Produce a higher-res version where the “fill-in” values are learned, not fixed.

• The network learns how to expand spatial resolution in a way that’s useful for the task.

In PyTorch: nn.ConvTranspose2d(in_channels, out_channels, kernel_size, stride=2)

Building the Decoder

Stack transposed convolutions mirroring the encoder: each step doubles spatial resolution and halves channels.

Encoder:  224 → 112 → 56 → 28 → 14 → 7    (spatial shrinks, channels grow)
Decoder:    7 →  14 → 28 → 56 → 112 → 224  (spatial grows, channels shrink)

The final layer has C output channels (one per class). Apply softmax at each pixel → per-pixel class probabilities.

ENCODER-DECODER

ENCODER-DECODER

This looks like a U

• Image goes in
• Downsample for “What?”
• Bottleneck
• Upsample for “Where?”
• Profit

This architecture is called a U-Net.

ENCODER-DECODER

Does this work well?

• Kinda!

• Let’s look at a brain tumor example in the notebook

The Problem with Encode-Then-Decode

A plain encoder-decoder loses too much spatial information at the bottleneck.

The bottleneck (7×7×512) knows what is in the image (“there’s a tumor”) but has lost a lot of where (“which pixels are tumor?”).

• The decoder has to reconstruct spatial detail from a representation that deliberately threw it away.

• It’s like asking my Dad for directions to something - he knows what’s near there (the old tractor supply store near the place where his buddy Ricky got really high and fell asleep in the back of a truck bed full of mulch) but nothing immediately useful for finding where it is.

• I just need the address!!!! I don’t want to look for an article online about where the old store used to be then call Ricky to see if he remembers (spoiler: he doesn’t because he smoked too much…)

U-Net: The Key Insight

The encoder at each resolution level has the spatial information — it just gets lost as we go deeper.

What if the decoder could peek at the encoder’s earlier feature maps?

The U-Net Architecture

Skip Connections: Same Principle, Different Application

ResNet skip connections: help gradients flow through depth. The skip passes information forward through layers.

U-Net skip connections: pass spatial detail across the bottleneck. The encoder’s high-resolution feature maps bypass the information-destroying bottleneck and feed directly into the decoder.

• Both are instances of the same principle: let information bypass a processing bottleneck to prevent useful signal from being lost.

What Each Path Contributes

Encoder features at a given resolution: high spatial detail, low semantic content.

→ “There are edges at these locations.”

Bottleneck features (upsampled): low spatial detail, high semantic content.

→ “There’s a tumor somewhere in this region.”

The U-Net in Detail

Encoder path (our familiar ResNet-18):

[Conv → BN → ReLU] × 2 → MaxPool at each level. Channels: 64 → 64 → 128 → 256 → 512.

Bottleneck: 512 channels at 7×7 (lowest resolution).

Decoder path (learned from scratch):

• TransposedConv (upsample) → Concatenate with encoder features → [Conv → BN → ReLU] × 2.

• Channels: 512 → 256 → 128 → 64 → 64.

Final layer: Conv 1×1 with C output channels → per-pixel class logits.

The U-Net in Detail

The encoder:

• Can and should start with a pre-trained CNN backbone with ImageNet weights!

• No need to reinvent the wheel.

The decoder:

• MUST BE TRAINED FOR OUR PROBLEM

• Set up to have the same feature map sizes as the encoder to enable skip connections

Loss Functions: Binary Cross Entropy

Per-Pixel Cross Entropy:

\[\mathcal{L}_{\text{BCE}} = -\frac{1}{HW}\sum_{i,j} \left[ g_{ij} \log p_{ij} + (1-g_{ij}) \log (1-p_{ij}) \right]\]

where \(p_{ij}\) is predicted tumor probability at pixel \((i,j)\) and \(g_{ij}\) is ground truth (0 or 1).

• Strength: every pixel gets a clear gradient signal. Stable training from the start.

• Weakness: treats each pixel independently — doesn’t care about the shape of the prediction. With class imbalance (only ~5% of pixels are tumor), gradients are dominated by easy background pixels.

Loss Functions: Dice Loss

The Dice coefficient measures overlap between predicted and ground truth masks as a whole:

\[\text{Dice} = \frac{2 \sum_{i,j} p_{ij} \cdot g_{ij}}{\sum_{i,j} p_{ij} + \sum_{i,j} g_{ij}}\]

• Perfect overlap → Dice = 1. No overlap → Dice = 0.
• Dice Loss = 1 − Dice

Loss Functions: Dice Loss

Key property: it’s ratio-based.

• If the tumor is 5% of pixels, Dice weights those pixels proportionally to their importance for the overlap score.

• Naturally handles class imbalance without explicit reweighting.

Weakness: can have noisy gradients early in training when both \(\sum p\) and \(\sum g\) are small.

Connection: Dice and IoU are monotonically related. Dice = 2·IoU / (1 + IoU). Optimizing one generally optimizes the other.

The Combined Loss

In practice, combine both:

\[\mathcal{L} = \alpha \cdot \mathcal{L}_{\text{BCE}} + (1 - \alpha) \cdot \mathcal{L}_{\text{Dice}}\]

• BCE provides stable, per-pixel gradients — the reliable training foundation
• Dice provides the shape-aware, imbalance-resistant objective
• Together: consistently outperforms either loss alone
• This is standard practice in medical segmentation

The Combined Loss

BCE is the log-likelihood (pixel-level data fit).

Dice is a structural prior (the prediction should overlap coherently with the ground truth shape).

• Combining them is the same principle as regularization — balancing data fit against structural quality.

U-Net: Using a Pretrained Encoder

Key practical insight: the encoder is just a CNN backbone. We can use a pretrained ResNet-18 as the encoder.

• Freeze or fine-tune the encoder — same transfer learning principles from last lecture. Only the decoder and skip connections need to be trained from scratch.

• This dramatically reduces the amount of labeled segmentation data needed — the encoder already knows how to extract features.

Skip Connections Are Better

Let’s look at the model with skip connections in the notebook

• Fully pre-trained

• LoRA rank 16 adapters to fine tune the ResNet18 backbone

The Practical Shortcut: segmentation-models-pytorch

In practice, a library handles the encoder-decoder plumbing:

import segmentation_models_pytorch as smp

model = smp.Unet(
    encoder_name="resnet18",
    encoder_weights="imagenet",
    in_channels=3,
    classes=1,
)

Supports 800+ encoder backbones. Automatically builds a matching decoder with skip connections. Exposes model.encoder, model.decoder, model.segmentation_head for differential learning rates.

Instance Segmentation

Semantic segmentation: all tumors get the same label. What if we need to distinguish individual objects?

Instance segmentation: detect each object (bounding box) AND predict a segmentation mask for each one.

Segment Anything Model

Meta’s SAM: trained on 11 million images with 1.1 billion masks.

Learns to find object “blobs” regardless of class label — a general-purpose segmentation foundation model. Can be prompted with points, boxes, or text. Returns candidate masks. Freely available.

The Encoder-Decoder Is a General Architecture

We’ve now seen the encoder-decoder pattern for segmentation.

Encoder: image → compressed latent representation.

Decoder: latent representation → output.

Reconstruction as a Training Objective

What if we trained an encoder-decoder where the target output is the input image itself?

\[\mathcal{L}_{\text{reconstruction}} = \| \mathbf{x} - \text{Decoder}(\text{Encoder}(\mathbf{x})) \|^2\]

The bottleneck is forced to capture the essential information about the image — enough to reconstruct it.

What Lives in the Bottleneck?

The bottleneck latent vector is a compact representation of the image.

• For segmentation: captures “what objects are present and roughly where”
• For reconstruction: captures “everything needed to rebuild this image”

If we could sample from the bottleneck space — generate new latent vectors — and run them through the decoder, we’d get new images.

• But how do we know what latent vectors are “valid”? How do we sample from the bottleneck in a way that produces coherent images?

The Variational Autoencoder (Preview)

Make the bottleneck probabilistic. Instead of encoding to a single vector, encode to a distribution:

\[\text{Encoder}(\mathbf{x}) \rightarrow (\boldsymbol{\mu}, \boldsymbol{\sigma}^2)\]

Sample from this distribution. Decode the sample.

Train with reconstruction loss + regularization keeping the distributions well-behaved (KL divergence — connects directly back to our Bayesian material).

The Roadmap

The encoder-decoder architecture is the foundation for the generative models we’ll cover next:

VAEs → the probabilistic bottleneck. How do we make the latent space structured and sampleable?

Diffusion models (Later) → what if we don’t use an explicit bottleneck, but instead learn to gradually denoise? (The U-Net reappears here — with skip connections — because the U-Net is predicting noise, not generating from a bottleneck.)

The same architectural building blocks — convolutions, skip connections, transposed convolutions — appear in all of them.

What’s Next

The encoder-decoder opens the door to generative models.

Next block: how to generate new images with the Encoder-Decoder architecture

• VAEs and the probabilistic bottleneck

Then sequences via transformers:

• The transformer architectures that power modern LLMs