DATASCI 447 Lecture 11: Bayesian Principles and CNNs - I promise I wasn’t wasting your time a few weeks ago!

Administrative Stuff

Convolutional Neural Networks: Architecture as Prior

Last time: how to train deep networks — gradient flow, batch normalization, regularization.

Today: how to encode domain knowledge about images directly into the network architecture.

Where We Left Off

From L10, we have a complete toolkit for training deep networks:

• He initialization, gradient clipping → prevent explosions
• Batch normalization → keep activations healthy during training
• Weight decay, dropout, early stopping → regularize and generalize

What Flattening Destroys

To feed an image into a fully connected network, we flatten it to a vector.

A 32×32×3 color image becomes a 3,072-dimensional vector.

Pixel (0,0) and pixel (31,31) are now equally “close” in the vector. Every spatial relationship — edges, shapes, neighborhoods — is destroyed.

The Parameter Cost of Ignorance

In a fully connected layer, every hidden unit connects to every pixel.

• We are giving our network the ability (really the directive) to learn mappings of all pixels to every hidden unit!

• The first FC layer of our L10 MLP: 3,072 inputs × 256 hidden units = 786,432 parameters.

The Bayesian Framing

In regression, we choose priors that reflect our beliefs about the data:

Three Beliefs About Images

Before seeing a single training image, we believe:

1. Locality

Useful visual features — edges, textures, corners — are computed from small spatial neighborhoods, not the entire image.

• The color of a dog’s fur is informative for the other fur parts close by, but not for what’s going on in the background or other parts of the dog!

Three Beliefs About Images

Before seeing a single training image, we believe:

2. Translation Equivariance

The same feature can appear anywhere. A vertical edge in the top-left is the same feature as a vertical edge in the bottom-right.

• The snout of a standing dog is the same as a dog snout of a silly dog rolling around on his back! It doesn’t matter where it is - a snout is a snout.

Three Beliefs About Images

Before seeing a single training image, we believe:

3. Hierarchy / Compositionality

Complex features are built from simpler features. Edges → textures → parts → objects.

• A dog’s face is made up of its ears, eyes, and snout, which are themselves made up of simpler features like edges and colors.

From Beliefs to Architecture

Each belief maps to a specific architectural constraint:

Equivariance: The Formal Statement

Let \(T\) be a transformation (e.g., shifting an image 5 pixels to the right).

Invariance: A Different Property

Invariance: A function \(f\) is invariant to \(T\) if:

\[f(T(\mathbf x)) = f(\mathbf x)\]

The output doesn’t change at all when the input is transformed.

• For classification, we want invariance: a cat is a cat regardless of where it appears.

Why This Matters

Equivariance is the precise mathematical statement of the inductive bias/prior that CNNs provide:

• It’s not just “CNNs are good at images.” It’s: “CNNs encode a specific symmetry — translation — that natural images possess.”

• When the symmetry matches the data generating process, generalization follows. This is the same logic as encoding an L2 prior in Ridge regression — but structural rather than parametric.

An Important Caveat

Conv layers are equivariant to translation but NOT to rotation or scale.

• A rotated cat produces a different representation in the hidden space, not a rotated representation.

• The network must learn rotation/scale invariance from data — or we provide it via data augmentation (next lecture).

The Convolution Operation

Now let’s build the layer. Recall from L9: each module has a forward pass that computes an output from an input.

Walking Through 1D Convolution

Before 2D, let’s see this in 1D:

\[[1,2,3,4,5,6] \circledast [1,2] = [5,8,11,14,17]\]

Walking Through 1D Convolution

Before 2D, let’s see this in 1D:

\[[\boxed{\color{red}1,2},3,4,5,6] \circledast [\boxed{\color{red}1,2}] = [\boxed{\color{red}5}, 8, 11,14,17]\] \[(1 \times 1) + (2 \times 2) = 5\]

Walking Through 1D Convolution

Before 2D, let’s see this in 1D:

\[[1,\boxed{\color{red}2,3},4,5,6] \circledast [\boxed{\color{red}1,2}] = [5, \boxed{\color{red}8}, 11,14,17]\] \[(2 \times 1) + (3 \times 2) = 8\]

Walking Through 1D Convolution

Before 2D, let’s see this in 1D:

\[[1,2,\boxed{\color{red}3,4},5,6] \circledast [\boxed{\color{red}1,2}] = [5, 8, \boxed{\color{red}11},14,17]\] \[(3 \times 1) + (4 \times 2) = 11\]

The filter slides across the input. Same weights at every position — that’s weight sharing.

Walking Through 2D Convolution

Now in 2D: a \(3 \times 3\) image with a \(2 \times 2\) filter.

\[ \begin{bmatrix} \boxed{a_{11}} & \boxed{a_{12}} & a_{13} \\ \boxed{a_{21}} & \boxed{a_{22}} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \circledast \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} \]

\[c_{11} = a_{11}w_{11} + a_{12}w_{12} + a_{21}w_{21} + a_{22}w_{22}\]

Walking Through 2D Convolution

\[ \begin{bmatrix} a_{11} & \boxed{a_{12}} & \boxed{a_{13}} \\ a_{21} & \boxed{a_{22}} & \boxed{a_{23}} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \circledast \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} \]

\[c_{12} = a_{12}w_{11} + a_{13}w_{12} + a_{22}w_{21} + a_{23}w_{22}\]

Walking Through 2D Convolution

\[ \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ \boxed{a_{21}} & \boxed{a_{22}} & a_{23} \\ \boxed{a_{31}} & \boxed{a_{32}} & a_{33} \end{bmatrix} \circledast \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} \]

\[c_{21} = a_{21}w_{11} + a_{22}w_{12} + a_{31}w_{21} + a_{32}w_{22}\]

Walking Through 2D Convolution

\[ \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & \boxed{a_{22}} & \boxed{a_{23}} \\ a_{31} & \boxed{a_{32}} & \boxed{a_{33}} \end{bmatrix} \circledast \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} \]

\[c_{22} = a_{22}w_{11} + a_{23}w_{12} + a_{32}w_{21} + a_{33}w_{22}\]

The filter slides over every position. Each output value is a local inner product — template matching at that location.

The Inner Product as Template Matching

The inner product \(\mathbf w^\top \mathbf x\) measures similarity between a weight vector and an input.

• High when they point in the same direction (positive correlation).
• Low when they point in opposite directions (negative correlation).
• Zero when uncorrelated.

The Inner Product as Template Matching

A convolutional filter is a template for a small visual pattern:

• A vertical edge filter has large positive weights on one side, large negative weights on the other
• When the filter slides over a vertical edge in the image, the inner product is large
• When it slides over a flat region, the inner product is near zero

After ReLU: only positive similarity survives. The output tells us where the pattern is present.

What the Filter Finds

The vertical edge filter responds strongly along the vertical edge (bright) and is silent elsewhere. The horizontal edge filter responds along the horizontal edge. Each filter detects one local pattern everywhere in the image.

What the Filter Finds

What the Filter Finds

Weight Sharing Is Why Equivariance Holds

The same filter \(\mathbf W\) slides over every spatial position.

Multiple Filters = Multiple Features

One filter detects one pattern. We want to detect many patterns: vertical edges, horizontal edges, color gradients, corners, blobs…

Multiple Filters = Multiple Features

The layer has learned a vocabulary of local features — each filter is a “word” for a different spatial pattern.

Multi-Channel Input

A color image has 3 input channels (R, G, B).

Each filter must have the same depth as the input:

• A \(3 \times 3\) filter on an RGB image is actually \(3 \times 3 \times 3 = 27\) weights
• The filter computes one inner product across all channels at each position
• When humans look at pictures, we don’t see 3 dims, we just see two. It rarely makes sense to think about convolving over the color channel!

Multi-Channel Input

In general, a conv layer with \(C_{\text{in}}\) input channels and \(C_{\text{out}}\) filters has:

\[C_{\text{out}} \times C_{\text{in}} \times k \times k \quad \text{parameters (+ } C_{\text{out}} \text{ biases)}\]

The Parameter Payoff

Let’s make this concrete. For Pets images resized to 224×224:

Forward/Backward: Same Module Pattern

A conv layer fits the same module pattern from L9:

The Backprop Rule for Convolutions

Recall from L9 that every linear module produces three things in its backward pass:

The Backprop Rule for Convolutions

The structure is identical — the operation changes from matrix multiply to convolution.

• Bias: sum the upstream gradient (over spatial positions instead of just passing through)
• Weights (A 3 x 3 matrix for each output channel): the outer product becomes a convolution of the input with the upstream \(\boldsymbol \delta\)
• Pass-through: multiply by \(\mathbf W\) becomes convolve with the rotated filter (full convolution to recover input size)

The Backprop Rule for Convolutions

Why this matters: The convolution layer is just more multiplies and adds

• Just done in a structured way!

• GPUs FLY through this kind of computation!

But there’s way more to do!

• For a 32 x 32 x 3 flattened image flattened with 256 hidden units: 786,432 multiply-adds

• For a 32 x 32 x 3 image with a 3 x 3 convolutional layer with 256 filters: 7,077,088 multiply-adds

Parameter count decreases but compute time increases!

• Trade RAM for processing power

The Backprop Rule for Convolutions

The chain of equations from L9 is identical:

\[\delta_0 \xrightarrow{\text{conv} \;\times} \delta_1 \xrightarrow{\odot \; \mathbb{I}} \delta_2 \xrightarrow{\text{conv} \;\times} \delta_3 \cdots\]

Only the internal operation of each \(\times\) module has changed. Everything from L10 — BN, gradient flow, regularization — still applies.

A Conv Layer in PyTorch

nn.Conv2d(
    in_channels=3,      # RGB input
    out_channels=64,     # learn 64 different filters
    kernel_size=3,       # each filter is 3×3
    padding=1,           # preserve spatial dimensions
)

model = nn.Sequential(
    nn.Conv2d(3, 64, 3, padding=1),
    nn.BatchNorm2d(64),   # BN for conv layers (per-channel)
    nn.ReLU(),
    nn.Conv2d(64, 128, 3, padding=1),
    nn.BatchNorm2d(128),
    nn.ReLU(),
    # ...
)

Practical Mechanics: Padding and Stride

The next few slides cover mechanical choices that control the shape of outputs.

These matter for getting the code to run, but they don’t fundamentally affect what the network learns. Think of these as plumbing, not architecture.

• As long as the choices are reasonable, performance is largely unchanged. The important decisions are filter size, number of filters, and depth.

Padding

A \(3 \times 3\) filter on a \(32 \times 32\) input produces a \(30 \times 30\) output — we lose 1 pixel on each border.

In general, a \(k \times k\) filter on an \(n \times n\) input gives:

\[(n - k + 1) \times (n - k + 1)\]

Padding

To preserve spatial dimensions, pad the input with zeros:

• Same padding is the norm. Most dimensionality reduction happens in the pooling layers (still to come).

Stride

By default, the filter moves one pixel at a time (stride = 1).

With stride = 2, the filter jumps two pixels → output spatial dimensions halve.

\[H_{\text{out}} = \left\lfloor \frac{H_{\text{in}} + 2p - k}{s} \right\rfloor + 1\]

Pooling: Building the Hierarchy

With padding = ⌊k/2⌋, conv layers preserve spatial dimensions.

But we want to reduce spatial resolution as we go deeper. Two reasons:

1. Computational efficiency — processing 224×224×64 tensors at every layer is expensive.

2. Hierarchy — deeper layers should capture larger-scale patterns. Reducing resolution forces each unit to cover a larger region of the original image.

This is Belief #3 (hierarchy): complex features are compositions of simpler ones. To compose, each layer must “see” a progressively larger region.

Max Pooling

The simplest downsampling: take non-overlapping \(2 \times 2\) windows, keep only the maximum (Max Pool 2, Stride 2):

\[ \begin{bmatrix} \boxed{1} & \boxed{2} & 3 & 4 \\ \boxed{5} & \boxed{6} & 7 & 8 \\ 9 & 10 & 11 & 12 \\ 13 & 14 & 15 & 16 \end{bmatrix} \xrightarrow{\text{MaxPool}(2)} \begin{bmatrix} \boxed{6} & 8 \\ 14 & 16 \end{bmatrix} \]

Remember that each value in the post-convolution + ReLU feature map says “Centered on this pixel, was the feature present?”

Max Pooling

Spatial dimensions halve: \(H \times W \to \frac{H}{2} \times \frac{W}{2}\). No learnable parameters.

What pooling says: “I care that this feature was detected in this \(2 \times 2\) region, but not exactly where within the region.”

This is local spatial invariance — robustness to small shifts.

The Funnel: Trading Space for Semantics

The Effective Receptive Field

Stacking \(3 \times 3\) convolutions with pooling gives each unit a progressively larger “view” of the original image:

Deeper networks capture longer-range spatial dependencies — not by having larger filters, but by composing many small ones.

Each \(3 \times 3\) filter is cheap (9 parameters). The expressive power comes from depth.

Reading the Funnel

Each pooling step halves the spatial dimensions while the number of channels (features) grows:

After Block 1 (112×112, 64 channels): Simple local features — edges at various orientations, color gradients. Each feature is detected in a small 3×3 patch.

Flatten and Classify

After the conv/pool stages: a small spatial grid of rich feature maps.

For example: \(7 \times 7 \times 512 = 25{,}088\) values.

Flatten and Classify

The conv layers did the representation learning — extracting meaningful hierarchical features from raw pixels.

The FC layers do the classification — mapping features to class probabilities.

The Bottleneck Problem

After the final conv block, we have a rich feature tensor — say \(7 \times 7 \times 512 = 25{,}088\) values.

If we flatten this directly and connect to a FC classifier with 512 hidden units:

\[25{,}088 \times 512 = \textbf{12.8 million parameters}\]

just in that one layer. This is more parameters than the entire conv backbone that built the features!

• Incredibly wasteful

Global Average Pooling

Key idea: Each of those 512 feature maps already represents “how much of feature \(k\) is present in this image.”

We don’t need 25,088 numbers to summarize that — we need one number per feature map.

Global Average Pooling

The classifier is now just \(\text{nn.Linear}(512, K)\) — 512 \(\times\) \(K\) parameters. For 10 classes, that’s 5,120 vs. 12.8 million.

• Good evidence that this drastically reduced training time and minimally impacts accuracy!

nn.AdaptiveAvgPool2d(1)  # output is 1×1 regardless of input spatial size

VGG: The Standard Template

VGG (Visual Geometry Group - Simonyan & Zisserman, 2014) is the cleanest instantiation of the three beliefs.

One design rule: always use \(3 \times 3\) filters, double the channels each time you halve the spatial dimensions.

VGG Architecture

Why Always 3×3?

Two stacked \(3 \times 3\) conv layers have the same effective receptive field as one \(5 \times 5\) layer.

Three \(3 \times 3\) layers match one \(7 \times 7\).

VGG Block in PyTorch

def vgg_block(in_ch, out_ch, n_convs):
    """VGG-style block: N × (Conv3×3 → BN → ReLU) → MaxPool"""
    layers = []
    for i in range(n_convs):
        layers.append(nn.Conv2d(
            in_ch if i == 0 else out_ch, out_ch,
            kernel_size=3, padding=1, bias=False
        ))
        layers.append(nn.BatchNorm2d(out_ch))
        layers.append(nn.ReLU(inplace=True))
    layers.append(nn.MaxPool2d(kernel_size=2, stride=2))
    return nn.Sequential(*layers)

Same module pattern. Same loss.backward(). Same optimizer.

Drop-in replacement for the FC layers from L10.

Why the Improvement?

The CNN doesn’t have more capacity than the MLP — the parameter counts are comparable.

Why the Improvement?

This is bias-variance in action:

• More bias (structural constraints → can’t represent functions with long-range non-local interactions)
• Dramatically less variance (fewer effectively free parameters → less overfitting with limited data)

What’s Still Missing

VGG with 13 conv layers works well. But:

Next lecture:

• Skip connections (ResNets) solve the depth problem — enabling 50, 100, 150+ layer networks
• Data augmentation encodes additional invariances the architecture doesn’t provide
• Representation learning: reframing CNNs not as classifiers but as feature transforms — which sets up transfer learning in L13