DATASCI 447 Lecture 11: Bayesian Principles and CNNs - I promise I wasn’t wasting your time a few weeks ago!
Kevin McAlister
February 17, 2026
Administrative Stuff
Convolutional Neural Networks: Architecture as Prior
Last time: how to train deep networks — gradient flow, batch normalization, regularization.
Today: how to encode domain knowledge about images directly into the network architecture.
The central idea: architecture is a prior. When the prior matches the data generating process, generalization follows.
Where We Left Off
From L10, we have a complete toolkit for training deep networks:
- He initialization, gradient clipping → prevent explosions
- Batch normalization → keep activations healthy during training
- Weight decay, dropout, early stopping → regularize and generalize
We applied all of this to CIFAR-10 (32×32 color images, 10 classes) using a 20-layer fully connected MLP and reached ~55% test accuracy.
That’s better than random (10%), but humans get ~94% on CIFAR-10.
Where’s the gap?
What Flattening Destroys
To feed an image into a fully connected network, we flatten it to a vector.
A 32×32×3 color image becomes a 3,072-dimensional vector.
Pixel (0,0) and pixel (31,31) are now equally “close” in the vector. Every spatial relationship — edges, shapes, neighborhoods — is destroyed.
The Parameter Cost of Ignorance
In a fully connected layer, every hidden unit connects to every pixel.
We are giving our network the ability (really the directive) to learn mappings of all pixels to every hidden unit!
The first FC layer of our L10 MLP: 3,072 inputs × 256 hidden units = 786,432 parameters.
Most of these connections link spatially distant pixels with no meaningful relationship. We’re estimating hundreds of thousands of parameters for relationships that don’t exist.
And it gets worse with resolution. For a 224×224 color image (standard for modern vision):
\[150{,}528 \times 256 = \textbf{38.5 million parameters}\]
in the first layer alone. For a dataset with a few thousand images, this is hopeless.
The Bayesian Framing
In regression, we choose priors that reflect our beliefs about the data:
| Prior | Belief | Method |
|---|---|---|
| Uniform | “I have no beliefs about which coefficients matter” | OLS |
| L2 (Gaussian) | “I believe most coefficients are small” | Ridge |
| L1 (Laplace) | “I believe most coefficients are zero” | LASSO |
A fully connected layer is a uniform prior over spatial structure: any pixel can influence any hidden unit, with no preference for local relationships.
For images, we have strong prior knowledge before seeing any data.
Can we encode it directly into the network architecture?
Three Beliefs About Images
Before seeing a single training image, we believe:
1. Locality
Useful visual features — edges, textures, corners — are computed from small spatial neighborhoods, not the entire image.
- The color of a dog’s fur is informative for the other fur parts close by, but not for what’s going on in the background or other parts of the dog!
Three Beliefs About Images
Before seeing a single training image, we believe:
2. Translation Equivariance
The same feature can appear anywhere. A vertical edge in the top-left is the same feature as a vertical edge in the bottom-right.
- The snout of a standing dog is the same as a dog snout of a silly dog rolling around on his back! It doesn’t matter where it is - a snout is a snout.
Three Beliefs About Images
Before seeing a single training image, we believe:
3. Hierarchy / Compositionality
Complex features are built from simpler features. Edges → textures → parts → objects.
- A dog’s face is made up of its ears, eyes, and snout, which are themselves made up of simpler features like edges and colors.
These aren’t engineering convenience. They follow from how the physical world produces images: light interacts locally, objects move, and parts compose into wholes.
From Beliefs to Architecture
Each belief maps to a specific architectural constraint:
| Belief | Constraint | Effect |
|---|---|---|
| Locality | Local receptive fields | Each unit sees a small patch, not all pixels |
| Translation equivariance | Weight sharing | Same filter applied at every position |
| Hierarchy | Stacking layers | Each layer builds on the previous one |
A convolutional layer encodes all three.
But before we build one, let’s be precise about what “translation equivariance” actually means. This is the theoretical heart of why CNNs work.
Equivariance: The Formal Statement
Let \(T\) be a transformation (e.g., shifting an image 5 pixels to the right).
Equivariance: A function \(f\) is equivariant to \(T\) if:
\[f(T(\mathbf x)) = T(f(\mathbf x))\]
Transforming the input, then applying \(f\), gives the same result as applying \(f\), then transforming the output.
Example: If we detect a cat in an image then shift the cat image 5 pixels right, then we should still detect the cat using the exact same set of rules
- The “what” is detected regardless of “where.”
Invariance: A Different Property
Invariance: A function \(f\) is invariant to \(T\) if:
\[f(T(\mathbf x)) = f(\mathbf x)\]
The output doesn’t change at all when the input is transformed.
- For classification, we want invariance: a cat is a cat regardless of where it appears.
But we don’t want invariance everywhere — early layers should track where features are (equivariance), and only the final classification should discard location (invariance).
The CNN pipeline:
\[\underbrace{\text{Conv layers}}_{\text{equivariant}} \to \underbrace{\text{Pooling}}_{\text{adds local invariance}} \to \underbrace{\text{Global pool + FC}}_{\text{fully invariant}}\]
The network gradually trades spatial precision for semantic abstraction.
Why This Matters
Equivariance is the precise mathematical statement of the inductive bias/prior that CNNs provide:
It’s not just “CNNs are good at images.” It’s: “CNNs encode a specific symmetry — translation — that natural images possess.”
When the symmetry matches the data generating process, generalization follows. This is the same logic as encoding an L2 prior in Ridge regression — but structural rather than parametric.
An Important Caveat
Conv layers are equivariant to translation but NOT to rotation or scale.
A rotated cat produces a different representation in the hidden space, not a rotated representation.
The network must learn rotation/scale invariance from data — or we provide it via data augmentation (next lecture).
The Convolution Operation
Now let’s build the layer. Recall from L9: each module has a forward pass that computes an output from an input.
In a fully connected layer, the forward pass is:
\[z_k = \mathbf w_k^\top \mathbf x + b_k\]
An inner product with the entire input. This is where the “uniform prior” comes from.
In a convolutional layer, the forward pass is:
\[z_{i,j} = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} W_{m,n} \cdot x_{i+m,\, j+n} + b\]
An inner product with a small local patch of the input. Same operation, just restricted to a \(k \times k\) neighborhood.
Walking Through 1D Convolution
Before 2D, let’s see this in 1D:
\[[1,2,3,4,5,6] \circledast [1,2] = [5,8,11,14,17]\]
Walking Through 1D Convolution
Before 2D, let’s see this in 1D:
\[[\boxed{\color{red}1,2},3,4,5,6] \circledast [\boxed{\color{red}1,2}] = [\boxed{\color{red}5}, 8, 11,14,17]\] \[(1 \times 1) + (2 \times 2) = 5\]
Walking Through 1D Convolution
Before 2D, let’s see this in 1D:
\[[1,\boxed{\color{red}2,3},4,5,6] \circledast [\boxed{\color{red}1,2}] = [5, \boxed{\color{red}8}, 11,14,17]\] \[(2 \times 1) + (3 \times 2) = 8\]
Walking Through 1D Convolution
Before 2D, let’s see this in 1D:
\[[1,2,\boxed{\color{red}3,4},5,6] \circledast [\boxed{\color{red}1,2}] = [5, 8, \boxed{\color{red}11},14,17]\] \[(3 \times 1) + (4 \times 2) = 11\]
The filter slides across the input. Same weights at every position — that’s weight sharing.
Walking Through 2D Convolution
Now in 2D: a \(3 \times 3\) image with a \(2 \times 2\) filter.
\[ \begin{bmatrix} \boxed{a_{11}} & \boxed{a_{12}} & a_{13} \\ \boxed{a_{21}} & \boxed{a_{22}} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \circledast \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} \]
\[c_{11} = a_{11}w_{11} + a_{12}w_{12} + a_{21}w_{21} + a_{22}w_{22}\]
Walking Through 2D Convolution
\[ \begin{bmatrix} a_{11} & \boxed{a_{12}} & \boxed{a_{13}} \\ a_{21} & \boxed{a_{22}} & \boxed{a_{23}} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \circledast \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} \]
\[c_{12} = a_{12}w_{11} + a_{13}w_{12} + a_{22}w_{21} + a_{23}w_{22}\]
Walking Through 2D Convolution
\[ \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ \boxed{a_{21}} & \boxed{a_{22}} & a_{23} \\ \boxed{a_{31}} & \boxed{a_{32}} & a_{33} \end{bmatrix} \circledast \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} \]
\[c_{21} = a_{21}w_{11} + a_{22}w_{12} + a_{31}w_{21} + a_{32}w_{22}\]
Walking Through 2D Convolution
\[ \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & \boxed{a_{22}} & \boxed{a_{23}} \\ a_{31} & \boxed{a_{32}} & \boxed{a_{33}} \end{bmatrix} \circledast \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} \]
\[c_{22} = a_{22}w_{11} + a_{23}w_{12} + a_{32}w_{21} + a_{33}w_{22}\]
The filter slides over every position. Each output value is a local inner product — template matching at that location.
The Inner Product as Template Matching
The inner product \(\mathbf w^\top \mathbf x\) measures similarity between a weight vector and an input.
- High when they point in the same direction (positive correlation).
- Low when they point in opposite directions (negative correlation).
- Zero when uncorrelated.
For zero-centered values, this is the covariance — high when \(\mathbf w\) and \(\mathbf x\) point in the same direction.
The Inner Product as Template Matching
A convolutional filter is a template for a small visual pattern:
- A vertical edge filter has large positive weights on one side, large negative weights on the other
- When the filter slides over a vertical edge in the image, the inner product is large
- When it slides over a flat region, the inner product is near zero
After ReLU: only positive similarity survives. The output tells us where the pattern is present.
What the Filter Finds
The vertical edge filter responds strongly along the vertical edge (bright) and is silent elsewhere. The horizontal edge filter responds along the horizontal edge. Each filter detects one local pattern everywhere in the image.