DATASCI 447 Lecture 8: Deep Learning

Administrative Stuff

Where We Left Off

• Single layer neural networks with ReLU activations are incredibly powerful
• The hidden layer learned a “representation” that made classification easy
• Width \(M\) controls complexity: from linear (\(M=0\)) to universal approximation (\(M \to \infty\))

But: We didn’t fully understand what the hidden layer actually captures

Today: Use MNIST to see exactly what’s happening inside the network

The Key Insight

The hidden layer transforms raw pixels into a space where classes are linearly separable

PCA: Optimizes for reconstruction (minimize \(\|x - \hat{x}\|^2\))

NN hidden layer: Optimizes for the task (minimize classification loss)

\[\phi_{\text{PCA}}(x) \neq \phi_{\text{NN}}(x)\]

The NN learns a representation that’s useful for prediction, not reconstruction

Today: Let’s visualize this directly with a tractable example

The Neural Network as Matrix Operations

Before we dive into examples, let’s establish the matrix formulation that will be essential for understanding backpropagation - the main method which allows us to optimize the complicated neural network loss functions.

Single-Layer Network: The Full Equation

For a single observation \(x\):

\[\hat{y} = \underset{(K \times 1)}{\sigma_{\text{out}}}(\underset{(K \times M)}{W_{\text{out}}} \cdot \underset{(M \times 1)}{\varphi}(\underset{(M \times P)}{W_1} \underset{(P \times 1)}{x} + \underset{(M \times 1)}{b_1}) + \underset{(K \times 1)}{b_{\text{out}}})\]

Where:

• \(K\) is the number of outputs per input (1 for regular regression, 1 or 2 for binary logistic regression, number of categories for multinomial, number of pixels if we’re generating an image)

• \(M\) is the number of hidden units in the hidden layer

• \(P\) is the number of input features

This equation is gnarly, but we can break it into pieces to show it as a chained equation

The Chain

Step 1: Linear transformation \[\underset{(M \times 1)}{z_1} = \underset{(M \times P)}{W_1} \underset{(P \times 1)}{x} + \underset{(M \times 1)}{b_1}\]

• The unaltered scores in PCA lingo

The Chain

Step 2: Nonlinear activation (ReLU) \[\underset{(M \times 1)}{h} = \underset{(M \times 1)}{\text{ReLU}}(\underset{(M \times 1)}{z_1}) = \underset{(M \times 1)}{\max}(0, \underset{(M \times 1)}{z_1})\]

• The activation function/nonlinearity is always applied elementwise!

The Chain

Step 3: Output linear transformation \[\underset{(K \times 1)}{z_{\text{out}}} = \underset{(K \times M)}{W_{\text{out}}} \underset{(M \times 1)}{h} + \underset{(K \times 1)}{b_{\text{out}}}\]

• The output layer that applies the regression coefficients and creates the linear predictor

The Chain

Step 4: Output activation \[\underset{(K \times 1)}{\hat{y}} = \underset{(K \times 1)}{\sigma}(\underset{(K \times 1)}{z_{\text{out}}})\]

• \(\sigma\) is the sigmoid function for binary classification

• \(\sigma\) is the softmax function for multi-class classification

• \(\sigma\) is the identity function for regression or any other method that wants predictions in \(\mathbb R\)

The Chain

Step 5: Compute Loss

For a training set with \(N\) observations:

\[ \mathcal L = \frac{1}{N} \sum_{i=1}^N \mathcal L(\hat{y}_i, y_i) \]

Our goal is to minimize this function!

The Nested Function View

We can write this as function composition:

\[\hat{y} = f_{\text{out}}(f_{\text{hidden}}(x))\]

Where:

• \(f_{\text{hidden}}: \mathbb{R}^P \to \mathbb{R}^M\) defined by \(f_{\text{hidden}}(x) = \text{ReLU}(W_1 x + b_1)\)
• \(f_{\text{out}}: \mathbb{R}^M \to \mathbb{R}^K\) defined by \(f_{\text{out}}(h) = \sigma(W_{\text{out}} h + b_{\text{out}})\)

The network is a composition:

\[\hat{y} = (f_{\text{out}} \circ f_{\text{hidden}})(x)\]

This view will be essential for computing gradients (backpropagation)!

MNIST — A Tractable Problem

Loading MNIST dataset...

MNIST: 70,000 handwritten digits (28×28 grayscale images)

• 784 pixels per image
• 10 digit classes (0-9)
• A classic benchmark for neural networks

Why MNIST instead of faces?

• Simpler: 784 dimensions (vs. 2,304 or 4,096)
• Fewer classes to start: we’ll use just 2
• FCNNs work well (~98% accuracy on full problem)
• We can actually visualize what the network learns!

MNIST — A Tractable Problem

The 4 vs. 9 Problem: The Hardest Pair

Why 4 vs. 9? This is the hardest pair for linear classifiers!

Why Is 4 vs. 9 Hard?

Both digits have: - A vertical stroke on the right side - Structure in the upper portion

The key difference: - 9 has a closed loop at the top - 4 has intersecting lines and an open top or an angular top

The Challenge: Loop vs. Angle

Detecting “loop” vs. “intersection” requires understanding shape, not just “where are the pixels?”

A linear classifier asks: “Is pixel (i, j) bright?”

But we need: “Do the bright pixels form a closed curve?”

How Well Does Logistic Regression Do?

Training Logistic Regression...

Logistic Regression Results:
  Training accuracy: 98.0%
  Test accuracy:     96.6%

~96% accuracy — pretty good!

This leaves a little room for improvement. Can a neural network do better?

The Errors Are Interpretable

Some are genuinely ambiguous cases!

But some seem weird…

What Does Logistic Regression “See”?

Logistic regression learns: “If pixel here is bright → probably a 9 (or 4)”

But it cannot learn: “If pixels form a closed curve → 9”

For Our MNIST 4 vs 9 Problem

A simple 4 hidden unit network

\[ \underset{(1 \times 1)}{\hat{y}} = \underset{(1 \times 1)}{\sigma}\left(\underset{(1 \times 4)}{W_{\text{out}}} \cdot \underset{(4 \times 1)}{\text{ReLU}}\left(\underset{(4 \times 784)}{W_1} \underset{(784 \times 1)}{\mathbf x} + \underset{(4 \times 1)}{b_1}\right) + \underset{(1 \times 1)}{b_{\text{out}}}\right) \]

Dimensions:

• \(P = 784\) (pixels in a 28×28 image)
• \(M = 4\) (hidden units)
• \(K = 1\) (classes: digit 9?)

Parameter Count

Parameters to learn:

With 3,145 parameters, we can classify handwritten digits better?

Compare to:

• Logistic regression on raw pixels: \(785\) parameters
• But logistic regression can’t capture nonlinear patterns…

Training the Neural Network

Neural Network (4 hidden units) Results:
  Training accuracy: 99.5%
  Test accuracy:     98.3%

Improvement over Logistic Regression:
  Test accuracy: 96.6% → 98.3% (+1.7%)

With only 4 hidden units, the neural network improves on logistic regression!

Key question: How does a 4-unit hidden layer outperform a linear model?

Answer: It learns better features. Let’s see what they are!

Part 4: What Hidden Units Learn

We trained a neural network with 4 hidden units that beats logistic regression.

The key question: What do those 4 units actually compute?

Recall the hidden layer: \[\underset{(4 \times 1)}{h} = \underset{(4 \times 1)}{\text{ReLU}}\left(\underset{(4 \times 784)}{W_1} \underset{(784 \times 1)}{x} + \underset{(4 \times 1)}{b_1}\right)\]

Each row of \(W_1\) is a vector of 784 weights — one per pixel.

The Weight Matrix

\[W_1 = \begin{bmatrix} — w_1^T — \\ — w_2^T — \\ — w_3^T — \\ — w_4^T — \end{bmatrix} \in \mathbb{R}^{4 \times 784}\]

Each \(w_k \in \mathbb{R}^{784}\) can be reshaped to 28×28 — a “template”!

Visualizing the 4 Hidden Unit Templates

Blue regions: “I want bright pixels here” (positive weights)

Red regions: “I want dark pixels here” (negative weights)

Each unit has learned a different pattern detector!

How a Hidden Unit Computes Its Activation

For hidden unit \(k\), the activation for input \(x\) is:

\[h_k = \underset{\text{scalar}}{\text{ReLU}}(\underset{1 \times 784}{w_k^T} \underset{784 \times 1}{x} + \underset{\text{scalar}}{b_k}) = \text{ReLU}\left(\sum_{p=1}^{784} w_{kp} \cdot x_p + b_k\right)\]

This is a template match:

• Compute weighted sum of pixel values
• Large positive sum → high activation (after ReLU)
• Negative sum → zero activation (ReLU clips to 0)

The unit “fires” when the input looks like its template!

Hidden Unit 1: What Does It Detect?

Hidden Unit 2: What Does It Detect?

Hidden Unit 3: What Does It Detect?

Hidden Unit 4: What Does It Detect?

The Output Layer: Combining Evidence

Now we have 4 hidden activations: \(h = [h_1, h_2, h_3, h_4]^T\)

The output layer computes: \[\underset{(1 \times 1)}{z_{\text{out}}} = \underset{(1 \times 4)}{W_{\text{out}}} \underset{(4 \times 1)}{h} + \underset{(1 \times 1)}{b_{\text{out}}}\]

• How much does each hidden unit contribute to the “9” class?

Visualizing Logistic Weights: Which Units Vote for Which Class?

Positive weight: “When this unit is active, vote for 9”

Negative weight: “When this unit is active, vote against 9”

Hidden Units and Their Votes

The network has learned:

• Different feature detectors (patterns related to loops, strokes, etc.)
• How to combine them for classification (voting weights)

The Hidden Space

Each image \(x\) is transformed into a 4D point \(h = [h_1, h_2, h_3, h_4]^T\)

In this 4D space, the two classes are more separable than in the original 784D pixel space!

Linear Separability in Hidden Space