DATASCI 447 Lecture 8: Deep Learning

Administrative Stuff

Where We Left Off

• Single layer neural networks with ReLU activations are incredibly powerful
• The hidden layer learned a “representation” that made classification easy
• Width \(M\) controls complexity: from linear (\(M=0\)) to universal approximation (\(M \to \infty\))

But: We didn’t fully understand what the hidden layer actually captures

Today: Use MNIST to see exactly what’s happening inside the network

The Key Insight

The hidden layer transforms raw pixels into a space where classes are linearly separable

PCA: Optimizes for reconstruction (minimize \(\|x - \hat{x}\|^2\))

NN hidden layer: Optimizes for the task (minimize classification loss)

\[\phi_{\text{PCA}}(x) \neq \phi_{\text{NN}}(x)\]

The NN learns a representation that’s useful for prediction, not reconstruction

Today: Let’s visualize this directly with a tractable example

The Neural Network as Matrix Operations

Before we dive into examples, let’s establish the matrix formulation that will be essential for understanding backpropagation - the main method which allows us to optimize the complicated neural network loss functions.

Single-Layer Network: The Full Equation

For a single observation \(x\):

\[\hat{y} = \underset{(K \times 1)}{\sigma_{\text{out}}}(\underset{(K \times M)}{W_{\text{out}}} \cdot \underset{(M \times 1)}{\varphi}(\underset{(M \times P)}{W_1} \underset{(P \times 1)}{x} + \underset{(M \times 1)}{b_1}) + \underset{(K \times 1)}{b_{\text{out}}})\]

Where:

• \(K\) is the number of outputs per input (1 for regular regression, 1 or 2 for binary logistic regression, number of categories for multinomial, number of pixels if we’re generating an image)

• \(M\) is the number of hidden units in the hidden layer

• \(P\) is the number of input features

This equation is gnarly, but we can break it into pieces to show it as a chained equation

The Chain

Step 1: Linear transformation \[\underset{(M \times 1)}{z_1} = \underset{(M \times P)}{W_1} \underset{(P \times 1)}{x} + \underset{(M \times 1)}{b_1}\]

• The unaltered scores in PCA lingo

The Chain

Step 2: Nonlinear activation (ReLU) \[\underset{(M \times 1)}{h} = \underset{(M \times 1)}{\text{ReLU}}(\underset{(M \times 1)}{z_1}) = \underset{(M \times 1)}{\max}(0, \underset{(M \times 1)}{z_1})\]

• The activation function/nonlinearity is always applied elementwise!

The Chain

Step 3: Output linear transformation \[\underset{(K \times 1)}{z_{\text{out}}} = \underset{(K \times M)}{W_{\text{out}}} \underset{(M \times 1)}{h} + \underset{(K \times 1)}{b_{\text{out}}}\]

• The output layer that applies the regression coefficients and creates the linear predictor

The Chain

Step 4: Output activation \[\underset{(K \times 1)}{\hat{y}} = \underset{(K \times 1)}{\sigma}(\underset{(K \times 1)}{z_{\text{out}}})\]

• \(\sigma\) is the sigmoid function for binary classification

• \(\sigma\) is the softmax function for multi-class classification

• \(\sigma\) is the identity function for regression or any other method that wants predictions in \(\mathbb R\)

The Chain

Step 5: Compute Loss

For a training set with \(N\) observations:

\[ \mathcal L = \frac{1}{N} \sum_{i=1}^N \mathcal L(\hat{y}_i, y_i) \]

Our goal is to minimize this function!

The Nested Function View

We can write this as function composition:

\[\hat{y} = f_{\text{out}}(f_{\text{hidden}}(x))\]

Where:

• \(f_{\text{hidden}}: \mathbb{R}^P \to \mathbb{R}^M\) defined by \(f_{\text{hidden}}(x) = \text{ReLU}(W_1 x + b_1)\)
• \(f_{\text{out}}: \mathbb{R}^M \to \mathbb{R}^K\) defined by \(f_{\text{out}}(h) = \sigma(W_{\text{out}} h + b_{\text{out}})\)

The network is a composition:

\[\hat{y} = (f_{\text{out}} \circ f_{\text{hidden}})(x)\]

This view will be essential for computing gradients (backpropagation)!

MNIST — A Tractable Problem

Loading MNIST dataset...

MNIST: 70,000 handwritten digits (28×28 grayscale images)

• 784 pixels per image
• 10 digit classes (0-9)
• A classic benchmark for neural networks

Why MNIST instead of faces?

• Simpler: 784 dimensions (vs. 2,304 or 4,096)
• Fewer classes to start: we’ll use just 2
• FCNNs work well (~98% accuracy on full problem)
• We can actually visualize what the network learns!

MNIST — A Tractable Problem

The 4 vs. 9 Problem: The Hardest Pair

Why 4 vs. 9? This is the hardest pair for linear classifiers!

Why Is 4 vs. 9 Hard?

Both digits have: - A vertical stroke on the right side - Structure in the upper portion

The key difference: - 9 has a closed loop at the top - 4 has intersecting lines and an open top or an angular top

The Challenge: Loop vs. Angle

Detecting “loop” vs. “intersection” requires understanding shape, not just “where are the pixels?”

A linear classifier asks: “Is pixel (i, j) bright?”

But we need: “Do the bright pixels form a closed curve?”

How Well Does Logistic Regression Do?

Training Logistic Regression...

Logistic Regression Results:
  Training accuracy: 98.0%
  Test accuracy:     96.6%

~96% accuracy — pretty good!

This leaves a little room for improvement. Can a neural network do better?

The Errors Are Interpretable

Some are genuinely ambiguous cases!

But some seem weird…

What Does Logistic Regression “See”?

Logistic regression learns: “If pixel here is bright → probably a 9 (or 4)”

But it cannot learn: “If pixels form a closed curve → 9”

For Our MNIST 4 vs 9 Problem

A simple 4 hidden unit network

\[ \underset{(1 \times 1)}{\hat{y}} = \underset{(1 \times 1)}{\sigma}\left(\underset{(1 \times 4)}{W_{\text{out}}} \cdot \underset{(4 \times 1)}{\text{ReLU}}\left(\underset{(4 \times 784)}{W_1} \underset{(784 \times 1)}{\mathbf x} + \underset{(4 \times 1)}{b_1}\right) + \underset{(1 \times 1)}{b_{\text{out}}}\right) \]

Dimensions:

• \(P = 784\) (pixels in a 28×28 image)
• \(M = 4\) (hidden units)
• \(K = 1\) (classes: digit 9?)

Parameter Count

Parameters to learn:

With 3,145 parameters, we can classify handwritten digits better?

Compare to:

• Logistic regression on raw pixels: \(785\) parameters
• But logistic regression can’t capture nonlinear patterns…

Training the Neural Network

Neural Network (4 hidden units) Results:
  Training accuracy: 99.5%
  Test accuracy:     98.3%

Improvement over Logistic Regression:
  Test accuracy: 96.6% → 98.3% (+1.7%)

With only 4 hidden units, the neural network improves on logistic regression!

Key question: How does a 4-unit hidden layer outperform a linear model?

Answer: It learns better features. Let’s see what they are!

Part 4: What Hidden Units Learn

We trained a neural network with 4 hidden units that beats logistic regression.

The key question: What do those 4 units actually compute?

Recall the hidden layer: \[\underset{(4 \times 1)}{h} = \underset{(4 \times 1)}{\text{ReLU}}\left(\underset{(4 \times 784)}{W_1} \underset{(784 \times 1)}{x} + \underset{(4 \times 1)}{b_1}\right)\]

Each row of \(W_1\) is a vector of 784 weights — one per pixel.

The Weight Matrix

\[W_1 = \begin{bmatrix} — w_1^T — \\ — w_2^T — \\ — w_3^T — \\ — w_4^T — \end{bmatrix} \in \mathbb{R}^{4 \times 784}\]

Each \(w_k \in \mathbb{R}^{784}\) can be reshaped to 28×28 — a “template”!

Visualizing the 4 Hidden Unit Templates

Blue regions: “I want bright pixels here” (positive weights)

Red regions: “I want dark pixels here” (negative weights)

Each unit has learned a different pattern detector!

How a Hidden Unit Computes Its Activation

For hidden unit \(k\), the activation for input \(x\) is:

\[h_k = \underset{\text{scalar}}{\text{ReLU}}(\underset{1 \times 784}{w_k^T} \underset{784 \times 1}{x} + \underset{\text{scalar}}{b_k}) = \text{ReLU}\left(\sum_{p=1}^{784} w_{kp} \cdot x_p + b_k\right)\]

This is a template match:

• Compute weighted sum of pixel values
• Large positive sum → high activation (after ReLU)
• Negative sum → zero activation (ReLU clips to 0)

The unit “fires” when the input looks like its template!

Hidden Unit 1: What Does It Detect?

Hidden Unit 2: What Does It Detect?

Hidden Unit 3: What Does It Detect?

Hidden Unit 4: What Does It Detect?

The Output Layer: Combining Evidence

Now we have 4 hidden activations: \(h = [h_1, h_2, h_3, h_4]^T\)

The output layer computes: \[\underset{(1 \times 1)}{z_{\text{out}}} = \underset{(1 \times 4)}{W_{\text{out}}} \underset{(4 \times 1)}{h} + \underset{(1 \times 1)}{b_{\text{out}}}\]

• How much does each hidden unit contribute to the “9” class?

Visualizing Logistic Weights: Which Units Vote for Which Class?

Positive weight: “When this unit is active, vote for 9”

Negative weight: “When this unit is active, vote against 9”

Hidden Units and Their Votes

The network has learned:

• Different feature detectors (patterns related to loops, strokes, etc.)
• How to combine them for classification (voting weights)

The Hidden Space

Each image \(x\) is transformed into a 4D point \(h = [h_1, h_2, h_3, h_4]^T\)

In this 4D space, the two classes are more separable than in the original 784D pixel space!

Linear Separability in Hidden Space

The hidden layer’s job: create a space where a hyperplane can separate the classes!

t-SNE Comparison: Raw Pixels vs. Hidden Layer

This Is Representation Learning

The network learns a transformation \(\phi(x) = \text{ReLU}(W_1 x + b_1)\)

This representation is optimized for:

• Classification loss (cross-entropy)
• The specific task (distinguishing 4 from 9)

NOT optimized for:

• Reconstruction (like PCA)
• Hand-crafted features (like HOG, SIFT)

Key insight: The features are learned from data, for the task

This is why neural networks can discover features that humans might not think of!

Why Isn’t One Layer Enough?

We got good results with 4 hidden units on 4 vs. 9.

But each unit is trying to do everything at once:

• Detect a complete pattern
• Map it directly to class membership

What if the pattern is more complex?

Hierarchical Logic for Digits

What makes a 9 a 9?

The Conceptual Hierarchy

The Single Layer Problem

A single layer must learn: “complete 9 pattern” vs “complete 4 pattern” directly from pixels

The Multi-Layer Solution

• Layer 1: Detect simple features (curves, lines, edges)
• Layer 2: Detect feature combinations (loops, intersections)
• Output: Combine evidence for final decision

The same “curve detector” helps for many digits!

The Compositionality Hypothesis

Real-world patterns are compositional:

• Complex patterns are built from simpler patterns
• The same simple patterns appear in many complex patterns

Examples:

Depth exploits compositionality; width must memorize.

Deep Networks — Matrix Formulation

Let’s formalize the multi-layer idea.

Two hidden layers:

\[h_1 = \underset{M_1 \times 1}{\text{ReLU}}(\underset{M_1 \times P}{W_1} \underset{P \times 1}{\mathbf x} + \underset{M_1 \times 1}{b_1})\]

\[h_2 = \underset{M_2 \times 1}{\text{ReLU}}(\underset{M_2 \times M_1}{W_2} \underset{M_1 \times 1}{h_1} + \underset{M_2 \times 1}{b_2})\]

\[\hat{y} = \underset{K \times 1}{\sigma}(\underset{K \times M_2}{W_{\text{out}}} \underset{M_2 \times 1}{h_2} + \underset{K \times 1}{b_{\text{out}}})\]

The Nested Function View

Two layers: \[\hat{y} = f_{\text{out}}(f_2(f_1(x)))\]

General \(D\)-layer network: \[\hat{y} = f_{\text{out}} \circ f_D \circ f_{D-1} \circ \cdots \circ f_1(x)\]

Each \(f_l\) is: \[f_l(h_{l-1}) = \text{ReLU}(W_l h_{l-1} + b_l)\]

Why this matters: The chain rule lets us differentiate through this composition!

\[\frac{\partial \mathcal{L}}{\partial W_1} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h_2} \cdot \frac{\partial h_2}{\partial h_1} \cdot \frac{\partial h_1}{\partial W_1}\]

Our Two-Layer MNIST Network

Architecture: \(784 \to 4 \to 3 \to 2\)

\[h_1 = \underset{4 \times 1}{\text{ReLU}}(\underset{4 \times 784}{W_1} \underset{784 \times 1}{x} + \underset{4 \times 1}{b_1})\]

\[h_2 = \underset{3 \times 1}{\text{ReLU}}(\underset{3 \times 4}{W_2} \underset{4 \times 1}{h_1} + \underset{3 \times 1}{b_2})\]

\[\hat{y} = \underset{1 \times 1}{\text{softmax}}(\underset{1 \times 3}{W_{\text{out}}} \underset{3 \times 1}{h_2} + \underset{1 \times 1}{b_{\text{out}}})\]

Total parameters: \((4 \times 784 + 4) + (3 \times 4 + 3) + (1 \times 3 + 1) = 3,159\)

Training the Deep Network

Shallow Network (784 → 4 → 2):
  Train accuracy: 99.5%
  Test accuracy:  98.3%

Deep Network (784 → 4 → 3 → 2):
  Train accuracy: 100.0%
  Test accuracy:  98.3%

Let’s look at what each layer learns!

What Does Layer 1 Detect?

Layer 1 detects low-level features: edges, curves, stroke patterns

These are similar to the shallow network — detecting basic visual elements.

What Does Layer 1 Detect?

Layer 1 Hidden Space

Some separation, but not complete. That’s okay!

Layer 1’s job isn’t to classify — it’s to extract useful features for Layer 2.

The Second Layer: Combining Layer 1 Features

\[h_2 = \underset{3 \times 1}{\text{ReLU}}(\underset{3 \times 4}{W_2} \underset{4 \times 1}{h_1} + \underset{3 \times 1}{b_2})\]

Layer 2 Hidden Space

Layer 2 Hidden Space

Layer 2 with Decision Boundary

The Deep Learning Recipe

\[\underset{784}{x} \xrightarrow{W_1, b_1} \underset{4}{h_1} \xrightarrow{W_2, b_2} \underset{3}{h_2} \xrightarrow{W_{\text{out}}, b_{\text{out}}} \underset{2}{\hat{y}}\]

• Layer 1: Raw pixels → low-level features (edges, curves, strokes)
• Layer 2: Low-level features → high-level features (loops, intersections)
• Output: High-level features → class prediction

Each layer learns a representation of the previous representation!

Why Early Layers Transfer

We trained on digits 4 vs. 9 only.

Key question: What if we trained on ALL 10 digits?

Would Layer 1 learn completely different features?

Full MNIST (10 Classes)

Deep Network (784 → 10 → 10 → 2) on 4 vs 9:
  Training accuracy: 100.0%
  Test accuracy:     98.1%

Deep Network (784 → 10 → 10 → 10) on Full MNIST:
  Training accuracy: 96.2%
  Test accuracy:     93.4%

Comparing Layer 1 Weights

Observation: Similar features! Edges, curves, stroke detectors appear in both networks.

Feature Reuse

Early layers learn general features that apply to many classes:

• Edge detectors help distinguish 0 from 1, but also 4 from 9, and 3 from 5
• Curve detectors help for 0, 3, 6, 8, 9…
• Stroke detectors help for 1, 4, 7…
• Loop detectors help for 0, 6, 8, 9…

Later layers learn task-specific combinations:

• “Loop at top + vertical stroke = 9”
• “Horizontal bar + vertical stroke = 4”
• These combinations depend on which classes you’re distinguishing

Transfer Learning

If early layers learn general features:

1. Train a deep network on a large dataset (e.g., ImageNet)
2. Freeze early layers (they learned general features)
3. Retrain later layers for your specific task

This is called transfer learning — we’ll revisit it later!

The Hierarchical Structure Enables Transfer

\[\underbrace{W_1, W_2, W_3}_{\text{general features (freeze)}} \quad \underbrace{W_4, W_{\text{out}}}_{\text{task-specific (retrain)}}\]

Why it works:

• Early layers: edges, textures (useful for ALL images)
• Middle layers: parts, shapes (useful for many tasks)
• Late layers: task-specific combinations

The hierarchical structure of deep networks makes this possible!

Why Depth Works — Theory and Evidence

We’ve seen that deep networks learn hierarchical features.

But is there a computational advantage to depth?

Question: For the same number of hidden units, does depth help?

The Wiggly Problem

\[P(y=1|x) \propto \sin(x_1^2 - x_2^2)\]

A complex, nonlinear decision boundary — perfect for testing depth!

Fixed Units, Varying Depth

Same total hidden units \(M\), different architectures:

Question: For the same “budget” of hidden units, does depth help?

Training Accuracy vs. Hidden Units

Test Accuracy vs. Hidden Units

Training Time vs. Hidden Units

Test Accuracy vs. Training Time

Why Does This Happen?

The wiggly boundary \(\sin(x_1^2 - x_2^2) = 0\) has compositional structure:

1. Compute \(x_1^2\) and \(x_2^2\) (quadratic features)
2. Compute the difference \(x_1^2 - x_2^2\)
3. Apply periodic structure (sine-like regions)

A deep network can learn this hierarchy:

• Layer 1: Learn quadratic-ish features
• Layer 2: Learn combinations
  
• Layer 3: Learn decision regions

A shallow network must learn the complete pattern directly!

The Intuition: Folding Space

Each ReLU unit creates a fold in the input space (a hyperplane where the function “bends”).

More folds = more complex functions!

Depth Multiplies Folds: 1D Illustration

Width adds regions: \(M\) units → \(M+1\) regions (1D)

Depth multiplies regions: \(D\) layers → regions grow exponentially in \(D\)!

Visualizing Folds in 2D: Single Layer

Adding width (M) creates more folds, but they’re all parallel hyperplanes.

Visualizing Folds in 2D: Adding Depth

Same 100 units, but deeper networks create more complex decision boundaries!

Linear Regions: The Theoretical Result

For a ReLU network with:

• Input dimension \(P\)
• \(D\) layers
  
• \(M\) units per layer

The maximum number of linear regions is:

\[ \text{total regions} = \mathcal{O}\left(\binom{M}{P}^{P(D-1)} M^P\right) \]

What This Formula Tells Us

\[ \text{total regions} = \mathcal{O}\left(\binom{M}{P}^{P(D-1)} M^P\right) \]

Depth 1: Regions \(\sim M^P\) (polynomial in width)

Depth D: Regions \(\sim M^{PD}\) (exponential in depth!)

But There’s a Catch…

Deeper networks are more expressive, but:

• Intense Non-Convexity — loss surfaces are complex
• Harder to train — gradients must flow through many layers
• Vanishing gradients — signal gets weaker as it propagates back
• Exploding gradients — signal can also blow up

Problem 1: Non-Convexity

Recall: For linear regression and logistic regression, the loss function is convex.

• One global minimum
• Gradient descent always finds it

Neural networks are different. Even a single-layer network has a non-convex loss!

A Simple Example: Why NNs Are Non-Convex

Consider a tiny problem:

• 2 observations: \(x_1 = 1, x_2 = -1\)
• Targets: \(y_1 = 1, y_2 = -1\)
• Network: 1 input → 2 hidden units (ReLU) → 1 output
• Loss: MSE

Network equations:

\[h_k = \text{ReLU}(w_k x + b_k) = \max(0, w_k x + b_k), \quad k = 1, 2\]

\[\hat{y} = \beta_1 h_1 + \beta_2 h_2 + b_{\text{out}}\]

Non-Convexity

Two different parameter settings can achieve the same low loss:

Setting A: \(w_1 = 1, w_2 = -1, b_1 = b_2 = 0, \beta_1 = 1, \beta_2 = -1, b_{\text{out}} = 0\)

For observation 1 (\(x_1 = 1, y_1 = 1\)): \[h_1 = \text{ReLU}(1 \cdot 1 + 0) = 1, \quad h_2 = \text{ReLU}(-1 \cdot 1 + 0) = 0\] \[\hat{y}_1 = 1 \cdot 1 + (-1) \cdot 0 + 0 = 1 \quad \checkmark\]

For observation 2 (\(x_2 = -1, y_2 = -1\)): \[h_1 = \text{ReLU}(1 \cdot (-1) + 0) = 0, \quad h_2 = \text{ReLU}(-1 \cdot (-1) + 0) = 1\] \[\hat{y}_2 = 1 \cdot 0 + (-1) \cdot 1 + 0 = -1 \quad \checkmark\]

\[\mathcal{L}(A) = \frac{1}{2}[(1-1)^2 + (-1-(-1))^2] = 0\]

Non-Convexity

Setting B: \(w_1 = -1, w_2 = 1, b_1 = b_2 = 0, \beta_1 = -1, \beta_2 = 1, b_{\text{out}} = 0\)

For observation 1 (\(x_1 = 1, y_1 = 1\)): \[h_1 = \text{ReLU}(-1 \cdot 1 + 0) = 0, \quad h_2 = \text{ReLU}(1 \cdot 1 + 0) = 1\] \[\hat{y}_1 = (-1) \cdot 0 + 1 \cdot 1 + 0 = 1 \quad \checkmark\]

For observation 2 (\(x_2 = -1, y_2 = -1\)): \[h_1 = \text{ReLU}(-1 \cdot (-1) + 0) = 1, \quad h_2 = \text{ReLU}(1 \cdot (-1) + 0) = 0\] \[\hat{y}_2 = (-1) \cdot 1 + 1 \cdot 0 + 0 = -1 \quad \checkmark\]

\[\mathcal{L}(B) = \frac{1}{2}[(1-1)^2 + (-1-(-1))^2] = 0\]

Both achieve perfect fit! (Settings A and B are equivalent, just with units swapped)

Non-Convexity

Now consider the average: \(w_1 = 0, w_2 = 0, \beta_1 = 0, \beta_2 = 0\)

For observation 1 (\(x_1 = 1, y_1 = 1\)): \[h_1 = \text{ReLU}(0 \cdot 1 + 0) = 0, \quad h_2 = \text{ReLU}(0 \cdot 1 + 0) = 0\] \[\hat{y}_1 = 0 \cdot 0 + 0 \cdot 0 + 0 = 0\]

For observation 2 (\(x_2 = -1, y_2 = -1\)): \[h_1 = \text{ReLU}(0 \cdot (-1) + 0) = 0, \quad h_2 = \text{ReLU}(0 \cdot (-1) + 0) = 0\] \[\hat{y}_2 = 0 \cdot 0 + 0 \cdot 0 + 0 = 0\]

\[\mathcal{L}(\text{avg}) = \frac{1}{2}[(0-1)^2 + (0-(-1))^2] = \frac{1}{2}[1 + 1] = 1\]

Non-Convexity

Convexity would require: \(\mathcal{L}(\text{avg}) \leq \frac{1}{2}\mathcal{L}(A) + \frac{1}{2}\mathcal{L}(B)\)

But here: \(1 > 0\)! The loss function is non-convex.

This happens because ReLU creates a non-convex response surface.

Depth Multiplies the Non-Convexity Problem

A single hidden layer already has multiple minima.

Adding depth makes it exponentially worse:

Problem 2: Gradient Flow Through Compositions

Recall our deep network:

\[\hat{y} = f_{\text{out}}(f_D(f_{D-1}(\cdots f_1(x))))\]

To update \(W_1\) (the first layer), we need:

\[\frac{\partial \mathcal{L}}{\partial W_1} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h_D} \cdot \frac{\partial h_D}{\partial h_{D-1}} \cdots \frac{\partial h_2}{\partial h_1} \cdot \frac{\partial h_1}{\partial W_1}\]

The gradient must flow backward through ALL \(D\) layers!

Vanishing and Exploding Gradients

Simple example: Suppose each layer just multiplies by weight \(w\):

\[h^{(D)} = w \cdot w \cdot \ldots \cdot w \cdot x = w^D \cdot x\]

The gradient with respect to \(w\):

\[\frac{\partial h^{(D)}}{\partial w} \propto w^{D-1}\]

The Historical Bottleneck

1980s-1990s: “Deep networks don’t work”

• Backpropagation was known (Rumelhart et al., 1986)
• But training networks deeper than 2-3 layers failed
• Vanishing gradients killed the signal to early layers

The Historical Bottleneck

The breakthrough came in stages:

Preview: Skip Connections

What if instead of: \[h_{l+1} = f_l(h_l)\]

We used: \[h_{l+1} = f_l(h_l) + h_l\]

This is called a skip connection or residual connection.

Preview: Skip Connections

Why Skip Connections Help

\[h_{l+1} = f_l(h_l) + h_l\]

Gradient flow:

\[\frac{\partial h_{l+1}}{\partial h_l} = \frac{\partial f_l(h_l)}{\partial h_l} + I\]

Even if \(\frac{\partial f_l}{\partial h_l}\) vanishes, the gradient can still flow through the \(+I\) term!

The layer only needs to learn the “residual” — what to change about \(h_l\).

We’ll see this in detail with ResNets and Transformers.

The Modern Deep Learning Recipe

To train deep networks successfully:

1. ReLU activations — Avoids vanishing gradients from saturating activations (unlike sigmoid/tanh)

2. Careful initialization — Start with reasonable gradient magnitudes (Xavier, He initialization)

3. Skip connections — Provide gradient highways through the network

4. Normalization layers — BatchNorm, LayerNorm stabilize activations

5. Adaptive optimizers — Adam, etc. adjust learning rates per-parameter

Details next lecture: Backpropagation!

Next Time: Backpropagation

We have: \(\hat{y} = f_{\text{out}}(f_2(f_1(x)))\)

We need: \(\frac{\partial \mathcal{L}}{\partial W_1}, \frac{\partial \mathcal{L}}{\partial W_2}, \frac{\partial \mathcal{L}}{\partial W_{\text{out}}}\)

Next lecture:

• How to compute gradients efficiently (computational graphs)
• The backpropagation algorithm
• Vanishing gradients in detail
• Modern solutions: initialization, normalization, skip connections

Come prepared: How do you differentiate \(f(g(x))\) with respect to parameters in \(g\)?