Last lecture we hit a wall: the VQ-VAE produces a grid of codebook indices, but generating those indices from scratch requires modeling their sequential dependencies.
Language is a sequence of words. Speech is a sequence of audio frames. Video is a sequence of images. Music, DNA, stock prices — any data where order matters.
The tools we build over the next few lectures apply to all of them.
“FEEDFORWARD” NEURAL NETWORKS
Thus far, we have only seen deep learning architectures where information flows upward through the model
Inputs go through layers and yield hidden values
Hidden values to more hidden values
Feedforward in the sense that the only direct influence of a layer is to its child
In computational graphs, there is no complicated structure that creates feedback between units
A single input leads to two (or more) independent outputs
Regression \(\rightarrow\) (Mean, Variance)
Bounding Box \(\rightarrow\) (Class, Location)
THE FIVE PATTERNS
Image classification (One image to One Class)
THE FIVE PATTERNS
Image captioning (One image to a sequence of words)
THE FIVE PATTERNS
Video classification (Many images/frames to one video class)
Sentiment analysis (Sequence of words to one sentiment label)
THE FIVE PATTERNS
Not necessarily equal number of inputs and outputs
Machine translation (Sequence of words in English to Seq in French)
Sentence continuation (First part of sentence to second part)
Time series (Stock prices for last 10 years to prices for the next week)
THE FIVE PATTERNS
Equal number of outputs and inputs
Per-frame video classification (Seq of Images to Seq of Words)
Per-sentence sentiment analysis (Seq of sentences to Class)
Next-token prediction (Seq of codebook indices to Seq of next indices)
FEEDFORWARD NETWORKS CAN’T DO THIS
A feedforward network maps one fixed-size input to one fixed-size output. No memory. No sense of order.
We could concatenate the entire sequence into one long vector and feed it through an MLP. But sequences have variable length, and the model would need position-specific weights for every possible length.
No weight sharing across time. No generalization to new sequence lengths.
If the order of the input matters, we need an architecture that preserves it.
RECURRENT NEURAL NETWORKS
Recurrent Neural Networks (RNNs) are the backbone of sequence modeling
Inputs or outputs make sense temporally
Knowing what came before should inform what we predict now
Adds knowledge to the NN architecture
Units need to talk backwards in time — all of the previous values need to influence what I guess next!
If the order of the input matters, then we should preserve that information!
THE RECURRENCE FORMULA
We can process a sequence of vectors, \(\mathbf{x}\), by applying a recurrence formula at every time step:
\(\mathbf{h}_{t-1}\) is the old state (some vector of numbers)
\(\mathbf{x}_t\) is the input at time \(t\) (some vector - think an embedding of length 32)
\(f_w()\) is a function of some parameters, \(\mathbf{W}\) (some collection of weight matrices)
\(\mathbf{h}_t\) is the new state (some vector)
THE HIDDEN STATE AS LATENT REPRESENTATION
The hidden state vector contains latent information about the current state of the model
For language models, it could tell us about the gender of pronouns in the sentence
Or the passiveness of the tone
For image tokens, it could encode the spatial context seen so far
This information should persist throughout the sequence of predictions!
Create memory from one prediction to the next (passive tone should persist).
The hidden state is a latent vector — a compressed representation of the sequence, just like \(\mathbf{z}\) in the VAE is a compressed representation of an image.
Can be pre-trained on large corpora — more on this when we get to transformers
THE MANY ARCHITECTURES
Many-to-One:
\(\mathbf{W}_{hy}\) is only used for the single output
THE MANY ARCHITECTURES
One-to-Many:
SEQ2SEQ: THE ENCODER-DECODER RNN
Pre-GPT, the seq2seq model was a main approach for question/answer and translation models
SEQ2SEQ: THE ENCODER-DECODER RNN
This is another example of an encoder/decoder architecture:
Use weights to determine the mapping of the inputs to a final hidden state
Use that hidden state as the beginning of a new mapping to the outcome
“The cat sat on the mat” \(\to\) encoder \(\to\)\(\mathbf{h}_6\)\(\to\) decoder \(\to\) “That durn cat is sittin on the rug”
The entire input sentence — every word, every grammatical relationship — must be compressed into a single vector \(\mathbf{h}_T\).
Sound familiar? This is the same information bottleneck as the VAE’s flat vector.
WHY TANH?
WHY TANH?
Tanh is used here over ReLU for a number of reasons:
RNNs aren’t going to promote sparsity since there isn’t a meaningful way to map hidden units to input features
RNNs have a really serious vanishing gradient problem (more to come)
Unlike CNNs, RNNs have a really serious exploding gradient problem!
Squish values to \([-1, 1]\) — prevents the hidden state from growing without bound since \(\mathbf{W}_{hh}\) is applied multiplicatively at every step
But tanh has a problem we’re about to see.
TRAINING: BACKPROPAGATION THROUGH TIME
Given \(\mathbf{X}\) and \(\mathbf{Y}\), we need to learn \(\mathbf{W}_{hh}\), \(\mathbf{W}_{xh}\), and \(\mathbf{W}_{hy}\)
The hidden states \(\{\mathbf{h}_1, \ldots, \mathbf{h}_T\}\) are just a function of these values
The influence of \(\mathbf{x}_1\) on \(\mathbf{y}_3\) goes through \(\mathbf{W}_{hh}^2\). On \(\mathbf{y}_{100}\) it goes through \(\mathbf{W}_{hh}^{99}\).
Small times small equals smaller!
THE SEQ2SEQ BOTTLENECK
This is also why Seq2Seq struggles with long sequences.
The entire input sentence — every word, every grammatical relationship — must be compressed into a single vector \(\mathbf{h}_T\).
For a 50-word paragraph, \(\mathbf{h}_T\) is the same size as for a 5-word sentence. Information loss is inevitable.
This is the same information bottleneck as the VAE’s flat vector. And the fix will be analogous — instead of compressing everything into one vector, let the decoder access the full sequence of encoder states.
That fix is called attention — next lecture.
THE SEQ2SEQ BOTTLENECK
THE LSTM: A CLEVER FIX
A clever solution to the vanishing gradient problem — The Long Short-Term Memory model (LSTM)
There’s a lot going on here. Let’s take it step-by-step.
LSTM: THE CELL STATE
LSTMs alter the RNN model by having two concurrently running hidden units — the hidden state and the cell state
The cell state runs straight down the chain with only some minor linear interactions
The gradient is easy to compute and doesn’t change exponentially!
LSTM: GATES CONTROL INFORMATION FLOW
The hidden state can add or remove information from the cell state through gates
A sigmoid layer (which maps a value between 0 and 1) determines when and how much the hidden state should influence the next one!
LSTMs have 3 of these gates to protect and control the cell state.
LSTM: THE FORGET GATE
The forget gate controls how we lose information from the previous memory
In a language model, maybe the hidden state encodes the gender of the subject
The subject changes and uses different pronouns?
It’s not worthwhile to remember the gender state in this case.
The sigmoid activation is applied element-wise — some parts of the vector might be more useful to remember than others.
LSTM: THE INPUT GATE
Two weight matrices — the input weights and the candidate gate
A candidate hidden vector \(\tilde{\mathbf{C}}_t\) is created using the standard update with a tanh activation
The input gate applies a value between 0 and 1 to each element — how much of the new information should be added?
LSTM: THE CELL STATE UPDATE
The new cell state is a linear combination of the old cell state and the candidate, weighted by the forget gate and the input gate:
No separate cell state. The update gate interpolates between old and new directly. Simpler, faster, often works just as well.
GRU: THE SIMPLIFIED VERSION
THE FUNDAMENTAL LIMITATION
LSTMs and GRUs mitigate vanishing gradients but don’t solve the fundamental problem:
The hidden state is still a fixed-size vector. At step 1000, the entire sequence history must fit in the same 256 or 512 dimensions that step 2 used for just one input.
Information gets overwritten. The Seq2Seq bottleneck persists — long sequences lose early information regardless of gating.
And training is sequential.\(\mathbf{h}_t\) depends on \(\mathbf{h}_{t-1}\) — you can’t parallelize across time steps. Long sequences are slow.
WHAT WE NEED
Two problems to solve:
The bottleneck: the decoder should access all encoder states, not just the final summary
The parallelism: we shouldn’t have to process the sequence one step at a time
Both problems are solved by attention — a mechanism that lets any position directly attend to any other position, bypassing the sequential hidden state chain entirely.
Next lecture: how attention works, and why it replaces recurrence.
SUMMARY
Sequences are everywhere — language, audio, video, image tokens
RNNs add memory through a recurrent hidden state with shared weights across time
The hidden state is a latent representation — the same concept as \(\mathbf{z}\) in a VAE
Backpropagation through time creates vanishing/exploding gradients — the product of \(T\) matrices shrinks or grows exponentially
LSTMs add a cell state with additive updates — gradient highways through time, same principle as ResNets
GRUs simplify to two gates and no cell state — similar performance, fewer parameters
The fundamental limitation: fixed-size hidden state, sequential processing, information bottleneck
Next: attention solves both the bottleneck and the parallelism problem
