DATASCI 447 Lecture 22: Attention

Kevin McAlister

April 2, 2026

Administrative Stuff

RECURRENT NEURAL NETWORKS

Recurrent Neural Networks (RNNs) are the backbone of sequence modeling

Inputs or outputs make sense temporally
Knowing what came before or what comes after should inform what we predict at a current point in time

Adds knowledge to the NN architecture

Units need to talk backwards in time — all of the previous values need to influence what I guess next!

If the order of the input matters, then we should preserve that information!

RECURRENT NEURAL NETWORKS

We can process a sequence of vectors, \(\mathbf{x}\), by applying a recurrence formula at every time step:

\[\mathbf{h}_t = f_w(\mathbf{h}_{t-1}, \mathbf{x}_t)\]

\(\mathbf{h}_{t-1}\) is the old state (some vector of numbers)
\(\mathbf{x}_t\) is the input state at time \(t\) (some vector)
\(f_w()\) is a function of some parameters, \(\mathbf{W}\) (some collection of vectors)
\(\mathbf{h}_t\) is the new state (some vector)

RECURRENT NEURAL NETWORKS

The hidden state vector contains latent information about the current state of the model

For language models, it could tell us about the gender of pronouns in the sentence
Or the passiveness of the tone

This information should persist throughout the sequence of predictions!

Create memory from one prediction to the next.

RECURRENT NEURAL NETWORKS

The vanilla update:

\[\underset{(m \times 1)}{\mathbf{h}_t} = \text{tanh}\left(\underset{(m \times m)}{\mathbf{W}_{hh}} \underset{(m \times 1)}{\mathbf{h}_{t-1}} + \underset{(m \times P)}{\mathbf{W}_{xh}} \underset{(P \times 1)}{\mathbf{x}_t} + \underset{(m \times 1)}{\mathbf{b}_h}\right)\]

\(\mathbf{h}_t\) and \(\mathbf{h}_{t-1}\) are hidden vectors of size \(m\)
\(\mathbf{W}_{hh}\) is a \(m \times m\) matrix of weights that controls the mapping of one hidden state vector to the next
\(\mathbf{W}_{xh}\) is a \(m \times P\) matrix of weights that controls the mapping of the input to the hidden state
\(\mathbf{b}_h\) is a vector of bias terms
\(\text{tanh}()\) is an activation function

RECURRENT NEURAL NETWORKS

The RNN is said to have infinite memory

What it saw before will always play a part (albeit with diminishing influence) in what it predicts next!
Recurrence!

SEQ2SEQ

Today, we’re going to look at the most popular RNN framework — Seq2Seq

SEQ2SEQ

Any sequence of inputs to any sequence of outputs

Machine translation (French to English)
Character Addition
General Question and Answer!

Seq2Seq is a generic approach to the sequence-to-sequence prediction problem!

SEQ2SEQ

SEQ2SEQ: WHAT THE DECODER USES

For each output, we look at:

The previous decoder state
The previous output
The final encoder state

Problem: The input sequence is bottlenecked through the final encoder state

Say a 512 vector that describes all of the language we previously saw
What if the length of the input was 1000?
Consider the aligned case — “we” \(\rightarrow\) “estamos”
- Our encoder is likely only loosely related to that input at the end of the input sequence!

THE BOTTLENECK AT SCALE

For large language models, Seq2Seq would need to summarize all of the inputs that it saw in the training stage in terms of a single vector of length 512

Large for small problems, but quite small for the entirety of the English language
Try to imagine all of the books you’ve ever read
Imagine all of that literature compressed to a single 512 vector of numerical representations
Not really a good strategy…

SEQ2SEQ WITH ATTENTION

Attention is a mechanism that allows the decoder to leverage all parts of the original inputs

Really the corresponding hidden states

At different steps of the decoder, let the model “focus” on different parts of the input!

The encoder still looks at the previous hidden state, but can also look at all previous hidden states seen in the encoder step