DATASCI 447 Lecture 22: Attention

Administrative Stuff

RECURRENT NEURAL NETWORKS

Recurrent Neural Networks (RNNs) are the backbone of sequence modeling

• Inputs or outputs make sense temporally
• Knowing what came before or what comes after should inform what we predict at a current point in time

Adds knowledge to the NN architecture

• Units need to talk backwards in time — all of the previous values need to influence what I guess next!

RECURRENT NEURAL NETWORKS

We can process a sequence of vectors, \(\mathbf{x}\), by applying a recurrence formula at every time step:

\[\mathbf{h}_t = f_w(\mathbf{h}_{t-1}, \mathbf{x}_t)\]

• \(\mathbf{h}_{t-1}\) is the old state (some vector of numbers)
• \(\mathbf{x}_t\) is the input state at time \(t\) (some vector)
• \(f_w()\) is a function of some parameters, \(\mathbf{W}\) (some collection of vectors)
• \(\mathbf{h}_t\) is the new state (some vector)

RECURRENT NEURAL NETWORKS

The hidden state vector contains latent information about the current state of the model

• For language models, it could tell us about the gender of pronouns in the sentence
• Or the passiveness of the tone

This information should persist throughout the sequence of predictions!

• Create memory from one prediction to the next.

RECURRENT NEURAL NETWORKS

The vanilla update:

\[\underset{(m \times 1)}{\mathbf{h}_t} = \text{tanh}\left(\underset{(m \times m)}{\mathbf{W}_{hh}} \underset{(m \times 1)}{\mathbf{h}_{t-1}} + \underset{(m \times P)}{\mathbf{W}_{xh}} \underset{(P \times 1)}{\mathbf{x}_t} + \underset{(m \times 1)}{\mathbf{b}_h}\right)\]

• \(\mathbf{h}_t\) and \(\mathbf{h}_{t-1}\) are hidden vectors of size \(m\)
• \(\mathbf{W}_{hh}\) is a \(m \times m\) matrix of weights that controls the mapping of one hidden state vector to the next
• \(\mathbf{W}_{xh}\) is a \(m \times P\) matrix of weights that controls the mapping of the input to the hidden state
• \(\mathbf{b}_h\) is a vector of bias terms
• \(\text{tanh}()\) is an activation function

RECURRENT NEURAL NETWORKS

RECURRENT NEURAL NETWORKS

The RNN is said to have infinite memory

• What it saw before will always play a part (albeit with diminishing influence) in what it predicts next!
• Recurrence!

SEQ2SEQ

Today, we’re going to look at the most popular RNN framework — Seq2Seq

SEQ2SEQ

Any sequence of inputs to any sequence of outputs

• Machine translation (French to English)
• Character Addition
• General Question and Answer!

Seq2Seq is a generic approach to the sequence-to-sequence prediction problem!

SEQ2SEQ

SEQ2SEQ

SEQ2SEQ

SEQ2SEQ

SEQ2SEQ

SEQ2SEQ: WHAT THE DECODER USES

For each output, we look at:

• The previous decoder state
• The previous output
• The final encoder state

THE BOTTLENECK AT SCALE

For large language models, Seq2Seq would need to summarize all of the inputs that it saw in the training stage in terms of a single vector of length 512

• Large for small problems, but quite small for the entirety of the English language
• Try to imagine all of the books you’ve ever read
• Imagine all of that literature compressed to a single 512 vector of numerical representations
• Not really a good strategy…

SEQ2SEQ WITH ATTENTION

Attention is a mechanism that allows the decoder to leverage all parts of the original inputs

• Really the corresponding hidden states

At different steps of the decoder, let the model “focus” on different parts of the input!

• The encoder still looks at the previous hidden state, but can also look at all previous hidden states seen in the encoder step

SEQ2SEQ WITH ATTENTION

SEQ2SEQ WITH ATTENTION

SEQ2SEQ WITH ATTENTION

SEQ2SEQ WITH ATTENTION

SEQ2SEQ WITH ATTENTION

SEQ2SEQ WITH ATTENTION

SEQ2SEQ WITH ATTENTION

SEQ2SEQ WITH ATTENTION

SEQ2SEQ WITH ATTENTION

ATTENTION IN ACTION: TRANSLATION

Input: “The agreement on the European Economic Area was signed in August 1992”

Output: “L’accord sur la zone economique europeenne a ete signe en aout 1992.”

ATTENTION IN ACTION: REORDERING

Input: “The agreement on the European Economic Area was signed in August 1992”

Output: “L’accord sur la zone economique europeenne a ete signe en aout 1992.”

The attention pattern roughly follows the diagonal — but not exactly. French and English don’t have the same word order. Attention handles reordering naturally.

AN IMPORTANT OBSERVATION

The decoder doesn’t use the fact that the encoder hidden states form an ordered sequence!

• It just treats them as an unordered set and looks for similarities

Order is baked into the hidden states via the encoder

• No need to double bake in order!

ATTENTION SCORES

How do we compute the attention weights?

\[e_{ti} = f(\mathbf{s}_{t-1}, \mathbf{h}_i)\]

\[a_{ti} = \text{softmax}(e_{ti})\]

• The attention scores should measure the level of similarity between the current context of the decoder and the original input states
• Give more weight if the current context should leverage a specific part of the input
• At time \(t\) of the decoder, create a weight for each state of the encoder

ATTENTION SCORES

A quick note:

Without loss, we can assume that the hidden state vectors of the encoder \(\mathbf{h}_i\) and the decoder \(\mathbf{s}_t\) are normalized to have norm 1

• These are just vectors, so we can always force them to have a specific size
• No change in direction, just change in norm

Additionally, lets just assume that the length of each \(\mathbf{h}_i\) and \(\mathbf{s}_t\) are the same!

• Need not be, but much easier computationally

ATTENTION SCORES

Important Review:

An inner product is equivalent to computing the cosine similarity between the two vectors:

\[ \mathbf x^T \mathbf y = \|\mathbf x\| \|\mathbf y\| \cos(\theta) \]

where \(\theta\) is the angle between the two vectors.

ATTENTION SCORES

\[\mathbf{h}_i \in \mathbb{R}^m \quad ; \quad \mathbf{s}_{t-1} \in \mathbb{R}^m\]

Learnable Scaled Dot Product:

\[e_{ti} = \frac{\mathbf{h}_i^T \mathbf{W}_g \mathbf{s}_{t-1}}{\sqrt{m}}\]

• Find the inner product of the two vectors and divide by the length to scale
• Learn a \(m \times m\) matrix that controls the mapping!

ATTENTION SCORES

Putting this all together, we get a model for each output of the decoder!

\[\mathbf{y}_t = f(\mathbf{s}_t) \quad ; \quad \mathbf{s}_t = f(\mathbf{y}_{t-1}, \mathbf{c}_t)\]

\[\mathbf{c}_t = \sum_{i=1}^{T_e} a_{ti} \mathbf{h}_i \quad ; \quad a_{ti} = \text{softmax}(e_{ti})\]

\[e_{ti} = \frac{\mathbf{h}_i^T \mathbf{W}_g \mathbf{s}_{t-1}}{\sqrt{m}}\]

where \(\mathbf{h}_i\) is learned via a RNN setup!

• Place weights where needed to map from one set of attention weighted encoder to a decoder output
• A recurrent weighted scheme for producing contextual seq2seq models
• Fully differentiable, so just let autodiff handle it!

ATTENTION SCORES

This basis admits a more generic approach that will be really important

• Called QKV specification

We’re going to switch to this language

• A little confusing at first, but important since this is the language for this space of deep learning!

QUERY / KEY / VALUE

Attention Layers will require us to change our terminology a little

• Query: vector from which the attention is looking. “What am I looking for?” In the seq2seq model, this is the decoder hidden state \(\mathbf{s}_{t-1}\)

• Key: vector at which the query looks to compute weights. “What do I represent?” In seq2seq, these are the encoder hidden states \(\mathbf{h}_i\)

• Value: the vector to be weighted by the similarity between the query and the key. “What information do I carry?” Also the encoder hidden states.

WHY DO WE NEED WEIGHT MATRICES?

In our Seq2Seq attention, we computed dot products between decoder and encoder hidden states.

\[e_{ti} = \frac{\mathbf{s}_{t-1}^T \mathbf{h}_i}{\sqrt{m}}\]

This only measures similarity in the original hidden state space. But the hidden states were learned for reconstruction, not for computing relevance.

WHY DO WE NEED WEIGHT MATRICES?

Example: when generating the Southern word “cyat,” the decoder state encodes “I need to produce a noun, animal-related.”

• The encoder state at “cat” encodes “common English noun, subject of the sentence.” These are related, but their raw hidden states might not have high dot product because they encode different types of information.

• We need to project both into a shared “relevance space” where the right things are similar.

THE PROJECTION MATRICES

Learn three separate projections:

\[\mathbf{Q} = \mathbf{S}\mathbf{W}_Q \quad ; \quad \mathbf{K} = \mathbf{H}\mathbf{W}_K \quad ; \quad \mathbf{V} = \mathbf{H}\mathbf{W}_V\]

• \(\mathbf{W}_Q\) \((m \times d_k)\): projects the decoder states into a query space. Transforms “what I need” into a form optimized for matching.

• \(\mathbf{W}_K\) \((D \times d_k)\): projects the encoder states into a key space. Transforms “what I represent” into the same matching space as the queries.

• \(\mathbf{W}_V\) \((D \times d_v)\): projects the encoder states into a value space. Transforms “what information I carry” into the form most useful for the decoder’s output.

\(d_k\) (the key/query dimension) controls the matching capacity — how rich the similarity computation is.

\(d_v\) (the value dimension) controls the information capacity — how much content each position can carry.

WHY SEPARATE K AND V?

Key insight: what makes two things similar is not the same as what information you want to extract.

THE DIMENSIONS

\[\mathbf{Q}: (N_Q \times d_k) \quad \mathbf{K}: (N_K \times d_k) \quad \mathbf{V}: (N_K \times d_v)\]

• \(N_Q\) = number of queries (decoder positions — how many outputs we’re generating)

• \(N_K\) = number of keys/values (encoder positions — how many inputs we’re attending to)

• \(d_k\) = dimension of the matching space (typically 64)

• \(d_v\) = dimension of the information space (typically 64)

Note: \(N_Q\) and \(N_K\) can be different lengths. This is what makes attention so flexible — a 10-word Southern sentence can attend to a 15-word English sentence with no architectural changes.

THE GENERAL ATTENTION FORMULA

\[\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V}\]

Step by step:

1. \(\mathbf{Q}\mathbf{K}^T\): \((N_Q \times d_k) \times (d_k \times N_K) = (N_Q \times N_K)\) — similarity score for every query-key pair
2. Divide by \(\sqrt{d_k}\): scale to prevent softmax saturation
3. Softmax (row-wise): normalize each query’s scores to a probability distribution over keys
4. Multiply by \(\mathbf{V}\): \((N_Q \times N_K) \times (N_K \times d_v) = (N_Q \times d_v)\) — weighted combination of values

Three matrix multiplications and a softmax. That’s it.

THE GENERAL ATTENTION FORMULA

THE ATTENTION LAYER

THE ATTENTION LAYER

THE ATTENTION LAYER

THE ATTENTION LAYER

THE ATTENTION LAYER

THE ATTENTION LAYER

THE ATTENTION LAYER

IMPORTANT Computing recurrence relations is incredibly slow

• Compute hidden state at 1, then at 2, then at 3, so on…

The attention setup can do everything in parallel

• Compute state at 1, 2, 3, so on… all at the same time!!!

This is really going to matter in a second.

PYTORCH DOES THIS FOR YOU

Everything we just derived — the Q/K/V projections, the scaled dot product, the softmax, the weighted sum — is implemented in a single PyTorch layer:

import torch.nn as nn
 
attn_layer = nn.MultiheadAttention(
    embed_dim=256,    # dimension of the input embeddings
    num_heads=8,      # number of attention heads (more next lecture)
    dropout=0.1,      # dropout on attention weights
)

USING MULTIHEADATTENTION

# Cross-attention: decoder queries encoder
# query: (seq_len_decoder, batch, embed_dim)
# key:   (seq_len_encoder, batch, embed_dim)
# value: (seq_len_encoder, batch, embed_dim)
 
output, attn_weights = attn_layer(
    query=decoder_states,
    key=encoder_states,
    value=encoder_states,
)
# output:       (seq_len_decoder, batch, embed_dim)
# attn_weights: (batch, seq_len_decoder, seq_len_encoder)

WHAT HAPPENS INSIDE

Under the hood, nn.MultiheadAttention does the following:

1. Project Q, K, V through learned weight matrices (\(\mathbf{W}_Q\), \(\mathbf{W}_K\), \(\mathbf{W}_V\))
2. Split into multiple heads (each head gets \(d_k / H\) dimensions) — more on this next lecture
3. Compute \(\text{softmax}(\mathbf{Q}\mathbf{K}^T / \sqrt{d_k})\mathbf{V}\) for each head independently
4. Concatenate the outputs from all heads
5. Project through an output matrix \(\mathbf{W}_O\) to combine information across heads
6. Apply dropout to the attention weights (regularization)

WHY WE DERIVED IT ANYWAY

We could have just said nn.MultiheadAttention and moved on. So why the math?

WHERE WE WERE

WHERE WE ARE

DOES THE ENCODER NEED TO BE RECURRENT?

Given the attention setup, do we need our encoder to be recurrent?

• The decoder looks back at every encoder state each time
• Does it matter if we encode the sequential nature of the input?

Yes: context of the input is sequential — different words make sense in the context of other input words. “Little” in “Bless Your Little Heart” is pejorative, not a descriptor!

No: the decoder doesn’t really care about the encoder’s internal ordering.

Recurrence is only needed to build context within the encoder hidden states!

• Turns out that logic is flawed!

THE BANK PROBLEM

Suppose that we have an input sentence:

I arrived at the bank after crossing the ____

SELF-ATTENTION: A SEQUENCE ATTENDS TO ITSELF

Allow the encoder to develop context of the input by looking at all other words in the input

“The animal didn’t cross the street because it was too wide.”

What does “it” refer to? Self-attention lets position 8 (“it”) directly attend to position 6 (“street”).

SELF-ATTENTION

SELF-ATTENTION

SELF-ATTENTION

SELF-ATTENTION: IN WORDS

For each word in the input, compute query-key-value sets (linear transformations of the input embeddings).

For each word \(i\):

1. Compute the dot product of \(\mathbf{q}_i\) with every key: \(e_{ij} = \frac{\mathbf{q}_i^T \mathbf{k}_j}{\sqrt{d_k}}\)
2. Softmax over all positions to get attention weights: \(a_{ij} = \text{softmax}_j(e_{ij})\)
3. Weighted sum of all values: \(\mathbf{y}_i = \sum_j a_{ij} \mathbf{v}_j\)

SELF-ATTENTION REPLACES THE RNN ENCODER

Each word gets a contextualized representation that incorporates information from every other word — no sequential processing needed.

“Bank” in “river bank” gets a different representation than “bank” in “bank account” because self-attention pulls in different context.

This replaces the RNN encoder entirely. No recurrence. Fully parallel.

SELF-ATTENTION REPLACES THE RNN ENCODER

Does this work?

LEARNED POSITIONAL EMBEDDINGS

Easyish fix - add learned positional embeddings to the input.

• Each position in the input sequence gets a unique embedding.
• These embeddings are learned during training, just like word embeddings.
• Allows the model to incorporate position information without recurrence.

Not that hard to implement at all in Pytorch!!!!

SELF-ATTENTION REALLY REPLACES THE RNN ENCODER

Since our inputs are always of fixed length (through padding), we can use learned positional embeddings to give the model information about the position of each word in the sequence.

• We’ve completely eliminated the wasteful recurrence calculations and replaced them with a set of fully parallelizable computations

• More compute resources = better models = faster training!

WHAT’S MISSING?

This is almost a transformer!

• Really. The transformer isn’t as complicated as it seems to be at first.

• Amazing how powerful the architecture is given its relative simplicity!

WHAT’S MISSING?

But we’re missing a few pieces:

WHAT’S MISSING?

But we’re missing a few pieces:

Feed-forward layers, residual connections, LayerNorm — the transformer block wraps attention with processing and gradient highways.