DATASCI 447 Lecture 23: Attention is All You Need

\[ \newcommand\hbb{{\hat{\boldsymbol \beta}}} \newcommand\bb{{\boldsymbol \beta}} \newcommand\expn{{\frac{1}{N} \sum \limits_{i = 1}^N}} \newcommand\sumk{\sum \limits_{k = 1}^K} \newcommand\argminb{\underset{\bb}{\text{argmin }}} \newcommand\argmaxb{\underset{\bb}{\text{argmax }}} \newcommand\gtheta{\mathbf g(\boldsymbol \theta)} \newcommand\htheta{\mathbf H(\boldsymbol \theta)} \]

Seq2Seq Problems

Seq2Seq models are RNNs that convert an input sequence to output sequences

• We’re going to restrict our attention to the unaligned variant

• The input sequence and the output sequence are not necessarily of the same length and may not have any direct one-to-one correspondence

Most commonly seen as question and answer or translation

• My screen is blank. \(\rightarrow\) Please check if the computer is plugged in.

• Bless your little heart \(\rightarrow\) You are sorely mistaken

Seq2Seq Problems

The “simple” model:

Seq2Seq Problems

In the training case

• We see “Bless Your Little Heart” and “You are Sorely Mistaken”. Train the model to maximize the probability that this translation occurs.

At test time

• Given a prompt to translate “Bless Your Little Heart” from Southern to English, return “You Are Sorely Mistaken” token-by-token given a trained model.

Seq2Seq Problems

The “simple” model:

Seq2Seq Problems

The hidden state updates as we move through the sequence

• For seq2seq, we don’t actually need to track in the next input token! All we care about is getting our desired answer!

The hidden state of the encoder is used as the input for the decoder

• Let it ride - just continue the recurrent sequence

Remember the issue with this from last time?

Seq2Seq Problems

The decoder may do better if it is allowed to look at all hidden states in the encoder instead of just looking at the final state

• All decoder hidden values carry information

• Every output token is related to the input directly

Attention

Attention

At \(t = 1\) of the decoder:

• Start with the input embedding for <sos>

• Using the previous decoder hidden state, \(\mathbf s_0\), find the dot product between the previous hidden state and all encoder hidden states

• Convert these dot products to attention weights

• Make the context vector a convex combination of all encoder hidden states weighted by the attention weights

• Proceed like a RNN

Attention

Attention allows the decoder to find relevant parts of the input for determining what should come next in the decoder.

Input:

My computer won’t turn on

Output:

Is…

[it,she,there,…]

Attention

With attention, we’re allowed to update context in the decoder!

A step forward from “simple” seq2seq models

Problems:

• Slow - each update must be done iteratively

• Very deep - unrolling this entire process into a feedforward style model shows that the number of layers is really high; one for each input and output token

• Relatively poor memory - the encoder still requires only looking one step back and can lose older information

Attention

A warning: this is all going to move relatively quickly

• Some steps to get from seq2seq with attention to transformers

Just keep in mind that autodiff will be able to backprop through everything here

• PyTorch will handle it all!

Move towards a machine that understands(?) the complexities of human language/images

Attention

Attention

Given the attention setup, do we need our encoder to be recurrent?

• The decoder looks back at every step each time

• Does it matter if we understand the sequential nature of the input?

Yes:

• Context of the input is sequential - different words make sense in the context of other input words

• Need to know that “Little” in “Bless Your Little Heart” is pejorative instead of a descriptor!

No:

• The decoder doesn’t really care

Recurrence is only needed to find the hidden states of the encoder!

Attention

Attention

Self-Attention

Allow the encoder to develop context of the input by looking at all other words in the input

Self-Attention

Self-Attention

Self-Attention

Self-Attention

In words:

• For each word in the input, compute query-key-value sets (linear transformations of the input embeddings)

• For each word \(i\):

  • Compute the dot product similarity between the query for \(i\) and all keys \(j \in T_e\) - \(\mathbf q_i^T \mathbf k_j\)

  • Softmax these similarities to get attention weights for each words - \(\mathbf w_i = [w_{i1},w_{i2},...,w_{iT_e}]\)

  • Compute the output, \(\mathbf o_i\), as the weighted combination of each \(v_j\) and the corresponding attention weight \(w_{ij}\)

The encoder looks forwards and backwards to see which words of the input correspond to each input word!

• Context!

Self-Attention

Self-Attention

Self-Attention

Each self-attention operation will correspond to one notion of context:

• Because \(\rightarrow\) [didn’t, cross] (what)

But, there are often layers of context:

• Because \(\rightarrow\) [wide] (why)

Allow each layer of context to be uncovered using multiple self-attention operators

Self-Attention

Multiheaded Self Attention: