DATASCI 447 Lecture 21: Sequences - The Old Ways

Administrative Stuff

WHY SEQUENCES?

Last lecture we hit a wall: the VQ-VAE produces a grid of codebook indices, but generating those indices from scratch requires modeling their sequential dependencies.

\[P(s_1, s_2, \ldots, s_{16}) = \prod_i P(s_i \,|\, s_1, \ldots, s_{i-1})\]

“FEEDFORWARD” NEURAL NETWORKS

Thus far, we have only seen deep learning architectures where information flows upward through the model

• Inputs go through layers and yield hidden values
• Hidden values to more hidden values

Feedforward in the sense that the only direct influence of a layer is to its child

• In computational graphs, there is no complicated structure that creates feedback between units

“FEEDFORWARD” NEURAL NETWORKS

For tabular data and image analysis:

• A single input relates to a single output
  • Classification \(\rightarrow\) Class
  • Image Segmentation Map \(\rightarrow\) Semantic Segmentation over \(H \times W\) image
• A single input leads to two (or more) independent outputs
  • Regression \(\rightarrow\) (Mean, Variance)
  • Bounding Box \(\rightarrow\) (Class, Location)

THE FIVE PATTERNS

• Image classification (One image to One Class)

THE FIVE PATTERNS

• Image captioning (One image to a sequence of words)

THE FIVE PATTERNS

• Video classification (Many images/frames to one video class)
• Sentiment analysis (Sequence of words to one sentiment label)

THE FIVE PATTERNS

Not necessarily equal number of inputs and outputs

• Machine translation (Sequence of words in English to Seq in French)
• Sentence continuation (First part of sentence to second part)
• Time series (Stock prices for last 10 years to prices for the next week)

THE FIVE PATTERNS

Equal number of outputs and inputs

• Per-frame video classification (Seq of Images to Seq of Words)
• Per-sentence sentiment analysis (Seq of sentences to Class)
• Next-token prediction (Seq of codebook indices to Seq of next indices)

FEEDFORWARD NETWORKS CAN’T DO THIS

A feedforward network maps one fixed-size input to one fixed-size output. No memory. No sense of order.

RECURRENT NEURAL NETWORKS

Recurrent Neural Networks (RNNs) are the backbone of sequence modeling

• Inputs or outputs make sense temporally
• Knowing what came before should inform what we predict now

Adds knowledge to the NN architecture

• Units need to talk backwards in time — all of the previous values need to influence what I guess next!

THE RECURRENCE FORMULA

We can process a sequence of vectors, \(\mathbf{x}\), by applying a recurrence formula at every time step:

\[\mathbf{h}_t = f_w(\mathbf{h}_{t-1}, \mathbf{x}_t)\]

• \(\mathbf{h}_{t-1}\) is the old state (some vector of numbers)
• \(\mathbf{x}_t\) is the input at time \(t\) (some vector - think an embedding of length 32)
• \(f_w()\) is a function of some parameters, \(\mathbf{W}\) (some collection of weight matrices)
• \(\mathbf{h}_t\) is the new state (some vector)

THE HIDDEN STATE AS LATENT REPRESENTATION

The hidden state vector contains latent information about the current state of the model

• For language models, it could tell us about the gender of pronouns in the sentence
• Or the passiveness of the tone
• For image tokens, it could encode the spatial context seen so far

UNROLLING THE RNN

Start with an initial state, \(\mathbf{h}_0\)

THE VANILLA UPDATE

\[\underset{(m \times 1)}{\mathbf{h}_t} = \text{tanh}\left(\underset{(m \times m)}{\mathbf{W}_{hh}} \underset{(m \times 1)}{\mathbf{h}_{t-1}} + \underset{(m \times P)}{\mathbf{W}_{xh}} \underset{(P \times 1)}{\mathbf{x}_t} + \underset{(m \times 1)}{\mathbf{b}_h}\right)\]

• \(\mathbf{h}_t\) and \(\mathbf{h}_{t-1}\) are hidden vectors of size \(m\)
• \(\mathbf{W}_{hh}\) is a \(m \times m\) matrix that controls the mapping of one hidden state to the next
• \(\mathbf{W}_{xh}\) is a \(m \times P\) matrix that controls the mapping of the input to the hidden state
• \(\mathbf{b}_h\) is a vector of bias terms
• \(\text{tanh}()\) is a nonlinear activation function that elementwise maps values between -1 and 1

THE VANILLA UPDATE

\[\underset{(m \times 1)}{\mathbf{h}_t} = \text{tanh}\left(\underset{(m \times m)}{\mathbf{W}_{hh}} \underset{(m \times 1)}{\mathbf{h}_{t-1}} + \underset{(m \times P)}{\mathbf{W}_{xh}} \underset{(P \times 1)}{\mathbf{x}_t} + \underset{(m \times 1)}{\mathbf{b}_h}\right)\]

Drawing some parallels to what we’ve seen before:

• \(\mathbf{h}_t\) is like a higher level of hidden values — a function of the previous hidden values passed through an activation
• \(\mathbf{W}_{hh}\) maps these two hidden layers to one another
• \(\mathbf{W}_{xh}\) grounds the new hidden state to its input

A FCNN with a twist!

THE OUTPUT AT EACH STEP

Note that \(\mathbf{x}_1\) only relates to \(\mathbf{y}_1\) through \(\mathbf{h}_1\)!

Given a hidden state, we say that \(\mathbf{y}_t\) is determined as:

\[\underset{(G \times 1)}{\mathbf{y}_t} = g\left(\underset{(G \times m)}{\mathbf{W}_{hy}} \underset{(m \times 1)}{\mathbf{h}_t} + \underset{(G \times 1)}{\mathbf{b}_y}\right)\]

• \(g()\) maps the linear predictor to the scale of the output (softmax for classes, identity for regression)

WEIGHT SHARING ACROSS TIME

The part that is so simple that it’s kinda dumb

• But works amazingly

HOW \(\mathbf{y}_2\) SEES THE WHOLE HISTORY

Let’s think about how \(\mathbf{y}_2\) is computed.

• \(\mathbf{y}_2\) is a function of \(\mathbf{h}_2\)
• \(\mathbf{h}_2\) is a function of \(\mathbf{h}_1\) and \(\mathbf{x}_2\)
• \(\mathbf{h}_1\) is a function of \(\mathbf{h}_0\) and \(\mathbf{x}_1\)

A SIMPLE LANGUAGE MODEL

Let’s look at an aligned many-to-many model!

Goal: Given input characters, build a model to predict the next character!

A SIMPLE LANGUAGE MODEL

A SIMPLE LANGUAGE MODEL

A SIMPLE LANGUAGE MODEL

A SIMPLE LANGUAGE MODEL

A SIMPLE LANGUAGE MODEL