DATASCI 447 Lecture 24: GPT and Autoregressive Generation

Administrative Stuff

THE TRANSFORMER RECAP

Last lecture we built the full transformer for machine translation:

• Encoder: self-attention over the input sequence builds bidirectional context
• Decoder: masked self-attention over the output sequence + cross-attention to the encoder
• Goal: map “Bless your little heart” \(\to\) “You are sorely mistaken”

THE THREE ATTENTION MECHANISMS

What does self attention do?

What does masked self attention do?

What does cross attention do?

THE THREE ATTENTION MECHANISMS

WHY CROSS-ATTENTION IS THE BOTTLENECK

Cross-attention couples the encoder and decoder. Every decoder position attends to every encoder position.

WHY CROSS-ATTENTION IS THE BOTTLENECK

The cross-attention itself isn’t sequential — but the decoder generation is sequential, and each step requires a full cross-attention pass against the encoder. You can’t parallelize across tokens during generation because each token depends on the previous one.

• This is the architectural friction that motivated the next big idea.

THE GREAT SCHISM, BRIEFLY

Some researchers asked: do we even need the encoder?

WHAT GPT REMOVES

Take the full transformer and remove the entire encoder.

Also remove the cross-attention layers from the decoder — they have nothing to attend to.

WHAT GPT REMOVES

THE GPT BLOCK

class GPTBlock(nn.Module):
    def __init__(self, embed_dim, num_heads, ff_dim):
        super().__init__()
        self.ln1 = nn.LayerNorm(embed_dim)
        self.attn = nn.MultiheadAttention(embed_dim, num_heads)
        self.ln2 = nn.LayerNorm(embed_dim)
        self.ffn = nn.Sequential(
            nn.Linear(embed_dim, ff_dim),
            nn.GELU(),
            nn.Linear(ff_dim, embed_dim),
        )

    def forward(self, x, mask):
        x = x + self.attn(self.ln1(x), self.ln1(x), self.ln1(x), attn_mask=mask)[0]
        x = x + self.ffn(self.ln2(x))
        return x

Stack \(N\) of these blocks. GPT-1 used 12. GPT-2 used 48. GPT-3 used 96. The architecture is the same — just deeper and wider.

THE FULL GPT

class GPT(nn.Module):
    def __init__(self, vocab_size, max_len, embed_dim, num_heads, num_layers):
        super().__init__()
        self.token_embed = nn.Embedding(vocab_size, embed_dim)
        self.pos_embed = nn.Embedding(max_len, embed_dim)
        self.blocks = nn.ModuleList([
            GPTBlock(embed_dim, num_heads, 4 * embed_dim)
            for _ in range(num_layers)
        ])
        self.ln_final = nn.LayerNorm(embed_dim)
        self.head = nn.Linear(embed_dim, vocab_size)

    def forward(self, tokens):
        B, T = tokens.shape
        pos = torch.arange(T, device=tokens.device)
        x = self.token_embed(tokens) + self.pos_embed(pos)
        mask = torch.triu(torch.ones(T, T), diagonal=1).bool()
        for block in self.blocks:
            x = block(x, mask)
        x = self.ln_final(x)
        return self.head(x)  # (B, T, vocab_size) — logits for each position

Input: a sequence of token IDs. Output: a distribution over the vocabulary at every position.

THE TRAINING OBJECTIVE

GPT is trained on one task: predict the next token.

Given a sequence \([x_1, x_2, \ldots, x_T]\), the model learns to predict:

\[P(x_2 | x_1), \quad P(x_3 | x_1, x_2), \quad \ldots, \quad P(x_T | x_1, \ldots, x_{T-1})\]

WHY MASKED SELF-ATTENTION IS ESSENTIAL

Without the mask, position \(t\) could see positions \(t+1, t+2, \ldots\) during training. The model would learn to “predict” the next token by just looking at it directly. No learning happens.

THE PARALLEL TRAINING TRICK

Here’s where the decoder-only architecture really shines.

For a sequence of length \(T\), we want to train the model on \(T-1\) prediction tasks:

• Predict \(x_2\) given \(x_1\)
• Predict \(x_3\) given \(x_1, x_2\)
• …
• Predict \(x_T\) given \(x_1, \ldots, x_{T-1}\)

THE PARALLEL TRAINING TRICK

A GIANT JOINT DISTRIBUTION

What does GPT learn?

The whole semester we’ve been thinking about generation as a distribution problem:

• VAEs learn \(P(\mathbf{x})\) for images
• VQ-VAEs learn \(P(\mathbf{z})\) over codebook indices
• GPT learns \(P(\text{token sequence})\) over text

A GIANT JOINT DISTRIBUTION

By the chain rule of probability, this conditional distribution gives us the full joint distribution over sequences:

\[P(x_1, x_2, \ldots, x_T) = P(x_1) \cdot P(x_2 | x_1) \cdot P(x_3 | x_1, x_2) \cdots P(x_T | x_1, \ldots, x_{T-1})\]

• Given the selection of the 50,000 tokens we have already seen, what is the distribution of the next token over the same 50k words?

• GPT is a giant probabilistic model of all possible text sequences!

SAMPLING FROM THE DISTRIBUTION

How do we generate text? Sample from the distribution autoregressively.

1. Start with a prompt \([x_1, \ldots, x_t]\)
2. Compute \(P(x_{t+1} | x_1, \ldots, x_t)\) — a distribution over 50,000 tokens
3. Sample \(x_{t+1}\) from this distribution
4. Append to the sequence: \([x_1, \ldots, x_t, x_{t+1}]\)
5. Compute \(P(x_{t+2} | x_1, \ldots, x_t, x_{t+1})\)
6. Sample \(x_{t+2}\)
7. Repeat

SAMPLING FROM THE DISTRIBUTION

Each step conditions on everything that came before — both the original prompt and all the tokens we’ve sampled so far.

• This is probabilistic generation done correctly.
• Sample from the distribution the model learned.
• No tricks.

BUT PURE SAMPLING HAS PROBLEMS

The probabilistically pure thing to do is sample from the full distribution at every step.

BUT PURE SAMPLING HAS PROBLEMS

The solution: modify how we sample to bias toward higher-quality outputs while still maintaining diversity.

We give up some probabilistic purity to gain practical quality.

GREEDY DECODING

The simplest strategy: always pick the highest-probability token.

next_token = logits.argmax(dim=-1)

Deterministic. Same prompt always gives the same output. Maximum likelihood at each individual step.

GREEDY DECODING

Almost always terrible in practice:

• Repetitive (“The cat sat on the mat. The cat sat on the mat. The cat…”)
• Boring (the most probable response is often the most generic)
• Gets stuck in degenerate loops

Maximum likelihood at each step \(\neq\) maximum likelihood of the full sequence.

TEMPERATURE SAMPLING

Sample from the distribution, but first control how peaked it is with a temperature parameter:

\[P(x_t = i) = \frac{\exp(z_i / \tau)}{\sum_j \exp(z_j / \tau)}\]

where \(z_i\) are the raw logits and \(\tau\) is the temperature.

TEMPERATURE SAMPLING

• \(\tau = 1\): the natural distribution learned by the model
• \(\tau < 1\): peakier, more confident, closer to greedy
• \(\tau > 1\): flatter, more random, more creative
• \(\tau \to 0\): becomes greedy
• \(\tau \to \infty\): becomes uniform random

Temperature is the dial you actually see in ChatGPT and Claude APIs.

TOP-K SAMPLING

Top-k: only sample from the top \(k\) most probable tokens. Zero out everything else, renormalize, then sample.

def top_k_sample(logits, k):
    top_k_values, top_k_indices = logits.topk(k)
    probs = F.softmax(top_k_values, dim=-1)
    sampled_index = torch.multinomial(probs, num_samples=1)
    return top_k_indices.gather(-1, sampled_index)

TOP-K SAMPLING

Typical values: \(k = 40\) or \(k = 50\).

• Prevents the long tail problem — nonsense tokens are excluded entirely
• Maintains diversity among reasonable continuations

Limitation: the fixed \(k\) doesn’t adapt to the shape of the distribution. If the model is very confident (top 2 tokens have 99% mass), \(k = 50\) still lets you sample from 48 unlikely tokens.

TOP-P (NUCLEUS) SAMPLING

Top-p (Holtzman et al., 2020): instead of a fixed number of tokens, use a dynamic number that captures a fixed probability mass.

Sort tokens by probability. Keep the smallest set whose cumulative probability exceeds \(p\). Sample from this “nucleus.”

TOP-P (NUCLEUS) SAMPLING

Example with \(p = 0.9\):

• If the distribution is peaked (top token has 0.85), the nucleus is just 2-3 tokens
• If the distribution is flat (no clear winner), the nucleus might be 100+ tokens

The nucleus adapts to the model’s confidence:

• High confidence \(\to\) small nucleus \(\to\) consistent generation
• Low confidence \(\to\) large nucleus \(\to\) diverse generation

Typical values: \(p = 0.9\) to \(0.95\). This is the default in most modern systems.

Modern Systems Combine All Three

In practice, most modern systems combine three pieces:

• top-k as a safety net (\(k = 50\))

• then top-p (\(p = 0.9\)),

• then temperature.

Three dials working together.

BEAM SEARCH

All the methods so far are stochastic — you sample one path through the space of possible sequences.

Beam search tries to find high-probability sequences deterministically by maintaining multiple candidates.

WHICH TO USE WHEN?

The sampling strategy is often more important than people realize. A well-tuned GPT-3.5 with good sampling can outperform a poorly-tuned GPT-4 with greedy decoding on many tasks.

WHY GPT-2 WON’T SHUT UP

Try this with GPT-2 and any sampling strategy:

Prompt: "The old lighthouse keeper noticed something strange. It was"
GPT-2:  "...a small, white object that looked like a piece of paper.
         The man picked it up and looked at it. It was a small, white
         object. He looked at it again. It was a small, white object..."

No sampling strategy fixes this. You can crank temperature, lower top-p, do beam search — the output is still mediocre.

SAMPLING STRATEGIES ARE NOT A CURE

Sampling strategies are post-hoc fixes for problems in the underlying distribution.

• If the model has learned a good distribution, sampling strategies smooth out the rough edges. Temperature controls randomness. Top-p prevents long-tail nonsense. These are useful tools.

• If the model has learned a bad distribution, sampling strategies can’t help. You can’t sample your way to good output from a model that doesn’t know what good output looks like.

If you had to guess, what do you think GPT-2 was trained on?

THE DATA STORY

The jump from GPT-2 to GPT-3 was as much about data scale as it was about parameter scale.

THE DATA STORY

The jump from GPT-3 to GPT-4 was as much about data quality as it was about data quantity. GPT-4 used heavily filtered, deduplicated, and curated training data — including a lot of “legally” obtained content (the legal status of much of this data is, shall we say, contested).

• Wikipedia and BookCorpus aren’t enough to learn the ins and outs of human language. To get models that feel human, you need the messy, diverse, contradictory, often-copyrighted reality of how humans actually write.

ATTENTION AS A UNIVERSAL FUNCTION APPROXIMATOR

Self-attention is doing something deep: it’s a non-parametric, non-linear estimator of the conditional dependence structure in the input.

• Each attention layer learns: “given everything I’ve seen so far, which positions are most relevant to predict what comes next?”

• This is essentially a learned, context-dependent covariance estimator.

ATTENTION AS A UNIVERSAL FUNCTION APPROXIMATOR

Like all deep models, more capacity means the model can represent more complex dependence structures.

• Simple models capture word-pair associations
• Larger models capture syntactic structure
• Even larger models capture semantic relationships
• The largest models capture reasoning patterns, world knowledge, and stylistic conventions

Language is extraordinarily complex. More parameters \(\to\) more capacity \(\to\) more humanlike output.

THE PARAMETER RACE

WHY PARALLELIZATION IS ESSENTIAL

Training GPT-3 on a single GPU would take thousands of years. It’s only feasible because GPT training parallelizes cleanly:

• Data parallelism: different GPUs process different batches
• Model parallelism: different GPUs hold different parts of the model
• Pipeline parallelism: different GPUs process different layers
• Tensor parallelism: large matrix multiplications split across GPUs

INFERENCE IS A DIFFERENT STORY

Training is embarrassingly parallel. Inference is not.

At inference time, we generate one token at a time. Each token requires a full forward pass through the model, and we can’t start generating token \(t+1\) until we’ve finished token \(t\).

• A 100-token response requires 100 sequential forward passes
• Each forward pass touches every parameter in the model
• For a 1T parameter model, this is expensive

INFERENCE IS A DIFFERENT STORY

This is why ChatGPT sometimes feels slow, why API costs scale with tokens generated, and why “smaller, faster” models have a huge market. Inference speed is the bottleneck for real-world deployment.

• More parameters \(\to\) better quality, but also slower inference.

• More data centers

• More hardware buyouts

• More electricity used

• More natural resources consumed

IMAGE GPT AND PIXEL RNN

What if we treated an image as a sequence of pixels and applied next-token prediction directly?

This is exactly what early models did:

• Pixel RNN (2016): treat each pixel as a token, generate row by row, top-to-bottom, left-to-right
• Image GPT (2020): same idea, but with a transformer instead of an RNN

IMAGE GPT AND PIXEL RNN

Figure saved as 'pixel_rnn_approach.png'

IMAGE GPT AND PIXEL RNN

It worked in principle. The model learned to generate coherent images.

It didn’t scale in practice. A 256×256 image is 65,536 pixels. Each pixel has 256³ possible values. Even with quantization tricks, the sequence length and vocabulary size are enormous.

Image GPT could generate decent 32×32 images. Scaling to 256×256 was computationally infeasible.

ENTER THE VQ-VAE

The VQ-VAE solves this elegantly.

ENTER THE VQ-VAE

Figure saved as 'vqvae_compression_comparison.png'

GPT FOR VQ-VAE CODEBOOK MAPS

Build a decoder-only transformer that takes flattened codebook index sequences as input:

# Image → VQ-VAE encoder → 4×4 grid of codebook indices → flatten to 16 tokens
indices = vq_vae.encode(image)  # shape: (16,) with values in [0, 511]

# Train a small GPT to predict the next codebook index
gpt = GPT(vocab_size=512, max_len=16, embed_dim=256, num_heads=8, num_layers=6)

GPT FOR VQ-VAE CODEBOOK MAPS

Training: same as text GPT. Cross-entropy loss on next-token prediction, fully parallel via causal masking.

Generation: sample one codebook index at a time, conditioned on the indices generated so far. After 16 sampling steps, you have a complete codebook map. Pass it through the VQ-VAE decoder to get an image.

2D ROPE FOR SPATIAL STRUCTURE

One issue: when we flatten the 4×4 grid into a sequence of 16 tokens, we lose spatial information.

Position 5 is actually at row 1, column 1. Position 10 is at row 2, column 2. The decoder needs to know this.

CONDITIONING THE TRANSFORMER

How do we tell the model to generate “a husky” instead of “a chihuahua”?

[BREED=husky] [COLOR=gray] [s_1] [s_2] ... [s_16]

CONDITIONING THE TRANSFORMER

The transformer’s masked self-attention naturally handles this. Every codebook position attends to the conditioning tokens (and to all earlier codebook positions). The model learns:

\[P(s_i | \text{conditioning}, s_1, \ldots, s_{i-1})\]

At generation time, fix the conditioning tokens and sample the rest.

WHY THIS WORKS BETTER THAN VQ-VAE CONDITIONING

Back in Lecture 20, we conditioned the VQ-VAE itself by passing labels into the encoder and decoder.

That worked, but it was finicky:

• Required spatial broadcasting of conditioning vectors
• Forced the encoder to entangle conditioning with image features
• Made the codebook learn to depend on conditioning indirectly
• Hard to add new conditioning attributes without retraining everything

WHY THIS WORKS BETTER THAN VQ-VAE CONDITIONING

Conditioning the transformer is cleaner:

• The VQ-VAE is trained once, unconditionally — it learns to compress images
• The transformer handles all conditioning logic via prepended tokens
• Adding a new conditioning attribute just means adding a new vocabulary entry and retraining the (much smaller) transformer
• The conditioning is inside the autoregressive generation loop — every token sees the conditioning, every token can be influenced by it

This is the standard approach for modern image generators.

THIS IS (ALMOST) DALL-E 1

You’ve now built every component of DALL-E 1:

• A VQ-VAE that encodes images into discrete tokens (Lectures 19-20)
• Categorical conditioning via prepended tokens (this lecture)
• A decoder-only transformer that generates token sequences autoregressively (this lecture)

THE OTHER SIDE OF THE SCHISM

Today: the decoder-only path. Masked self-attention, autoregressive generation, GPT.

But there’s another side. What if we used the encoder only?

• Bidirectional self-attention over the entire input
• No causal mask, no autoregression
• Output: contextualized representations of every input position

This is the path that produced BERT (for text) and ViT (for images). It turns out to be the SOTA for nearly every downstream task that isn’t generation: classification, retrieval, similarity, embedding-based search, transfer learning.

THE OTHER SIDE OF THE SCHISM

Next lecture: encoder-only models, representation learning, and how the same transformer architecture serves a completely different purpose.