DATASCI 447 Lecture 25: Encoder-Only Architectures

Administrative Stuff

WHERE WE’VE BEEN

Last lecture we built GPT — the decoder-only side of the transformer.

• Masked self-attention over a single sequence
• Next-token prediction as the training objective
• Autoregressive generation at inference time

WHY BIDIRECTIONAL MATTERS

GPT only sees tokens from the left. When predicting the next token, it must work with incomplete context.

WHY BIDIRECTIONAL MATTERS

WHAT ENCODER-ONLY MODELS PRODUCE

GPT produces tokens — text.

BERT and ViT produce representations — dense vectors that encode meaning.

WHAT ENCODER-ONLY MODELS PRODUCE

These representations are the foundation for everything downstream:

• Classification: what category does this input belong to?
• Retrieval: find inputs similar to this one
• Clustering: group related inputs together
• Transfer learning: use these representations as features for new tasks

BERT

Bidirectional Encoder Representations from Transformers (Devlin et al., 2018)

Architecture: the encoder half of the original transformer. Nothing fancy.

• Self-attention + feed-forward + LayerNorm + residuals
• Stacked 12 times (BERT-base) or 24 times (BERT-large)
• No causal mask — every position attends to every other position

The innovation isn’t the architecture. It’s the training objective.

THE TRAINING PROBLEM

For GPT, the training objective was obvious: predict the next token. A left-to-right signal that every sequence provides for free.

MASKED LANGUAGE MODELING

Randomly mask 15% of input tokens. Ask the model to predict what was masked, using the bidirectional context.

MASKED LANGUAGE MODELING

THE MASKING STRATEGY

The mask is more subtle than “replace 15% with [MASK]”:

• 80% of selected tokens: replaced with [MASK]
• 10% of selected tokens: replaced with a random token
• 10% of selected tokens: left unchanged

NEXT SENTENCE PREDICTION

BERT also trains on a second objective: next sentence prediction (NSP).

Given two sentences \(A\) and \(B\), predict whether \(B\) actually follows \(A\) in the original corpus, or was randomly sampled from elsewhere.

THE [CLS] TOKEN

Every BERT input starts with a special token: [CLS]. A learned beginning-of-sequence token that serves as a summary representation for the entire input.

• After BERT processes the sequence, the [CLS] token’s final embedding serves as a summary of the entire input — a single vector representation of the whole sentence (or pair of sentences).

This is the token you use for sentence-level classification tasks. Pass the [CLS] embedding through a small linear head to produce class logits:

cls_embedding = bert(input_ids).last_hidden_state[:, 0, :]  # (B, 768)
logits = classifier_head(cls_embedding)  # (B, num_classes)

• The [CLS] embedding isn’t magic — it’s just the first position, which the model learns to populate with useful summary information during training (because both MLM and NSP incentivize this).

THE [CLS] TOKEN

BERT SIZES

• Tiny by modern standards. GPT-3 (2020) was 1,600× larger than BERT-base.

But BERT’s release in 2018 was the watershed moment.

• Before BERT: every NLP task required its own model, trained from scratch.

• After BERT: one pretrained model, fine-tuned for everything.

A FAMILY OF BERTS

BERT was released in October 2018. Within 18 months, dozens of variants appeared, each addressing some limitation of the original.

• The core architecture barely changed. What changed: training strategy, training data, parameter efficiency, and downstream adaptation.

• Let’s look at the major descendants. Each one teaches something about what matters in pretraining.

A FAMILY OF BERTS

ROBERTA: BETTER TRAINING, NOT BETTER ARCHITECTURE

RoBERTa (Liu et al., 2019) is BERT with no architectural changes. But it significantly outperforms BERT.

The changes:

• More training data: 160GB instead of 16GB
• Longer training: 500K steps instead of 100K
• Larger batches: 8K instead of 256
• Removed NSP: trained only on MLM (NSP turned out to hurt performance)
• Dynamic masking: mask different tokens each epoch instead of masking once
• Longer sequences: no more 512-token cap during training

The lesson: BERT was undertrained. The architecture was fine — the recipe needed improvement.

• This is a recurring pattern in deep learning. Don’t assume architectural innovations are necessary. Often, better training protocols on the same architecture produce larger gains than new architectures.

DISTILBERT: COMPRESSING BERT

DistilBERT (Sanh et al., 2019): a BERT that’s 40% smaller, 60% faster, retains 97% of BERT’s performance on downstream tasks.

How? Knowledge distillation.

DISTILBERT: COMPRESSING BERT

Train a small student model (6 layers instead of 12) to match the output distribution of the full teacher BERT. The student learns to mimic the teacher’s predictions across many inputs.

Loss function combines:

• Hard loss: predict the true labels (like normal training)

• Soft loss: match the teacher’s output distribution (KL divergence)

• Cosine loss: match the teacher’s hidden states

• Why this works: the teacher’s output distribution contains more information than hard labels. “90% dog, 9% wolf, 1% cat” teaches the student something that “1.0 dog, 0 wolf, 0 cat” doesn’t — namely, that dogs and wolves are similar.

• Distillation became a standard technique for deploying large models on edge devices or in latency-sensitive applications.

ALBERT: PARAMETER SHARING

ALBERT (Lan et al., 2019): “A Lite BERT” — dramatically reduces parameters with two tricks.

• Factorized embeddings: instead of a \(V \times H\) embedding matrix (vocabulary × hidden), use \(V \times E\) and \(E \times H\) where \(E \ll H\). Drops embedding parameters by ~85%.

• Cross-layer parameter sharing: use the same attention weights across all 12 layers. The model still has 12 computational layers but only 1 layer’s worth of parameters.

Result: ALBERT-base has 12M parameters (vs BERT-base’s 110M) and matches or slightly beats BERT on most tasks.

ELECTRA: A SMARTER PRETRAINING OBJECTIVE

ELECTRA (Clark et al., 2020): “Efficiently Learning an Encoder that Classifies Token Replacements Accurately”

MLM has an inefficiency: the model only learns from the 15% of tokens that were masked. The other 85% provide no training signal.

ELECTRA’s insight: instead of predicting masked tokens, detect replaced tokens.

• A small generator replaces ~15% of input tokens with plausible-but-wrong alternatives
• The main model (the discriminator) predicts, for every token, whether it was replaced
• Training signal comes from every token, not just the masked 15%

Result: ELECTRA trains ~4× faster than BERT for equivalent performance. Small ELECTRA matches BERT-base with 1/30 the compute.

THE OLD WAY

Before BERT, the NLP workflow for each task looked like this:

1. Collect labeled data for your specific task (sentiment, NER, Q&A, etc.)
2. Design a task-specific architecture (possibly including word embeddings, attention, etc.)
3. Train from scratch (or with word embeddings pretrained on a language modeling objective)
4. Hope you have enough labeled data

THE BERT WAY

The paradigm shift:

1. Pretrain once on a massive corpus using self-supervised objectives (MLM + NSP)
2. Fine-tune the pretrained model for each specific downstream task

THE BERT WAY

The pretrained model already knows:

• Grammar and syntax
• Word meanings in context
• Common factual relationships
• How language is structured

Fine-tuning teaches it to apply this knowledge to a specific task, not to learn language from scratch.

This works because most of what a sentiment classifier needs to know (grammar, vocabulary, semantics) is the same as what a Q&A model needs to know. Pretraining captures the shared foundation.

FINE-TUNING IN PRACTICE

The mechanics are beautifully simple:

from transformers import BertModel, BertTokenizer
 
# Load pretrained BERT (downloaded once, used for everything)
bert = BertModel.from_pretrained("bert-base-uncased")
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
 
# Add a task-specific head
classifier = nn.Linear(768, num_classes)
 
# Train on your task
for batch in dataloader:
    outputs = bert(**batch)
    cls_embedding = outputs.last_hidden_state[:, 0, :]
    logits = classifier(cls_embedding)
    loss = F.cross_entropy(logits, labels)
    loss.backward()
    optimizer.step()

During fine-tuning, both BERT’s weights and the classifier head are updated. The pretrained knowledge isn’t frozen — it’s a starting point that adapts to the specific task.

WHY THIS WORKS SO WELL

Three forces combine:

1. Massive pretraining data. BERT was trained on Wikipedia + BookCorpus (~3.3 billion words). No downstream task has this much labeled data. Pretraining extracts knowledge from unlabeled text that fine-tuning couldn’t possibly learn.

2. Shared linguistic foundation. Every NLP task requires understanding grammar, vocabulary, and semantics. Pretraining captures this once, fine-tuning adapts it.

3. Small labeled datasets become sufficient. Because most of the model’s knowledge comes from pretraining, fine-tuning can succeed with just a few thousand labeled examples — sometimes even a few hundred.

THE MANY FACES OF FINE-TUNING

Different tasks use different “heads” on top of BERT:

• Sentence classification (sentiment, topic): linear head on [CLS]
• Token classification (NER, POS tagging): linear head on each token
• Sentence pair classification (NLI, similarity): linear head on [CLS] with [A][SEP][B] input
• Span prediction (Q&A): two linear heads predicting start and end positions
• Sentence similarity: cosine similarity between [CLS] embeddings

The backbone is always the same BERT. Only the head and the input format change.

CAN WE DO THIS FOR IMAGES?

BERT treats text as a sequence of word tokens. The transformer encoder processes the sequence bidirectionally.

Can we do the same for images?

• The answer from Dosovitskiy et al. (2021): “An Image is Worth 16×16 Words.”

IMAGES AS SEQUENCES

The key insight: treat an image as a sequence of patches instead of pixels.

IMAGES AS SEQUENCES

VIT ARCHITECTURE

After the patch embedding step, ViT is architecturally identical to BERT:

• Stack of transformer encoder blocks (self-attention + FFN + LayerNorm + residuals)
• Bidirectional self-attention over all patches
• [CLS] token whose final embedding serves as the image representation

The only differences are the input preprocessing (patches vs word embeddings) and the training objective (supervised classification vs MLM).

THE SCALE QUESTION

Here’s where ViT gets interesting.

Trained on ImageNet (1.3M images): ViT underperforms comparable CNNs. ResNet-50 beats ViT-Base. Why?

WHEN VIT WINS

Trained on JFT-300M (300 million images): ViT matches or beats every CNN. Why?

WHEN VIT WINS

THE DEMOCRATIZATION POINT

This has real consequences.

CNNs work for everyone. A research group with a few GPUs and a few million images can train a competitive CNN.

ViTs only work at massive scale. You need hundreds of millions of training images and enormous compute to get ViTs to beat CNNs.

PRETRAINED VITS

Fortunately, you don’t need JFT-300M to use a ViT. Pretrained models are freely available:

from torchvision.models import vit_b_16, ViT_B_16_Weights
 
model = vit_b_16(weights=ViT_B_16_Weights.IMAGENET1K_V1)

Or from HuggingFace:

from transformers import ViTModel
model = ViTModel.from_pretrained("google/vit-base-patch16-224")

LORA FOR TRANSFORMERS: SAME IDEA, NEW LIBRARY

You’ve seen LoRA before — low-rank updates to frozen pretrained weights, trained efficiently, merged at deployment.

For transformers in the HuggingFace ecosystem, the standard tool is the PEFT library (Parameter-Efficient Fine-Tuning).

LORA FOR TRANSFORMERS: SAME IDEA, NEW LIBRARY

PEFT wraps any model from transformers with LoRA adapters in a few lines:

from transformers import AutoModelForSequenceClassification
from peft import LoraConfig, get_peft_model
 
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2
)
 
lora_config = LoraConfig(
    r=8,                           # rank of the low-rank decomposition
    lora_alpha=16,                 # scaling factor for the update
    target_modules=["query", "value"],  # which weights to adapt
    lora_dropout=0.1,
    bias="none",
    task_type="SEQ_CLS",
)
 
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 295,682 || all params: 109,778,690 || trainable%: 0.27%

Fine-tuning a 110M-parameter BERT with LoRA updates ~300K parameters. For a 70B-parameter Llama with LoRA, you update ~35M parameters — trainable on a single consumer GPU.

WHICH WEIGHTS TO ADAPT?

The target_modules parameter controls which weights get LoRA adapters. Different choices for different architectures:

For BERT and most encoder models:

target_modules = ["query", "value"]  # attention Q and V projections

Empirically, adapting Q and V is sufficient for most tasks. Adding K projections helps marginally.

For LLMs (Llama, GPT-style):

target_modules = ["q_proj", "v_proj", "k_proj", "o_proj", 
                  "gate_proj", "up_proj", "down_proj"]

Newer practice adapts attention projections and MLP layers. More parameters, better performance on complex reasoning tasks.

For ViT:

target_modules = ["query", "value"]  # same as BERT

WHERE REPRESENTATION LEARNING IS GOING

We’ve spent the whole semester on representation learning. From PCA to CNNs to VAEs to BERT to ViT, the core question has always been:

How do we extract meaningful structure from unstructured data?

THE CNN DIDN’T DIE

Before we talk about transformers taking over vision, let’s correct a common misconception.

CNNs are still extremely important. They haven’t been replaced — they’ve been supplemented.

THE CNN DIDN’T DIE

The lesson: architecture choice matters less than training protocol and scale. A well-trained CNN with modern techniques matches a well-trained ViT on most benchmarks.

WHERE CNNS STILL WIN

CNNs have real practical advantages over pure ViTs:

Efficiency at small scale. CNNs’ inductive biases (locality, translation equivariance) mean they learn well from smaller datasets. If you have 10K images, use a CNN.

Mobile and edge deployment. CNN operations are highly optimized for mobile GPUs and NPUs. Modern phones can run EfficientNet in real-time; ViTs struggle without significant optimization.

Dense prediction tasks. Semantic segmentation, depth estimation, optical flow — tasks that need pixel-accurate outputs. CNNs’ spatial structure aligns naturally with these outputs. U-Net (CNN-based) remains the default for medical imaging.

Real-time video. Object detection and tracking in videos need to run at 30+ FPS on modest hardware. CNN detectors (YOLO series, particularly YOLOv8/v9/v10) dominate here.

CNN FINE-TUNING BEATS VITS IN SPECIALIZED DOMAINS

Despite ViT’s dominance on web-scale datasets, LoRA fine-tuning of pretrained CNNs (especially ResNet variants) continues to win in specific domains:

Scientific imaging: ImageNet-pretrained ResNet-50 or ResNet-152, fine-tuned with LoRA, consistently outperforms ViT on:

• Medical imaging (radiology, histopathology, dermatology) — limited labeled data, fine-grained texture patterns, standard evaluation benchmarks
• Satellite and remote sensing imagery — spatial resolution matters, domain-specific datasets are smaller than ImageNet
• Microscopy and cell biology — pattern recognition at scales where translation equivariance is genuinely valuable
• Materials science (crystal structures, defect detection) — highly structured local patterns

CNN FINE-TUNING BEATS VITS IN SPECIALIZED DOMAINS

Despite ViT’s dominance on web-scale datasets, LoRA fine-tuning of pretrained CNNs (especially ResNet variants) continues to win in specific domains:

Sports analytics: ResNet backbones dominate in:

• Player tracking and pose estimation — real-time inference constraints favor CNN efficiency
• Action recognition from broadcast footage — temporal CNN architectures remain state-of-the-art
• Performance analysis on limited-scale custom datasets

THE RISE OF HYBRID ARCHITECTURES

The real state-of-the-art isn’t pure CNN or pure ViT. It’s hybrid architectures that combine both.

The intuition: convolutions are efficient at capturing local spatial patterns. Self-attention is efficient at capturing long-range dependencies. Use each where it works best.

THE RISE OF HYBRID ARCHITECTURES

THE RISE OF HYBRID ARCHITECTURES

Key hybrid architectures:

Swin Transformer (Liu et al., 2021): windowed self-attention that processes local regions (like convolutions) with periodic shifts to enable cross-window communication. Dominant backbone for detection and segmentation in 2021-2023.

CoAtNet (Dai et al., 2021): “Coupling Attention with Convolution” — uses convolutions in early layers (to leverage locality) and attention in later layers (for global context). Wins ImageNet at every model size.

ConvNeXt V2 (Woo et al., 2023): CNNs updated with masked autoencoder pretraining. Matches self-supervised ViT performance.

EfficientFormer (Li et al., 2022): designed specifically for mobile deployment. Hybrid conv + attention that runs as fast as MobileNet while matching ViT accuracy.

WHY HYBRIDS WIN

The pure architectures have complementary weaknesses:

Hybrid architectures get the best of both:

• Early layers use convolutions to efficiently extract local features with strong inductive biases
• Later layers use attention to model long-range relationships between higher-level features

MASKED AUTOENCODERS: BERT FOR VISION

Masked Autoencoders (He et al., 2022): BERT-style self-supervised pretraining applied to vision.

The recipe:

1. Split image into patches (ViT-style)
2. Mask 75% of patches randomly
3. Train an encoder-decoder to reconstruct the masked patches from the visible ones
4. After pretraining, discard the decoder and use the encoder for downstream tasks

MASKED AUTOENCODERS: BERT FOR VISION

MASKED AUTOENCODERS: BERT FOR VISION

Why this beats supervised pretraining:

• No labels needed — can train on hundreds of millions of unlabeled images
• Learns richer features than classification (must understand full image content, not just category-discriminating features)
• Dramatically higher mask ratio than BERT (75% vs 15%) because images have more redundancy than text

SELF-SUPERVISED LEARNING WITHOUT LABELS

A broader trend: move entirely away from labeled datasets during pretraining.

Key techniques:

• Contrastive learning (SimCLR, MoCo, DINO): learn representations by pulling together different views of the same image and pushing apart views of different images
• Masked modeling (BERT, MAE, BEiT): predict hidden parts of the input from visible context
• Clustering-based methods (SwAV, DeepCluster): alternately cluster embeddings and train the encoder to match cluster assignments

The backbone architecture (CNN, ViT, hybrid) matters less than the self-supervised objective. Pick the backbone that fits your deployment constraints; pick the objective that fits your data.

FOUNDATION MODELS FOR VISION

Two competing approaches to vision foundation models:

DINOv2 (Oquab et al., 2023): Meta’s self-supervised ViT trained on 142M curated images. Produces general-purpose visual features useful for classification, segmentation, depth estimation, retrieval. One encoder, many tasks, no task-specific training.

SAM — Segment Anything (Kirillov et al., 2023): trained on 1 billion segmentation masks across 11M images. Accepts prompts (points, boxes, text) and produces pixel-accurate masks. Foundation model for dense prediction.

Both represent the dream: the vision analog of BERT/GPT. Solve the representation problem once, then apply it everywhere.

BEYOND BERT FOR TEXT

Text representation learning has also moved past vanilla BERT.

Sentence-BERT (SBERT): fine-tune BERT so that the [CLS] embedding directly produces useful sentence vectors. Specialized for retrieval and similarity.

E5, BGE, Cohere Embed: dedicated embedding models trained with contrastive objectives on massive retrieval datasets. Outperform general BERT on vector search, RAG, and semantic similarity.

Multilingual models (XLM-R, mBERT): pretrain on 100+ languages simultaneously, producing shared representations across languages.

MULTIMODAL REPRESENTATIONS

The cutting edge: learn representations that work across multiple modalities simultaneously.

CLIP (next lecture): text and images in a shared embedding space.

ImageBind (Meta, 2023): six modalities in a shared space — images, text, audio, depth, thermal, IMU (motion) data. Any modality can be used to retrieve any other.

DINO + CLIP: combining self-supervised visual features with language alignment produces vision models that “understand” content in ways earlier models couldn’t.

THE REPRESENTATION LEARNING THESIS

The unifying claim of modern ML:

Almost every problem becomes tractable if you have a good representation.

• If a text encoder produces meaningful sentence embeddings, classification becomes a nearest-neighbor problem. Retrieval becomes vector search. Clustering becomes k-means. Similarity becomes cosine distance.

• If a visual encoder (CNN, ViT, or hybrid) produces meaningful image embeddings, all the same tools apply to images.

• If a multimodal encoder aligns text and images in one space, cross-modal retrieval and zero-shot classification become trivial.

The hard part isn’t solving downstream tasks. It’s learning good representations.

THE REPRESENTATION LEARNING THESIS

That’s what this semester has been about. From PCA to ConvNeXt to ViT, every lecture has added tools to the representation-learning toolkit. You now have the modern state-of-the-art arsenal.

WHY GPT ALSO DOES THIS

A conceptual aside: GPT is also a pretrained model.

Pretraining: next-token prediction on web text. Fine-tuning for chatbots: supervised fine-tuning on instruction-following data + RLHF.

• The paradigm is the same — pretrain for broad capability, then specialize.

CONNECTING TEXT AND IMAGES

We now have:

• BERT: produces text representations (vectors)
• ViT / CNN / hybrids: produce image representations (vectors)

What if we trained these together so that images and their captions produce similar representations?

• A shared embedding space where “a photo of a dog” and an image of a dog land in nearly the same place.

• This is CLIP. Next lecture.

And once we have CLIP, we can finish the DALL-E 1 and introduce the true bleeding edge - Stable Diffusion.