DATASCI 447 Lecture 26: CLIP — Aligning Manifolds Across Modalities

Administrative Stuff

WHERE WE ARE

We’ve now built encoders for two modalities separately:

• Last lecture: BERT for text \(\to\) representations of sentences
• Also last lecture: ViT for images \(\to\) representations of images Both produce dense vectors in \(\mathbb{R}^d\) (typically \(d = 768\)).

A natural question: are these the same kind of vector? If I have a BERT vector for “a photo of a dog” and a ViT vector for an actual dog image, is there any reason these should be similar?

THE MANIFOLD HYPOTHESIS, REVISITED

Recall from our early discussions of representation learning:

The manifold hypothesis: real-world high-dimensional data doesn’t fill its space. It lies on (or near) a much lower-dimensional manifold embedded in that space.

Examples:

• Images: a 224×224 RGB image lives in \(\mathbb{R}^{150528}\), but the set of “natural images” forms a tiny manifold within that space. Most points in \(\mathbb{R}^{150528}\) are random noise, not images.
• Text: a sentence is a sequence of tokens, but the set of “meaningful English sentences” is a vanishingly small subset of all possible token sequences.

The job of a good encoder is to discover and parameterize this manifold. Take the high-dimensional input, map it to a low-dimensional representation that captures the structure of the data.

TWO MANIFOLDS, SAME CONCEPTS

If both text and images obey the manifold hypothesis, then somewhere in BERT’s output space there’s a text manifold, and somewhere in ViT’s output space there’s an image manifold.

Both manifolds describe the same underlying world. The same set of concepts — dogs, cars, people, sunsets, sadness, motion — appear on both.

• A picture of a dog and the words “a photo of a dog” describe the same thing. They live on different manifolds (one image, one text), but they refer to the same point in semantic reality.

EMBEDDINGS HAVE SEMANTIC MEANING

Let’s be precise about what we mean by “embedding spaces have semantic meaning.”

In a useful embedding space:

• Distance corresponds to semantic similarity
• Direction corresponds to semantic relationships
• Clusters correspond to semantic categories
• Trajectories correspond to semantic interpolation

EMBEDDINGS HAVE SEMANTIC MEANING

Classic word2vec example:

\[\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{queen}}\]

• Gender as a direction in space. Royalty as a direction in space. Arithmetic in embedding space corresponds to semantic composition.

This isn’t a quirk of word2vec. Any good embedding space exhibits this property because semantic relationships have geometric structure. Encoders that successfully discover the manifold preserve this structure.

THE INVARIANCE PRINCIPLE

Here’s the critical observation that students often miss:

The semantic meaning of an embedding space is invariant to certain transformations.

Specifically, if you apply a rotation, reflection, or rescaling to all embeddings in a space, the semantic relationships are preserved:

• Distances between points: preserved (under rotation/reflection)
• Angular relationships: preserved
• Cluster structure: preserved
• Compositional arithmetic: preserved

All of these properties are about relative positions of points. Rotating or reflecting the entire space doesn’t change which points are near each other or how they’re arranged.

THE INVARIANCE PRINCIPLE

TWO MANIFOLDS, TWO COORDINATE SYSTEMS

So now we have:

• A text manifold living in BERT’s output space (in some orientation)
• An image manifold living in ViT’s output space (in some other orientation)

Both manifolds capture the same underlying semantic structure. But each was discovered independently by its own encoder, which means each was placed in its own arbitrary coordinate system.

THE ALIGNMENT PROBLEM

To compare a text embedding with an image embedding, we need them in the same coordinate system.

THE ALIGNMENT PROBLEM

But there are problems:

• You need pre-existing correspondences (matched text-image pairs) to compute the rotation
• A single rotation only works if the manifolds have the same intrinsic structure
• Independently trained encoders rarely produce manifolds that align well even after optimal rotation
• Real manifolds are nonlinear — a single linear transformation can’t fully align them

WHY POST-HOC ALIGNMENT FAILS

The deeper problem: independently trained encoders learn different aspects of the data.

BERT learns whatever structure helps it predict masked tokens. ViT learns whatever structure helps it classify images.

These objectives produce manifolds that are structurally different, not just rotated versions of each other.

• BERT’s text manifold organizes words by distributional contexts
• ViT’s image manifold organizes images by visual features

The “dog” region in text-space might be characterized by syntactic features (“noun, animate, four-legged”), while the “dog” region in image-space is characterized by visual features (“furry, snouted, tail-having”). These regions can be similar in size and shape but oriented very differently in their respective high-dimensional spaces.

WHY POST-HOC ALIGNMENT FAILS

THE CLIP INSIGHT

Here’s where CLIP’s brilliance becomes clear:

Don’t try to align two pre-existing manifolds. Train both encoders simultaneously so the manifolds emerge already aligned.

1. Set up a shared embedding space ahead of time
2. Train both encoders so they map their respective inputs into this shared space
3. Use a clever loss function that incentivizes alignment during training, not after

The “rotation” that aligns the two manifolds isn’t computed post-hoc. It’s baked into the encoders themselves through joint training.

• This is one of those ideas that’s obvious in retrospect but required someone to actually try it at scale. CLIP did, with 400 million image-text pairs.

THE OBVIOUS APPROACH: CROSS-ATTENTION

The original transformer used cross-attention to bridge encoder (input language) and decoder (output language). Why not use cross-attention to bridge text and images?

Architecturally:

• Encode the image patches with a vision encoder

• Encode the text tokens with a text encoder

• Add cross-attention layers where text attends to image (or vice versa)

• Train end-to-end on some joint task

• This is exactly how the original transformer handled machine translation. It would seem to be the “standard” answer to combining two modalities.

THE PROBLEMS WITH CROSS-ATTENTION FOR MULTIMODAL LEARNING

Three big issues at scale:

Problem 1: Coupling. With cross-attention, the model processes image and text together in every forward pass. You can’t encode an image without a text prompt. You can’t encode text without an image. There’s no separable “image embedding” or “text embedding” you can extract and reuse.

Problem 2: Computational cost. Cross-attention between \(N\) image patches and \(M\) text tokens costs \(O(NM)\) per layer. For 400M training pairs, this gets astronomically expensive.

Problem 3: It defeats the purpose. If we use cross-attention, we never produce the shared embedding space we wanted in the first place. We’d just produce joint representations of (image, text) pairs. That’s useful for some tasks but not what we’re after.

THE USE CASE DRIVES THE ARCHITECTURE

Imagine building an image search engine with 100 million images.

With cross-attention: for each query text, run the full cross-attention model against every image in the database. 100 million forward passes per query. Completely infeasible.

THE USE CASE DRIVES THE ARCHITECTURE

With separate encoders + shared space:

1. Pre-compute image embeddings for all 100M images, store them on disk (just 200 GB)
2. For each query, encode the text once (one forward pass)
3. Compute 100M dot products (extremely cheap, ~0.1 seconds on a GPU)
4. Return top matches

For most real-world applications — retrieval, classification, similarity, conditioning — embedding reusability is dramatically more valuable than the few percentage points of joint accuracy cross-attention provides.

THE BEST OF BOTH WORLDS (FORESHADOWING)

A note on cross-attention: it isn’t gone forever in the multimodal world.

Stable Diffusion (which we’ll cover in the final lectures) uses CLIP’s text encoder to produce text embeddings, then uses cross-attention inside the U-Net to condition image generation on those embeddings.

THE TWO ENCODERS

CLIP uses two separate encoders:

Text encoder: a transformer (similar to GPT-2 or a small BERT)

• Input: tokenized text

• Output: a single embedding vector per text sample (typically the [EOS] token’s representation) Image encoder: a ViT (or ResNet in some variants)

• Input: an image

• Output: a single embedding vector per image (typically the [CLS] token’s representation)

THE TWO ENCODERS

Both outputs are passed through a linear projection head that maps them into a shared \(d\)-dimensional space (typically \(d = 512\)).

• The projection heads are critical.
• Before projection, the encoders’ native representations live in their own coordinate systems.
• After projection, they live in a shared coordinate system — but only if training teaches both projections to produce aligned representations.

THE TWO ENCODERS

THE TRAINING DATA

CLIP’s training data is massive and simple:

400 million (image, caption) pairs scraped from the web.

No class labels, no annotations, no careful curation. Just naturally occurring images with their associated text from the open internet.

THE CONTRASTIVE LOSS

Here’s the clever loss function that makes alignment happen during training.

For a batch of \(N\) image-text pairs, encode all of them. Get \(N\) image embeddings \(\mathbf{i}_1, \ldots, \mathbf{i}_N\) and \(N\) text embeddings \(\mathbf{t}_1, \ldots, \mathbf{t}_N\). L2-normalize them all (project to unit sphere).

Compute the full similarity matrix:

\[\mathbf{S}_{jk} = \mathbf{i}_j \cdot \mathbf{t}_k / \tau\]

THE CONTRASTIVE LOSS

This is an \(N \times N\) matrix of cosine similarities, scaled by a learned temperature \(\tau\).

• Diagonal entries (\(j = k\), matched pairs): should be HIGH
• Off-diagonal entries (\(j \neq k\), mismatched pairs): should be LOW

Apply softmax cross-entropy in both directions, treating the diagonal as the correct answer:

\[\mathcal{L}_{\text{CLIP}} = \frac{1}{2}\left[ \text{CE}(\mathbf{S}, \text{diag}) + \text{CE}(\mathbf{S}^T, \text{diag}) \right]\]

THE SIMILARITY MATRIX

THE SIMILARITY MATRIX

Key observations:

• For each image, the model must identify the correct caption among \(N\) choices
• For each caption, the model must identify the correct image among \(N\) choices
• Larger batch size \(\to\) more confusable distractors \(\to\) harder task \(\to\) better representations
• CLIP’s original batch size: 32,768 image-text pairs

THIS LOSS HAS A NAME: INFONCE

The CLIP loss isn’t an arbitrary construction — it’s an instance of a well-studied loss function called InfoNCE (Information Noise Contrastive Estimation), introduced by van den Oord et al. in 2018 for representation learning.

For a single anchor (image \(j\)) and its positive partner (text \(j\)) among \(N-1\) negatives:

\[\mathcal{L}_j = -\log \frac{\exp(\mathbf{i}_j \cdot \mathbf{t}_j / \tau)}{\sum_{k=1}^{N} \exp(\mathbf{i}_j \cdot \mathbf{t}_k / \tau)}\]

• “given this image, identify which of the \(N\) texts is its true partner.” CLIP applies this in both directions (image-as-anchor and text-as-anchor) and averages.

THE INFORMATION-THEORETIC JUSTIFICATION

Minimizing the InfoNCE loss is equivalent to maximizing a lower bound on the mutual information between the two modalities.

\[\mathcal{L}_{\text{InfoNCE}} \approx -I(\mathbf{i}; \mathbf{t}) + \log N\]

• The loss provides a tractable way to estimate (and maximize) how much information the image embedding carries about the text embedding, and vice versa.

THE INFORMATION-THEORETIC JUSTIFICATION

Why does this matter?

• We want embeddings that capture shared semantic content between modalities
• “Shared content” is exactly what mutual information measures
• We can’t compute \(I(\mathbf{i}; \mathbf{t})\) directly in high dimensions — the joint distribution is intractable
• InfoNCE gives us a practical proxy that we can actually optimize with gradient descent

This is why CLIP-style training works so well across modalities. We’re not just pulling matched pairs together — we’re maximizing the mutual information between modality-specific representations of the same concept.

WHY BATCH SIZE MATTERS — AND WHY WE DON’T DIVIDE BY \(N\)

Look at the InfoNCE loss again:

\[\mathcal{L}_{\text{InfoNCE}} \approx -I(\mathbf{i}; \mathbf{t}) + \log N\]

That \(\log N\) term is doing real work.

Notice: the loss isn’t normalized by batch size. We don’t divide by \(N\). Each batch contributes a full categorical cross-entropy, summed across all \(N\) anchors.

WHY BATCH SIZE MATTERS — AND WHY WE DON’T DIVIDE BY \(N\)

Why? Because the number of negatives is the source of learning signal:

• With \(N = 2\) pairs in a batch: the model picks between 1 positive and 1 negative. Trivial.
• With \(N = 256\) pairs: 1 positive vs 255 negatives. Genuinely hard.
• With \(N = 32{,}768\) pairs: 1 positive vs 32,767 negatives. The model must learn precise semantic distinctions to succeed.

The bound becomes tighter with more negatives:

\[I(\mathbf{i}; \mathbf{t}) \geq \log N - \mathcal{L}_{\text{InfoNCE}}\]

• More negatives \(\to\) tighter bound \(\to\) better mutual information estimate \(\to\) more informative gradients.

WHY BATCH SIZE MATTERS — AND WHY WE DON’T DIVIDE BY \(N\)

This is why CLIP’s original batch size was 32,768 — and why every subsequent contrastive paper obsesses over batch size. It’s not for computational efficiency. It’s because the negatives ARE the signal.

Practical implication: if you’re training a contrastive model and you can’t fit a large batch, use techniques like memory banks (MoCo) or distributed batches across GPUs to artificially increase the effective negative count.

THE LEARNABLE TEMPERATURE

CLIP doesn’t use a fixed temperature. It learns \(\tau\) as a parameter, jointly with the encoders. Specifically, CLIP parameterizes \(\log(1/\tau)\) as a free parameter and bounds it for stability.

Why learnable? The optimal sharpness changes during training:

• Early training: the encoders produce rough, noisy embeddings. A high \(\tau\) (soft distribution) gives smooth gradients and prevents the model from collapsing onto a few high-confidence predictions.
• Late training: the encoders produce sharp, discriminative embeddings. A low \(\tau\) (peaked distribution) lets the model exploit fine semantic distinctions and produce confident predictions.

The model learns to anneal \(\tau\) down over training, automatically adjusting its own “confidence.” No manual hyperparameter tuning, no scheduled annealing — the model figures it out.

WHAT TEMPERATURE IS DOING

WHAT TEMPERATURE IS DOING

Why we expect \(\tau\) to decrease during training:

Early on, the encoders barely know what they’re doing. If we used a low \(\tau\) from the start, gradients would be dominated by a few high-confidence (often wrong) predictions, and training would be unstable.

As training progresses, the embeddings genuinely become more discriminative. The model can afford to be more decisive. Lower \(\tau\) extracts more learning signal from each comparison by sharpening the distinction between the true match and the distractors.

• Letting the model learn its own temperature is a beautifully simple solution — the model decides how confident to be based on how good its representations actually are.

WHAT THIS LOSS ACTUALLY DOES

Let’s connect this back to the manifold framing.

• For a matched pair (image \(j\), caption \(j\)), the loss says: “these two embeddings should be close.” This pulls the corresponding points on the two manifolds toward each other.

• For a mismatched pair (image \(j\), caption \(k \neq j\)), the loss says: “these two embeddings should NOT be close.” This pushes unrelated points on the two manifolds away from each other.

WHAT THIS LOSS ACTUALLY DOES

Over millions of batches and 400M pairs, this produces a remarkable effect:

• Every concept that appears in the training data acquires a consistent location in the shared space
• The text encoder learns to map “a photo of a dog” near where the image encoder maps actual dog images
• Related concepts (dogs and puppies) cluster together
• Unrelated concepts (dogs and bicycles) are far apart

The two manifolds co-evolve during training. They don’t start aligned — they become aligned because the loss function rewards alignment.

THE KEY INSIGHT

This is the moment to step back and appreciate what’s happening.

• CLIP isn’t doing anything fancy with the encoders themselves. The text encoder is just a transformer. The image encoder is just a ViT. Neither has any awareness of the other modality.

The only thing that creates cross-modal alignment is the loss function.

• The loss function says: “for matched pairs, your embeddings should be similar.” That’s it.

WHAT YOU GET AFTER TRAINING

Two trained encoders, one shared embedding space.

import clip
model, preprocess = clip.load("ViT-B/32")
 
# Encode an image
image_embedding = model.encode_image(preprocess(image).unsqueeze(0))
 
# Encode text
text_tokens = clip.tokenize(["a photo of a dog", "a photo of a cat"])
text_embeddings = model.encode_text(text_tokens)
 
# Compare
similarity = image_embedding @ text_embeddings.T

Both outputs are in the same 512-dimensional space. Cosine similarity between them is semantically meaningful. Image and text live on the same manifold.

ZERO-SHOT CLASSIFICATION

Classify an image without training any classifier.

Given an image and a set of candidate labels, compute similarity between the image embedding and each label’s text embedding. Pick the most similar.

def zero_shot_classify(image, candidate_labels, model):
    image_emb = model.encode_image(image)
    text_prompts = [f"a photo of a {label}" for label in candidate_labels]
    text_embs = model.encode_text(clip.tokenize(text_prompts))
    similarities = (image_emb @ text_embs.T).softmax(dim=-1)
    return candidate_labels[similarities.argmax()]

Geometrically: we’re asking “which point on the text manifold is closest to this point on the image manifold?” Because the manifolds are aligned, that text point names what’s in the image.

THE IMAGENET RESULT

The original CLIP paper’s headline result:

• Train CLIP on 400M web image-text pairs
• Evaluate zero-shot on ImageNet (a dataset CLIP has never been explicitly trained on)
• CLIP zero-shot achieves ~76% top-1 accuracy on ImageNet

For comparison: a ResNet-50 fine-tuned end-to-end on ImageNet achieves ~76% top-1 accuracy.

PROMPT ENGINEERING

Zero-shot accuracy depends heavily on how you phrase the class descriptions.

Compare:

• Just the class name: “dog” \(\to\) lower accuracy
• Template: “a photo of a {class}” \(\to\) higher accuracy
• Context: “a photo of a {class}, a type of animal” \(\to\) even higher

This is the original “prompt engineering” — not for LLMs, but for CLIP.

PROMPT ENGINEERING

Why does this matter? CLIP’s text encoder was trained on web captions. “A photo of a golden retriever” matches the distribution of training captions. Just “golden retriever” does not.

• Geometrically: Adding “a photo of a…” moves the text embedding to where natural image captions actually live.

• Ensembling across multiple prompts (averaging embeddings from several templates) gains 1-3% accuracy. This is finding the centroid of multiple nearby points on the manifold.

CROSS-MODAL RETRIEVAL

Text \(\to\) Image retrieval: given a text query, find matching images.

# Pre-compute embeddings for an image database (do this once)
image_embs = model.encode_image(all_images)  # (N, 512)
 
# Encode query (do this for each query)
query_emb = model.encode_text(clip.tokenize("a red car on a beach"))
 
# Find top-k matches (very fast)
similarities = query_emb @ image_embs.T
top_k = similarities.topk(5)

CLIP or CLIP-derivatives power nearly every modern image retrieval system: Google Images, Pinterest visual search, stock photo search, and many recommendation feeds.

THE EMBEDDING SPACE IS COMPOSITIONAL

Because CLIP learns a semantic geometry, embeddings support compositional queries:

• “red car” \(\to\) red car images
• “red car on a beach” \(\to\) red car images, specifically on beaches
• “red car on a beach at sunset” \(\to\) even more specific

Each additional descriptor shifts the embedding toward a more specific region of the manifold. The compositional structure of language gets reflected in the geometric structure of the embedding space.

THE EMBEDDING SPACE IS COMPOSITIONAL

This is emergent. Nobody told CLIP that “red car on a beach” should be the intersection of “red car” and “beach.” It learned this from seeing millions of natural image captions where compositional descriptions paired with compositionally-described scenes.

The manifold has internal structure that respects the compositional semantics of the modalities.

• The artificial system mimics human reasoning and intelligence

THE KILLER APPLICATION: CONDITIONING GENERATION

Remember from Lecture 20: we built a conditional VQ-VAE with categorical breed/color labels. It worked, but the conditioning signal was rigid — you could only condition on the exact labels in the training set.

With CLIP, we can use arbitrary text as a conditioning signal:

1. Encode a text prompt with CLIP’s text encoder \(\to\) a 512-dim vector
2. Feed that vector as conditioning to a generative model
3. The model generates images matching the text

This is what DALL-E, Stable Diffusion, Midjourney, and every modern text-to-image model uses.

RECALLING WHERE WE LEFT OFF

In Lecture 20, we built the conditional VQ-VAE:

• Encode image \(\to\) 4×4 grid of codebook indices (16 tokens)
• Decode codebook indices \(\to\) reconstructed image
• Condition generation on breed + color labels . . .

In Lecture 24, we built the autoregressive prior:

• Train a small GPT on flattened sequences of codebook indices
• Condition with prepended [BREED] [COLOR] tokens
• Generate new images by sampling codebook sequences

We noted at the end of Lecture 24: the only thing missing from DALL-E 1 was a way to use arbitrary text as the conditioning signal.

THE FULL DALL-E 1 ARCHITECTURE

THE “IT WAS ALL WORTH IT” MOMENT

Every piece you’ve built this semester plays a role:

• Autoencoders (L17): compression into latent space \(\to\) VQ-VAE’s foundation
• VAEs (L17-18): probabilistic latent models \(\to\) generative grounding
• VAE-GAN (L19): perceptual losses \(\to\) critical for image quality
• VQ-VAE (L19-20): discrete latent codes \(\to\) image “vocabulary”
• Conditioning (L20): shaping generation \(\to\) categorical \(\to\) CLIP embeddings
• Attention (L22-23): the transformer machinery \(\to\) powers the prior
• GPT (L24): sequential generation \(\to\) the codebook transformer
• BERT (L25): representation learning \(\to\) CLIP’s text encoder architecture
• CLIP (today): manifold alignment \(\to\) the conditioning signal that makes it all work

You can now understand and build DALL-E 1 from first principles (multimodal AI until 2023). Every component, every loss function, every architectural decision.

WHY DALL-E 1 WAS SUPERSEDED

DALL-E 1 was an important milestone but was quickly replaced by diffusion-based approaches (DALL-E 2, Stable Diffusion).

Limitations of autoregressive image generation over VQ-VAE tokens:

• Sequential generation: each token depends on previous tokens, so generation is slow
• Discrete bottleneck: VQ-VAE’s discrete codebook loses fine detail
• Limited resolution: 16×16 or 32×32 grids can only represent so much

Diffusion solves these:

• Parallel denoising: all pixels/patches refined simultaneously at each step
• Continuous latents: no discrete bottleneck
• High resolution: operates directly on latent spaces matching megapixel images

WHY DALL-E 1 WAS SUPERSEDED

But CLIP remains essential. Every modern text-to-image model uses CLIP (or CLIP-derivatives) to encode text prompts. The conditioning mechanism didn’t change — only the image generator did.

CLIP is the part that didn’t get replaced. The shared embedding space is too useful to give up.

MULTIMODAL MODELS TODAY

The CLIP pattern — separate encoders + shared embedding space — has spread far beyond text and images:

BLIP / BLIP-2: add a generation capability on top of CLIP-style alignment. BLIP jointly trains three heads — contrastive (ITC), matching (ITM), and language modeling (LM) — so one model can do retrieval AND captioning.

ImageBind (Meta, 2023): six modalities aligned to a common space — text, image, audio, depth, thermal, IMU. Alignment composes transitively: align everything to images, and you get all pairwise alignments for free.

Modern VLMs (GPT-4V, Claude with vision, Gemini): vision encoder + LLM, with visual features projected into the LLM’s token space. Same philosophy as CLIP but with generation built in.

THE TRAJECTORY

The field is converging on a simple recipe:

1. Build specialized encoders for each modality
2. Train them to produce embeddings in a shared space using contrastive or generation objectives
3. Use that shared space for downstream tasks

Everything we’ve built this semester fits this pattern. Representation learning, alignment, and generation are three aspects of the same underlying framework.

THE TRAJECTORY

Next two lectures: diffusion models. A completely different generative mechanism that replaces autoregressive token sampling with iterative denoising — but still uses CLIP embeddings as the conditioning signal. Stable Diffusion is what happens when you combine diffusion’s generative power with CLIP’s shared embedding space.