DATASCI 447 Lecture 28: The Omnimodal Frontier

Administrative Stuff

• Course Evals

• Change: PS5 optional. If you want to do it, go for it. If not, no worries. Big reason - a lot of you are actually doing what you would’ve done in the problem set in your project. Use your colab budget for that instead.

• Posters by tomorrow. Must be printed and dropped off in PAIS by 11:30 AM. If you want to present, make sure that you reach out to Ms. Charles to get on the schedule. A good way to show off the project!

• Final project deliverables by May 5th at 11:59 PM. A final paper, website, etc. Must also submit a link to your replicable code Github. Make sure that your code is well documented and can be run by someone else. This is a requirement for the project. Make a nice README to explain what’s going on.

WHERE WE LEFT OFF

Last lecture we built the complete diffusion framework:

• Forward process: gradually add noise until data becomes \(\mathcal{N}(\mathbf{0}, \mathbf{I})\)
• Reverse process: a U-Net learns to predict and remove the noise, step by step
• Training: MSE on noise prediction — the simplest loss of any generative model
• Sampling: DDPM (stochastic) or DDIM (deterministic, 50 steps instead of 1000)

THE DIFFUSION FRAMEWORK IN 60 SECONDS

The core insight: instead of forcing a VAE encoder to match \(\mathcal{N}(\mathbf{0}, \mathbf{I})\) in one step (which creates prior holes), gradually walk from data to noise over \(T = 1000\) tiny steps, then learn to reverse the walk.

THE TRAINING LOOP AND THE LOSS

The entire complex ELBO collapsed to:

\[L_{\text{simple}} = \mathbb{E}_{t,\,\mathbf{x}_0,\,\boldsymbol{\epsilon}}\left[\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right\|^2\right]\]

In plain English: pick a random image, add random noise at a random scale, train the network to predict what noise you added. MSE. That’s it.

THE TRAINING LOOP AND THE LOSS

Saved: diffusion_pipeline_seed42.png

THE TRAINING LOOP AND THE LOSS

Saved: diffusion_pipeline_seed43.png

THE TRAINING LOOP AND THE LOSS

Saved: diffusion_pipeline_seed44.png

SAMPLING AND DDIM

DDPM sampling: start from pure noise, iteratively denoise over \(T\) steps. Each step: predict the noise, subtract most of it, add a tiny bit of fresh randomness. Stochastic — same starting noise gives different images.

THE CONDITIONING PROBLEM

Unconditional diffusion generates random images from the learned distribution. We want text-to-image. The network needs to know WHAT to generate, not just HOW to denoise.

We have CLIP text embeddings (big idea!) — a rich, semantically meaningful conditioning signal that lives in the same space as images (Lecture 26).

CROSS-ATTENTION CONDITIONING

Remember cross-attention from Lecture 22? Queries from one sequence, keys and values from another.

In the diffusion U-Net:

• Queries: U-Net feature maps (the image being denoised)
• Keys and Values: CLIP text embeddings (the conditioning signal)

Cross-attention layers are sandwiched between the conv/attention blocks at multiple resolutions in the U-Net. At every level of the hierarchy, the image features can “look at” the text to decide what content to generate.

In plain English: at each denoising step, the U-Net asks “what does the text want here?” and adjusts its noise prediction accordingly. The text steers the compass.

CROSS-ATTENTION CONDITIONING

Saved: unet_cross_attention.png

CROSS-ATTENTION CONDITIONING

Saved: diffusion_pipeline_conditioned_seed44.png

Look Where We Are

Look at what’s happening in that cross-attention layer:

• The U-Net architecture: from semantic segmentation
• The cross-attention mechanism: from transformers (L22)
• The CLIP text embeddings: from contrastive learning (L26)
• The denoising objective: from diffusion (L27)

Four separate threads from four different parts of the semester, meeting in one architectural component.

This is not a coincidence. It’s the payoff of building each piece carefully, understanding what it does, and recognizing where it fits in the larger system.

CLASSIFIER-FREE GUIDANCE

Cross-attention lets the model see the text. But in practice, the conditioning signal is too weak — generated images vaguely match the prompt but lack specificity.

The trick: train ONE model that sometimes sees the text and sometimes doesn’t. During training, randomly drop the text embedding (replace with a null embedding) with ~10-20% probability.

THE GUIDANCE FORMULA

At inference, combine both predictions:

\[\hat{\boldsymbol{\epsilon}} = \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing) + w \cdot \left(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \mathbf{c}) - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing)\right)\]

In plain English: start with the unconditional prediction (“where does the data manifold generally live?”), then add an amplified version of the difference the conditioning makes.

THE GUIDANCE FORMULA

The guidance scale \(w\) controls how strongly the text steers generation:

• \(w = 1\): normal conditioning (often too weak)
• \(w = 7\text{-}15\): typical range for good text-to-image results
• \(w > 20\): oversaturated, artifacted, but very literal

This is the same idea as the negative prompts from the CLIP notebook — push toward what you want, push away from the generic baseline. Classifier-free guidance is the diffusion version of that exact technique.

WHY CLASSIFIER-FREE GUIDANCE WORKS

The difference \(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \mathbf{c}) - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing)\) isolates what the conditioning ADDS — the pure “dog-ness” direction in noise-prediction space, stripped of all the generic “image-ness” that both predictions share.

• Multiplying by \(w > 1\) amplifies this direction. You’re saying: “I don’t just want a slight lean toward dogs — I want DOGS, emphatically.”

THE PROBLEM WITH PIXEL-SPACE DIFFUSION

Running the U-Net 50 times on a 512×512×3 image is expensive. Each forward pass processes a 786,432-dimensional tensor through a billion-parameter network. 50 sequential evaluations, no parallelization across steps.

• The same problem we solved with VQ-VAE back in Lecture 19: raw pixels are too big. We need compression.

LATENT DIFFUSION: THE STABLE DIFFUSION INSIGHT

Rombach et al. (2022): run diffusion in a compressed latent space instead of pixel space.

LATENT DIFFUSION: THE STABLE DIFFUSION INSIGHT

Saved: diffusion_pipeline_conditioned_seed44.png

THE SAME TRICK AS VQ-VAE

Same two-stage philosophy as DALL-E 1:

• Stage 1: learn a good compression (VAE / VQ-VAE)
• Stage 2: do generation in the compressed space (transformer / diffusion)

WHY THIS MADE DIFFUSION PRACTICAL

The numbers:

• Pixel-space diffusion on 512×512: each U-Net pass processes ~786K values
• Latent diffusion on 64×64×4: each U-Net pass processes ~16K values
• That’s 48× less computation per step

WHY THIS MADE DIFFUSION PRACTICAL

The result: Stable Diffusion generates a 512×512 image in ~5 seconds on a consumer GPU. Pixel-space diffusion at equivalent quality would take minutes.

ONE ARCHITECTURE

The complete Stable Diffusion stack,:

• VAE encoder/decoder (L17-19): compresses images to/from latent space
• Perceptual + adversarial losses (L19): make VAE reconstructions sharp
• U-Net with skip connections (L14, L27): the denoising backbone
• Self-attention in the U-Net (L22-23): global context at each resolution
• Cross-attention (L22): text conditioning injection point
• CLIP text encoder (L26): text \(\to\) aligned embedding
• InfoNCE / contrastive training (L26): how CLIP learned alignment
• Diffusion forward/reverse process (L27): the generative mechanism
• Noise prediction + simplified ELBO (L27): the training objective
• DDIM (L27): efficient deterministic sampling
• Classifier-free guidance (Lecture 20 + Today): amplified conditioning

EVERYTHING YOU’VE BUILT, IN ONE ARCHITECTURE

Demo

THE FOUNDATION MODEL ARGUMENT

Stable Diffusion’s continuous latent space acts as a foundation model for visual generation.

• Unlike VQ-VAE, which requires new codebook entries for new concepts, the continuous space can represent anything — including concepts it’s never explicitly seen.

• “An astronaut riding a horse on Mars” works because the latent space is smooth and compositional, not because anyone trained on that specific image.

THE FOUNDATION MODEL ARGUMENT

One model, arbitrary prompts, infinite concepts. The continuous space + CLIP conditioning + diffusion creates a system that generalizes far beyond its training data.

EXTENSIONS THAT PROVE THE FOUNDATION

The Stable Diffusion base model spawned an entire ecosystem:

ControlNet: add new conditioning signals (edges, depth maps, poses) without retraining the base model. Just learn a new cross-attention pathway. The foundation’s representations are rich enough to support arbitrary conditioning.

LoRA for diffusion: apply low-rank adapters to the cross-attention weights. Train custom styles, characters, or concepts on a consumer GPU in minutes. Same technique from L25, applied to diffusion.

DreamBooth: personalize the model to generate specific subjects (your face, your pet) from 5-10 reference images. Fine-tune a tiny subset of parameters.

THE DEATH OF “JUST AN LLM”

The term “large language model” is legacy. Modern frontier models are omnimodal foundation models.

THE UNIVERSAL CONCEPT OF “DOG”

The word “dog,” a bark waveform, a photograph of a golden retriever, and a 3D mesh of a dog are no longer different modalities.

• They are the same mathematical coordinate in a shared latent space, aligned by contrastive learning (CLIP and its successors from L26).

THE HYBRID ENGINE

Modern AI uses two complementary mechanisms:

THE HYBRID ENGINE

These aren’t competing paradigms. They’re complementary tools that pass universal concepts back and forth through shared embedding spaces.

The LLM reasons about what to create. The diffusion model creates it. The representations flow seamlessly between them because they share the same latent geometry.

THE HARDWARE WALL

The math we teach assumes infinite precision and unlimited memory. The engineering reality is different.

Modern frontier models are memory-bandwidth bound. The bottleneck isn’t compute speed — it’s how fast you can move weights from VRAM to compute cores. We’re using essentially all available VRAM in the world already.

• I even struggle to implement real code examples on my machine. It’s a beast too.

• I love my computer, but it can’t run a 70B-parameter model with 32GB of VRAM and 96GB of DDR5. The hardware wall is real.

THE HARDWARE WALL

Where do we go from here?

• Better silicon: HBM4, faster interconnects, more VRAM per chip. Slow (2-3 year cycles), expensive, power-hungry.
• Extreme quantization: run 70B models in 2-4 bit precision. Fits consumer VRAM, some quality loss. This is what makes local AI possible.
• Algorithmic efficiency: consistency models, flow matching, distillation. Do more with less compute.

Every deployment is a compromise between model quality and hardware constraints. Students entering industry will spend more time on these tradeoffs than on architecture design.

BREAKING THE DENOISING TAX

Diffusion’s Achilles heel: 20-50 sequential U-Net evaluations per image. Can we do better?

Consistency Models (Song et al., 2023): train a model that maps any noise level directly to the clean image in one step. Distill the iterative process into a single function. Near-diffusion quality at 1-4 steps.

Flow Matching (Lipman et al., 2023): instead of the noising/denoising Markov chain, learn a continuous flow field that transports noise to data along straight paths. Simpler math, often faster convergence. Stable Diffusion 3 uses flow matching internally.

DIT: TRANSFORMERS EATING EVERYTHING

The U-Net is being replaced by Diffusion Transformers (DiT). Transformer Eat World

Patchify the noisy latent (exactly like ViT patches an image from L25), add timestep and conditioning embeddings, run through a standard transformer.

Why this matters: it’s the same architecture everywhere now.

• GPT for text generation
• ViT for visual representation
• DiT for visual generation

One architecture family, every modality, every task.

DIT: TRANSFORMERS EATING EVERYTHING

Sora (RIP in peace) is the concrete example: spacetime patches through a DiT, conditioned on text. Video generation using the same attention mechanism you learned in Lecture 23.

• The architectural distinction between “text models” and “image models” and “video models” is dissolving. It’s all transformers, all the way down.

NEW CAPABILITIES, NEW ETHICAL PROBLEMS

Every frontier we just discussed creates new ethical questions without settled answers:

NEW CAPABILITIES, NEW ETHICAL PROBLEMS

Every frontier we just discussed creates new ethical questions without settled answers:

Consent and data: Stable Diffusion was trained on billions of images scraped from the web. Artists whose styles are reproducible never consented. Is this fair use? Courts are still deciding.

NO QUICK ANSWERS

We don’t have good answers for most of these questions.

NOTICE WHAT WE NEVER DID

We never called this an “AI class.” The course is DATASCI 447. We talked about autoencoders, variational inference, attention mechanisms, contrastive learning, stochastic optimization, Markov chains.

NOTICE WHAT WE NEVER DID

THIS IS A COP OUT. Don’t accept this logic that AI is “reasoning like a human”. You know better.

• There’s a fine line between firing neurons and artificial systems.

• Marketing garbage

NOTICE WHAT WE NEVER DID

The Blake Lemoine incident (Google, 2022) — A Google engineer publicly claimed LaMDA was sentient based on conversations where the model expressed feelings and fears. He was fired.

• Stoked fear and more “anthropomorphizing AI” headlines. No technical basis for the claim. Just a human projecting onto a black box.

• The “reasoning” we see is RHLF - match what the raters liked and don’t match what they didn’t.

• Racism, sexism, and all.

NOTICE WHAT WE NEVER DID

When a model “refuses on moral grounds,” that’s a learned mapping from prompt patterns to refusal outputs that maximized reward during training. When it “expresses feelings,” it’s producing text that pattern-matches to emotional expression in its training corpus.

THE REAL LINEAGE

The work that powers “AI” in 2026 was enabled by decades of advances that nobody called AI at the time:

• Statistics: maximum likelihood estimation, Bayesian inference, the EM algorithm, variational methods — the conceptual foundations of every model we built
• Applied mathematics: optimization theory, differential equations, information theory, manifold geometry — the tools that make training possible
• Computer science: automatic differentiation, GPU computing, distributed systems — the engineering that makes scale possible

THE REAL LINEAGE

ChatGPT isn’t a breakthrough in “artificial intelligence.” It’s a breakthrough in applied Bayesian statistics running on very fast hardware.

• The transformer is a function approximator. Diffusion is a Markov chain. CLIP is a mutual information estimator. The ELBO is a variational bound. These aren’t AI concepts — they’re statistics concepts with good marketing.

THE ENGINE AND THE CHASSIS

The people who will make the next big contributions are the ones who understand what’s happening under the hood — not just the ones who can call .generate() on a pretrained model.

After this class, you understand the engine - not just “Machine go brr”

WHAT YOU NOW KNOW

After making it through this course, I’d argue you’re better informed than 90% of people in the ML world (loosely defined).

WHAT YOU NOW KNOW

Most people in “AI” can use APIs. You understand what the APIs are doing.

WHAT I TRIED THIS YEAR

This is the first time I’ve taught the course this way.

The bet: start with probabilistic ML and Bayesian foundations before touching neural networks. Build the statistical scaffolding first, then show that every modern architecture is a special case.

WHAT I TRIED THIS YEAR

The arc: probability \(\to\) likelihood \(\to\) Bayesian inference \(\to\) “the stupid integral” \(\to\) curse of dimensionality \(\to\) PCA \(\to\) neural networks as nonlinear PCA \(\to\) CNNs \(\to\) the transfer paradigm \(\to\) generation from VAEs \(\to\) transformers \(\to\) autoregressive generation \(\to\) richest representations driving generation \(\to\) CLIP for omnimodal alignment \(\to\) the actual modern pipeline.

WHAT I THINK WORKED

The Bayesian foundations were in the right place. By the time we hit the ELBO in VAEs, you already knew what marginal likelihood was and why the integral is intractable. That’s a 10-lecture head start over most ML courses.

WHAT I’M GOING TO FIX

The first 6 lectures were too aggressive. Figure out the right pacing.

WHERE TO GO FROM HERE

If you want to go deeper:

• Read the original papers. You now have the vocabulary to understand them. Start with “Attention Is All You Need,” the CLIP paper, and the DDPM paper.

• Build something. Fine-tune a model for a problem you care about. Train a small diffusion model from scratch. Contribute to an open-source project.

• Take follow-up courses. Deep reinforcement learning, probabilistic graphical models, information theory, optimization theory.

The field is moving incredibly fast. What you learned this semester is the foundation. The specific architectures will evolve, but the underlying principles — representation learning, variational inference, attention, contrastive alignment — will remain.

The END

We started the semester with logistic regression.

We ended with Stable Diffusion.

The END

That’s the real lesson: complex systems are built from simple, well-understood parts.

Understanding those parts — not just using them, but truly understanding them — is what separates the people who advance the field from the people who use what the field produces.

The END

You now understand the parts.