DATASCI 447 Lecture 3: Regularization Is Bayesian

Administrative Stuff

What We Saw Last Time

Let’s discuss:

• How does minimizing a loss function correspond to probability distributions?

• What is an empirical distribution? How does MLE fit to the empirical distribution?

• What does minimizing MSE correspond to in distributional terms? Binary cross entropy?

• For linear models fit under MSE, what happens to \(\sigma^2\) as model complexity increases under MSE?

• Why does MLE create pressure toward overfitting?

What We Saw Last Time

What is overfitting from a probabilistic perspective?

• Really important!

Under Pressure

Under Pressure

Maximizing the log-likelihood introduces a pressure on the model.

• Since all we see is the empirical distribution, our goal is to place as much mass as possible at the observed locations

• Respecting the assumed conditional distribution structure and the constraint that the mean of the distribution follows the mean function

For regression - \(f_{\theta}(\mathbf x) = \mathbf x^T \boldsymbol \beta\)

Under Pressure

Under Pressure

The Key Problem

MLE methods solve a problem:

\[ \underset{\theta}{\max} \sum \limits_{i = 1}^N P(y_i | \mathbf x_i , \theta) \]

• Find the conditional distributions defined by our hypothesis class that minimize the KL divergence to the empirical distribution

The Key Problem

Collapse is not bad behavior. It is the optimizer doing its job

• If I give a dog a whole bag of food at once, I shouldn’t be mad if it eats it all!

• The minimization algorithm is just minimizing the function to the best of its ability by placing as much mass as possible at the observed data points.

The Key Problem

Question 1: Why do we even think that this is a bad thing?

The Key Problem

The big one:

• We have some beliefs about the way that world works.

• We know that what we see is just a sample and may not be indicative of the true empirical distribution due to lack of coverage

Zero Beliefs Imply Degenerate Solutions

Central claim:

If we truly had zero beliefs about how the world works, the degenerate MLE solution is correct.

Bayesian Inference

Bayes’ rule for parameters given data:

\[ p(\theta | \mathcal D) = \frac{p(\mathcal D | \theta) p(\theta)}{p(\mathcal D)} \]

Components:

• \(p(\mathcal D | \theta)\): Likelihood - probability of data given parameters
• \(p(\theta)\): Prior - beliefs about \(\theta\) before seeing data
• \(p(\theta | \mathcal D)\): Posterior - beliefs about \(\theta\) after seeing data
• \(p(\mathcal D)\): Evidence - normalizing constant (marginal likelihood/model evidence). We’re going to avoid this today, but it is going to be a central quantity next lecture!

Bayesian Inference

The posterior:

\(p(\theta | \mathcal D)\)

is a proper probability density function

• Defines a distribution over the values of the unknowns conditioned on the data we observed

• Mean, variance, covariance, etc.

Bayesian Inference

The Maximum A Posteriori (MAP) estimate is defined as:

\[ \hat{\theta} = \max_{\theta} P(\mathcal D | \theta) P(\theta) \]

• Drop the normalizing constant because we’re maximizing w.r.t. \(\theta\) and it doesn’t have anything to do with it

• Think of this like a Bayesian MLE…

The Flat Prior

Suppose we use a flat (improper) prior:

\[ p(\theta) \propto 1 \quad \text{(constant everywhere)} \]

Maximum A Posteriori (MAP) estimation:

\[ \hat \theta_{\text{MAP}} = \underset{\theta}{\max} p(\theta | \mathcal D) = \underset{\theta}{\max} p(\mathcal D | \theta) p(\theta) \]

What Does a Flat Prior Mean?

\(p(\theta) \propto 1\) says:

• All parameter values are equally plausible a priori
• No preference for simple vs complex models
• No structural assumptions about the problem
• No domain knowledge encoded
• Complete ignorance about \(\theta\)

Collapse is Rational Under Zero Information

If we truly have no beliefs about \(\theta\):

• Posterior \(\propto\) Likelihood
• MAP = MLE
• Mass collapse is the correct Bayesian inference

The Philosophical Punch

Objecting to overfitting is objecting to agnosticism.

Bayesian Inference as Constrained Probability Geometry

We’ve established that we need priors to avoid collapse.

Now: How do priors work mechanically?

Bayes’ Rule (Structural Form)

Start from Bayes’ rule:

\[ p(\theta | \mathcal D) = \frac{p(\mathcal D | \theta) p(\theta)}{p(\mathcal D)} \]

The Negative Log Posterior

Multiply by \(-1\):

\[ -\log p(\theta | \mathcal D) = -\log p(\mathcal D | \theta) - \log p(\theta) + \text{const} \]

Posterior Energy Decomposition

\[ -\log p(\theta | \mathcal D) = \underbrace{\mathcal{L}_N(\theta)}_{\text{Data/Likelihood}} + \underbrace{\mathcal{R}(\theta)}_{\text{Regularization/Prior}} + \text{const} \]

Data term \(\mathcal{L}_N(\theta)\):

• Negative log-likelihood
• Measures how well model explains observed data
• Smaller when model fits data well

Regularization term \(\mathcal{R}(\theta)\):

• Negative log-prior
• Measures complexity/implausibility of \(\theta\)
• Smaller when \(\theta\) aligns with prior beliefs

Geometric Interpretation

Geometric Interpretation

Geometric Interpretation

What the Prior Encodes

The prior \(p(\theta)\) encodes structural beliefs:

• Magnitude: Expected size of parameters
• Sparsity: How many parameters should be nonzero
• Smoothness: Relationships between neighboring parameters
• Constraints: Hard boundaries (e.g., \(\theta > 0\))
• Correlations: Dependencies between parameters
• Domain knowledge: Problem-specific structure

The Critical Identity

Regularization is not “added later.”

Sample Size Changes Everything

So far, we’ve established:

• Likelihood pushes toward mass collapse
• Prior pushes toward structure
• Posterior balances the two

The Scaling Question

Recall the negative log posterior:

\[ -\log p(\theta \mid \mathcal D) = \mathcal{L}_N(\theta) + \mathcal{R}(\theta) + \text{const} \]

where

\[ \mathcal{L}_N(\theta) = -\sum_{i=1}^N \log p(y_i \mid \mathbf x_i, \theta) \]

\[ \mathcal{R}(\theta) = -\log p(\theta) \]

How Do These Terms Scale?

Key observation:

• The likelihood term \(\mathcal{L}_N(\theta)\) grows with sample size
• The prior term \(\mathcal{R}(\theta)\) does not

Likelihood Scaling

Each observation contributes a term:

\[ -\log p(y_i \mid \mathbf x_i, \theta) \]

So the total likelihood penalty is:

\[ \mathcal{L}_N(\theta) = \sum_{i=1}^N \ell_i(\theta) \]

This means:

\[ \mathcal{L}_N(\theta) = O(N) \]

Prior Scaling

The prior is specified once:

\[ p(\theta) \]

Its contribution to the objective is:

\[ \mathcal{R}(\theta) = -\log p(\theta) \]

This does not depend on \(N\).

\[ \mathcal{R}(\theta) = O(1) \]

The Asymmetry

Putting this together:

\[ \mathcal{L}_N(\theta) = O(N), \quad \mathcal{R}(\theta) = O(1) \]

As \(N\) grows, the likelihood term dominates the posterior geometry.

Consequence for MAP

Recall MAP estimation:

\[ \hat\theta_{\text{MAP}} = \min_\theta \big[ \mathcal{L}_N(\theta) + \mathcal{R}(\theta) \big] \]

MAP Converges to MLE

Under standard regularity conditions:

\[ \hat\theta_{\text{MAP}} \;\longrightarrow\; \hat\theta_{\text{MLE}} \]

Priors disappear asymptotically.

Interpretation

• Priors matter most when data are scarce
• Priors matter least when data are abundant
• Sample size determines which force wins

Sample size is the ultimate regularizer.

Finite Samples Are the Real Problem

Regularization exists because:

• \(N\) is finite
• Data do not cover the space
• Inverse problems are ill-posed

Without priors, inference is unstable.

High Dimensions Make This Worse

In modern ML:

• \(N\) may be large
• \(P\) may be larger
• Effective sample size per parameter is small (e.g. curse of dimensionality)

This is why regularization does not “go away” in practice.

• Even when we have millions of samples, we’re still going to rely on regularization via priors to assist in finding structured models!

A Concrete Example

Random values, \(x_i\), that we believe are generated from a normal distribution with variance 1

• Goal: Learn the mean of the normal distribution using the data

A Concrete Example

Real Priors for ML

This is great and all, but most of the time we aren’t placing priors on the values of parameters in our models because we have beliefs as to what they are

• A lot of the time our models don’t even explicitly have coefficients (think tree methods and KNN)

• But the lessons about priors still hold here

So what gives?

Real Priors for ML

In practice, there are two common strategies for regularizing function spaces:

• Truncating the hypothesis class

• Placing structural priors on parameters to encourage nice solution types

Truncating the Hypothesis Class

One way to regularize is to remove functions from consideration entirely.

Truncation in Practice

Many standard ML methods are function-class truncation in disguise.

Structural Priors on Function Space

In practice, we often write models in linear form:

\[ f_\theta(\mathbf x) = \mathbf x^\top \boldsymbol\beta \]

What We Actually Believe

Given limited data, we often believe:

• Large coefficients are unlikely
• Extreme cancellations are fragile
• Many reasonable explanations are near the origin
• Simpler functions should be preferred unless data demand otherwise

These are beliefs about functions, expressed through parameters.

Independent Normal Priors

A common structural assumption:

\[ \beta_j \sim \mathcal N(0, \tau^2) \quad \text{independently} \]

Probabilistic Geometry of the Prior

Probabilistic Geometry of the Prior

Quick identity: the prior predictive distribution

\[ P(y | \mathbf x) = \int \limits_{\theta} P(y | \mathbf x, \theta) P(\theta) d(\theta) \]

• With respect to the prior, what is the marginal distribution of \(y\) given a vector \(\mathbf x\).

• We can easily simulate this distribution

Probabilistic Geometry of the Prior

Probabilistic Geometry of the Prior

The functional prior:

• At the edges of the domain for our input, we don’t know and don’t have many opinions about what the function should look like

• In the middle section where the bulk of the data will be our prior says that the function should essentially be flat

• e.g. minimal complexity

How This Fights Mass Collapse

Recall the collapse pressure:

• Likelihood wants to place arbitrarily high mass at observed points
• Flexible functions enable extreme curvature
• \(\hat\sigma^2 \to 0\) under MLE

The Gaussian prior pushes back by:

• Penalizing large coefficient norms
• Disfavoring sharply oscillatory functions
• Spreading mass away from pathological interpolants

This is soft regularization of function space.

Probabilistic Geometry of the Prior

Probabilistic Geometry of the Prior

Probabilistic Geometry of the Prior

Our prior structure on the functional space causes the straight line to have lower loss than the 9th degree fit!

• This is what we want because this is what we believe is true.

• If we’re wrong, we need more data relative to the strength of our prior to offset this belief

• The beautiful balance of Bayes’ works well for ML!

MAP Estimate Under Normal Priors

Okay. So now we know that this works.

• Now we need to figure out what the optimal solution is under this prior penalized loss.

MAP Estimate Under Normal Priors

The log prior:

\[ \log P(\boldsymbol \beta) \propto \sum \limits_{j = 1}^P - \frac{1}{2 \tau^2} \beta_j^2 \]

MAP Estimate Under Normal Priors

The prior penalized negative log likelihood under MSE with normal priors:

\[ \|\mathbf y - \mathbf X \boldsymbol \beta\|^2 + \frac{1}{2 \tau^2}\sum \limits_{j = 1}^P \beta_j^2 \]

Look familiar?

MAP Estimate Under Normal Priors

The Ridge regression you’ve been doing is inherently Bayesian!

• We choose a value of \(\lambda\)

• \(\lambda = \frac{1}{2 \tau^2}\) implies \(\tau^2 = \frac{1}{2 \lambda}\)

• This is the variance of our normal prior that is centered at 0!

Probabilistic Geometry of the Prior

Probabilistic Geometry of the Prior

Probabilistic Geometry of the Prior

MAP Estimate Under Normal Priors

Different day, same story

• The probabilistic story is the one that we’ve been using the whole time
• Sparsity and shrinkage aren’t “discovered” from the data - it is assumed in the prior geometry
• ML isn’t just CS and algorithms - it’s a commitment to a particular probabilistic worldview!
• Seeing this link is key to advancing beyond the basics

MAP Estimates

This relationship runs deep

• You’ll see a different example in problem set 2!

In general, our common regularization techniques that penalize functional complexity fall into this class of models:

• Ridge and LASSO

• Total variation priors for images

• Explicit 2nd derivative penalization for splines and kernel based methods

MAP Is Still a Point Estimate

So far, we have focused on:

\[ \hat\theta_{\text{MAP}} = \arg\max_\theta p(\theta \mid \mathcal D) \]

This gives us:

• A single parameter value
• A single function
• A single explanation of the data

But probability theory gave us a distribution, not a point.

What MAP Throws Away

The posterior contains:

\[ p(\theta \mid \mathcal D) \]

MAP keeps only:

\[ \max_\theta p(\theta \mid \mathcal D) \]

MAP discards:

• Posterior uncertainty
• Posterior shape
• Posterior volume
• Alternative explanations that are nearly as plausible

What MAP Throws Away

MAP answers:

“What is the most likely explanation?”

It does not answer:

“How many explanations are plausible?”

• Note that this is very closely related to the concept of generalization

• Small changes to the input don’t result in large changes to the output!

A Warning Sign

Suppose we have two models with the same MAP estimates:

• Model A: one very sharp posterior mode
• Model B: one flatter posterior mode

The Generative Goal

Our stated goal is not just prediction.

We want a generative model:

\[ p(x, y) \quad \text{or} \quad p(y \mid x) \]

that:

• Represents uncertainty
• Can generate new data
• Allows probabilistic reasoning
• Supports downstream inference

Why the Full Posterior Matters

Predictions under a Bayesian model are:

\[ p(y^\ast \mid x^\ast, \mathcal D) = \int p(y^\ast \mid x^\ast, \theta)\, p(\theta \mid \mathcal D)\, d\theta \]

MAP replaces this integral with:

\[ p(y^\ast \mid x^\ast, \hat\theta_{\text{MAP}}) \]

This ignores parameter uncertainty entirely.

The Missing Quantity

Everything we’ve done so far has avoided:

\[ p(\mathcal D) = \int p(\mathcal D \mid \theta)\, p(\theta)\, d\theta \]

This is the marginal likelihood (evidence).

It answers:

• How well does this model class explain the data?
• How much posterior mass exists?
• How much fine-tuning was required?

Where This Leaves Us

MAP was a necessary step.

But for generative modeling:

• Point estimates are insufficient
• Uncertainty is essential
• Model comparison matters