DATASCI 447 Lecture 3: Regularization Is Bayesian

Administrative Stuff

What We Saw Last Time

Let’s discuss:

• How does minimizing a loss function correspond to probability distributions?

• What is an empirical distribution? How does MLE fit to the empirical distribution?

• What does minimizing MSE correspond to in distributional terms? Binary cross entropy?

• For linear models fit under MSE, what happens to \(\sigma^2\) as model complexity increases under MSE?

• Why does MLE create pressure toward overfitting?

What We Saw Last Time

What is overfitting from a probabilistic perspective?

• Really important!

Under Pressure

Under Pressure

Maximizing the log-likelihood introduces a pressure on the model.

• Since all we see is the empirical distribution, our goal is to place as much mass as possible at the observed locations

• Respecting the assumed conditional distribution structure and the constraint that the mean of the distribution follows the mean function

For regression - \(f_{\theta}(\mathbf x) = \mathbf x^T \boldsymbol \beta\)

Under Pressure

Under Pressure

The Key Problem

MLE methods solve a problem:

\[ \underset{\theta}{\max} \sum \limits_{i = 1}^N P(y_i | \mathbf x_i , \theta) \]

• Find the conditional distributions defined by our hypothesis class that minimize the KL divergence to the empirical distribution

The Key Problem

Collapse is not bad behavior. It is the optimizer doing its job

• If I give a dog a whole bag of food at once, I shouldn’t be mad if it eats it all!

• The minimization algorithm is just minimizing the function to the best of its ability by placing as much mass as possible at the observed data points.

The Key Problem

Question 1: Why do we even think that this is a bad thing?

The Key Problem

The big one:

• We have some beliefs about the way that world works.

• We know that what we see is just a sample and may not be indicative of the true empirical distribution due to lack of coverage

Zero Beliefs Imply Degenerate Solutions

Central claim:

If we truly had zero beliefs about how the world works, the degenerate MLE solution is correct.

Bayesian Inference

Bayes’ rule for parameters given data:

\[ p(\theta | \mathcal D) = \frac{p(\mathcal D | \theta) p(\theta)}{p(\mathcal D)} \]

Components:

• \(p(\mathcal D | \theta)\): Likelihood - probability of data given parameters
• \(p(\theta)\): Prior - beliefs about \(\theta\) before seeing data
• \(p(\theta | \mathcal D)\): Posterior - beliefs about \(\theta\) after seeing data
• \(p(\mathcal D)\): Evidence - normalizing constant (marginal likelihood/model evidence). We’re going to avoid this today, but it is going to be a central quantity next lecture!

Bayesian Inference

The posterior:

\(p(\theta | \mathcal D)\)

is a proper probability density function

• Defines a distribution over the values of the unknowns conditioned on the data we observed

• Mean, variance, covariance, etc.

Bayesian Inference

The Maximum A Posteriori (MAP) estimate is defined as:

\[ \hat{\theta} = \max_{\theta} P(\mathcal D | \theta) P(\theta) \]

• Drop the normalizing constant because we’re maximizing w.r.t. \(\theta\) and it doesn’t have anything to do with it

• Think of this like a Bayesian MLE…

The Flat Prior

Suppose we use a flat (improper) prior:

\[ p(\theta) \propto 1 \quad \text{(constant everywhere)} \]

Maximum A Posteriori (MAP) estimation:

\[ \hat \theta_{\text{MAP}} = \underset{\theta}{\max} p(\theta | \mathcal D) = \underset{\theta}{\max} p(\mathcal D | \theta) p(\theta) \]

What Does a Flat Prior Mean?

\(p(\theta) \propto 1\) says:

• All parameter values are equally plausible a priori
• No preference for simple vs complex models
• No structural assumptions about the problem
• No domain knowledge encoded
• Complete ignorance about \(\theta\)

Collapse is Rational Under Zero Information

If we truly have no beliefs about \(\theta\):

• Posterior \(\propto\) Likelihood
• MAP = MLE
• Mass collapse is the correct Bayesian inference

The Philosophical Punch

Objecting to overfitting is objecting to agnosticism.

Bayesian Inference as Constrained Probability Geometry

We’ve established that we need priors to avoid collapse.

Now: How do priors work mechanically?

Bayes’ Rule (Structural Form)

Start from Bayes’ rule:

\[ p(\theta | \mathcal D) = \frac{p(\mathcal D | \theta) p(\theta)}{p(\mathcal D)} \]

The Negative Log Posterior

Multiply by \(-1\):

\[ -\log p(\theta | \mathcal D) = -\log p(\mathcal D | \theta) - \log p(\theta) + \text{const} \]

Posterior Energy Decomposition

\[ -\log p(\theta | \mathcal D) = \underbrace{\mathcal{L}_N(\theta)}_{\text{Data/Likelihood}} + \underbrace{\mathcal{R}(\theta)}_{\text{Regularization/Prior}} + \text{const} \]

Data term \(\mathcal{L}_N(\theta)\):

• Negative log-likelihood
• Measures how well model explains observed data
• Smaller when model fits data well

Regularization term \(\mathcal{R}(\theta)\):

• Negative log-prior
• Measures complexity/implausibility of \(\theta\)
• Smaller when \(\theta\) aligns with prior beliefs

Geometric Interpretation

Geometric Interpretation

Geometric Interpretation

What the Prior Encodes

The prior \(p(\theta)\) encodes structural beliefs:

• Magnitude: Expected size of parameters
• Sparsity: How many parameters should be nonzero
• Smoothness: Relationships between neighboring parameters
• Constraints: Hard boundaries (e.g., \(\theta > 0\))
• Correlations: Dependencies between parameters
• Domain knowledge: Problem-specific structure

The Critical Identity

Regularization is not “added later.”

Sample Size Changes Everything

So far, we’ve established:

• Likelihood pushes toward mass collapse
• Prior pushes toward structure
• Posterior balances the two

The Scaling Question

Recall the negative log posterior:

\[ -\log p(\theta \mid \mathcal D) = \mathcal{L}_N(\theta) + \mathcal{R}(\theta) + \text{const} \]

where

\[ \mathcal{L}_N(\theta) = -\sum_{i=1}^N \log p(y_i \mid \mathbf x_i, \theta) \]

\[ \mathcal{R}(\theta) = -\log p(\theta) \]

How Do These Terms Scale?

Key observation:

• The likelihood term \(\mathcal{L}_N(\theta)\) grows with sample size
• The prior term \(\mathcal{R}(\theta)\) does not

Likelihood Scaling

Each observation contributes a term:

\[ -\log p(y_i \mid \mathbf x_i, \theta) \]

So the total likelihood penalty is:

\[ \mathcal{L}_N(\theta) = \sum_{i=1}^N \ell_i(\theta) \]

This means:

\[ \mathcal{L}_N(\theta) = O(N) \]

Prior Scaling

The prior is specified once:

\[ p(\theta) \]

Its contribution to the objective is:

\[ \mathcal{R}(\theta) = -\log p(\theta) \]

This does not depend on \(N\).

\[ \mathcal{R}(\theta) = O(1) \]

The Asymmetry

Putting this together:

\[ \mathcal{L}_N(\theta) = O(N), \quad \mathcal{R}(\theta) = O(1) \]

As \(N\) grows, the likelihood term dominates the posterior geometry.

Consequence for MAP

Recall MAP estimation:

\[ \hat\theta_{\text{MAP}} = \min_\theta \big[ \mathcal{L}_N(\theta) + \mathcal{R}(\theta) \big] \]

MAP Converges to MLE

Under standard regularity conditions:

\[ \hat\theta_{\text{MAP}} \;\longrightarrow\; \hat\theta_{\text{MLE}} \]

Priors disappear asymptotically.

Interpretation

• Priors matter most when data are scarce
• Priors matter least when data are abundant
• Sample size determines which force wins

Sample size is the ultimate regularizer.

Finite Samples Are the Real Problem

Regularization exists because:

• \(N\) is finite
• Data do not cover the space
• Inverse problems are ill-posed

Without priors, inference is unstable.

High Dimensions Make This Worse

In modern ML:

• \(N\) may be large
• \(P\) may be larger
• Effective sample size per parameter is small (e.g. curse of dimensionality)

This is why regularization does not “go away” in practice.

• Even when we have millions of samples, we’re still going to rely on regularization via priors to assist in finding structured models!

A Concrete Example

Random values, \(x_i\), that we believe are generated from a normal distribution with variance 1

• Goal: Learn the mean of the normal distribution using the data

A Concrete Example

Real Priors for ML

This is great and all, but most of the time we aren’t placing priors on the values of parameters in our models because we have beliefs as to what they are

• A lot of the time our models don’t even explicitly have coefficients (think tree methods and KNN)

• But the lessons about priors still hold here

So what gives?

Real Priors for ML

In practice, there are two common strategies for regularizing function spaces:

• Truncating the hypothesis class

• Placing structural priors on parameters to encourage nice solution types

Truncating the Hypothesis Class

One way to regularize is to remove functions from consideration entirely.

Truncation in Practice

Many standard ML methods are function-class truncation in disguise.

Structural Priors on Function Space

In practice, we often write models in linear form:

\[ f_\theta(\mathbf x) = \mathbf x^\top \boldsymbol\beta \]

What We Actually Believe

Given limited data, we often believe:

• Large coefficients are unlikely
• Extreme cancellations are fragile
• Many reasonable explanations are near the origin
• Simpler functions should be preferred unless data demand otherwise

These are beliefs about functions, expressed through parameters.

Independent Normal Priors

A common structural assumption:

\[ \beta_j \sim \mathcal N(0, \tau^2) \quad \text{independently} \]

Probabilistic Geometry of the Prior

Probabilistic Geometry of the Prior

Quick identity: the prior predictive distribution

\[ P(y | \mathbf x) = \int \limits_{\theta} P(y | \mathbf x, \theta) P(\theta) d(\theta) \]

• With respect to the prior, what is the marginal distribution of \(y\) given a vector \(\mathbf x\).

• We can easily simulate this distribution

Probabilistic Geometry of the Prior