DATASCI 447 Lecture 2: Probabilistic ML

Administrative Stuff

Discriminative Models

We’re going to start with a familiar class of learning methods and use these as the building blocks for new stuff

Discriminative Models

The predictive goal:

Find values of \(\hat{\boldsymbol \beta} \in \mathbb R^P\) that do a good job of predicting \(y\) for any viable feature combination \(\mathbf x\) that could occur

• e.g. not necessarily in the training data

Discriminative Models

\[ \hat{y} = g(\mathbf x^T \boldsymbol \beta^*) \]

Choose \(\boldsymbol \beta^*\) that makes \(\hat{y} \approx y\)

How do we do that?

Discriminative Models

Without specification of a loss function, we have no meaningful way to judge how good our predictive model is.

Discriminative Models

Our goal, then, is to find the coefficients that minimize the loss!

Discriminative Models

The predictive goal revolves around minimizing the true risk or generalization error

Assume each instance of \(\{ \mathbf x, y\}\) is drawn from the Nature box:

\[ P(X,Y) = \mathcal T \]

The true risk that we want to minimize is:

\[ E_{\mathcal T}[L(y , \mathbf x^T \beta^*)] \]

Can we directly minimize this?

Discriminative Models

We don’t know the Nature box!

• So we can’t compute the true loss directly.

Discriminative Models

\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} \frac{1}{N} \sum \limits_{i = 1}^N L(y_i , \mathbf x_i^T \boldsymbol \beta^*) \]

Is this the actual loss that we want to minimize when building a good predictive model?

Discriminative Models

1. Analytical approximations like the AIC/BIC when the model variance has a known form
2. Simulation methods
   1. Cross validation
   2. Out of bag errors (with bagging)
   3. Single Train/Validation splits when \(N\) is large enough

Discriminative Models

So, the last thing we need to do is specify the loss function

Discriminative Models

If \(y\) is a continuous outcome, the most common choice is mean squared error:

\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} \frac{1}{N} \sum \limits_{i = 1}^N (y_i - \theta_i^*)^2 \]

\[ \theta_i^* = \mathbf x_i^T \boldsymbol \beta^* \]

This admits the standard ordinary least squares solution:

\[ \hat{\boldsymbol \beta} = (\mathbf X^T \mathbf X)^{-1} \mathbf X^T \mathbf y \]

Discriminative Models

This isn’t the only choice - we could fit to minimize mean absolute error:

\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} \frac{1}{N} \sum \limits_{i = 1}^N |y_i - \theta_i^*| \]

\[ \theta_i^* = \mathbf x_i^T \boldsymbol \beta^* \]

• No closed form solution, but can be solved using computational optimization techniques

Question: What does MAE do that MSE does not? When is this a good loss function to use?

Discriminative Models

When our outcome is binary (0/1), the most common choice is to use binary cross entropy after transforming the linear predictor via the sigmoid function (e.g. logistic regression):

\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} -\frac{1}{N} \sum \limits_{i = 1}^N \left[ y_i \log \theta_i + (1 - y_i)\log(1 - \theta_i) \right] \]

\[ P(y_i = 1 | \boldsymbol \beta^*, \mathbf x) = \theta_i^* = \sigma(\mathbf x_i^T \boldsymbol \beta^*) \]

Discriminative Models

When our outcome is categorical ( \(y_i \in (1,2,...,K)\) ), the most common choice is cross entropy after transforming the predictor via the softmax function (e.g. multinomial logistic regression):

\[ \hat{\mathbf W} = \min_{\mathbf{W}} \mathcal{L}(\mathbf{W}) = - \frac{1}{N} \sum_{i=1}^N \sum_{k=1}^K \mathbb{1}(y_i = k) \log \theta_{ik} \]

\[ P(y_i = k | \mathbf W^*, \mathbf x) = \theta_{ik} = \frac{\exp[z_{ik}]}{\sum \limits_{h = 1}^K \exp[z_{ih}]} \]

\[ z_{ik} = \mathbf x_i^T \mathbf w_k \]

where \(\mathbf W\) is a \(P \times K\) matrix where the \(k^{th}\) column contains coefficients for the \(k^{th}\) class.

Discriminative Models

More broadly, we can define these loss functions as the minimization targets for any function \(f_{\theta}(\mathbf x)\) that maps the inputs to outputs:

• Linear models, random forests, boosted trees, neural networks all fall under this framework!

Discriminative Models

These loss functions are the basis of most discriminative machine learning.

Probabilistic Learning

It turns out that you’ve been doing it the whole time without even realizing it

Probabilistic Learning

Our \(N\) training instances form an empirical distribution from which we are trying to learn a reasonable functional approximation

Probabilistic Learning

The empirical conditional distributions that we learn say that:

\[ P(Y = y_i | X = x_i) = 1 \]

• All we observe is a single instance for each value of \(x\)

Taken at face value, the data we observe is indicative of what we’ll always see!

• But, we know that this isn’t true.

Probabilistic Learning

\[ y = f_{\theta}(\mathbf x) + \epsilon \]

where \(\epsilon\) is a draw from the noise distribution

• Idiosyncratic errors that are made due to randomness or inability of our current feature set to explain the variation

Probabilistic Learning

It turns out that your choice of loss function also tells us what the distribution of that noise is!

• In turn, tells us the distribution of the outcome conditional on the input

To show this, we’re going to start with an assumption and back our way into the identity

Probabilistic Learning

Since adding a constant to a distribution changes only the expectation:

\[ y = f_{\theta}(\mathbf x) + \epsilon \]

implies

\[ y \sim \mathcal N(y | f(\mathbf x) , \sigma^2) \]

• Each \(y\) is a draw from a normal distribution dictated by the input and noise

Probabilistic Learning

The goal: find the normal distributions that are most likely to have generated the observed outcomes!

Probabilistic Learning

Goal: Find a form of the conditional distribution that is most likely to have generated the data

Key Assumption: Each \(y | x\) is an independent and identically distributed draw.

Probabilistic Learning

Given a set of \(y\) values and corresponding \(\mathbf x\) values and a choice of function, our goal is to find the conditional distribution type such that:

\[ \underset{\theta}{\max} \prod \limits_{i = 1}^N P(y_i | \mathbf x_i , \boldsymbol \theta) \]

where \(\boldsymbol \theta\) is the set of unknowns that we’re trying to learn

Probabilistic Learning

This approach to estimating the values of the unknowns is called maximum likelihood estimation

• Given an assumption about the distribution that generated the data, find values for the unknowns that maximize the joint probability/likelihood that the data was generated from the distributional structure

Probabilistic Learning

Back to our normal assumption:

\[ \underset{\{\theta, \sigma^2\}}{\max} \sum \limits_{i = 1}^N \log \mathcal N(y_i | f_{\theta}(\mathbf x_i) , \sigma^2) \]

The normal distribution:

\[ \frac{1}{2 \pi \sigma^2} \exp \left[- \frac{1}{2 \sigma^2} (y_i - f_{\theta}(\mathbf x_i))^2 \right] \]

Taking the log:

\[ - \log 2 \pi \sigma^2 - \frac{1}{2 \sigma^2} (y_i - f_{\theta}(\mathbf x_i))^2 \]

Probabilistic Learning

Now the log of the joint:

\[ \sum \limits_{i = 1}^N - \log 2 \pi \sigma^2 - \frac{1}{2 \sigma^2} (y_i - f_{\theta}(\mathbf x_i))^2 \]

Making some simplifications:

\[ - \log 2 \pi \sigma^2 - \frac{1}{2 \sigma^2 N} \sum \limits_{i = 1}^N (y_i - f_{\theta}(\mathbf x_i))^2 \]

Probabilistic Learning

Wait - the only part where \(\theta\) comes in is in the summation

• \(\sigma\) is a part of the normalizing constant since we assume \(\sigma\) is constant across observations

\[ \underset{\{\theta\}}{\max} - Z \frac{1}{N} \sum \limits_{i = 1}^N(y_i - f_{\theta}(\mathbf x_i))^2 \]

Probabilistic Learning

By choosing to minimize the MSE, we’re implicitly choosing values for unknowns that will maximize the probability that they were generated from a normal distribution

Probabilistic Learning

Probabilistic Learning

Linear regression via MLE under MSE Loss

\[ \hat{y} = \mathbf x_i^T \boldsymbol \beta \]

\[ y \sim \mathcal N(y | \mathbf x_i^T \boldsymbol \beta , \sigma^2) \]

We’ll skip the derivation here because it’s going to look familiar:

\[ \hat{\boldsymbol \beta} = (\mathbf X^T \mathbf X)^{-1} \mathbf X^T \mathbf y \]

\[ \hat{\sigma^2} = \frac{1}{N} \sum \limits_{i = 1}^N (y_i - \mathbf x_i^T \hat{\boldsymbol \beta})^2 \]

Probabilistic Learning

Probabilistic Learning

Probabilistic Learning

Log-likelihoods reward overlapping the bumps with the empirical spikes

Log-likelihoods punish us for leaving spikes in low-density regions

e.g. Log-likelihoods punish us when our “loss” is big!

Probabilistic Learning

This distribution to loss relationship runs even deeper…

Probabilistic Learning

Probabilistic Learning

For classification, assume \(y | x\) follows a Bernoulli distribution and \(f(\mathbf x)\) is contrained to be between 0 and 1:

\[ P(y_i | \mathbf{x}_i) = f(\mathbf{x}_i)^{y_i} (1 - f(\mathbf{x}_i))^{1 - y_i} \]

Log-likelihood:

\[ \ell(\theta) = \sum_{i=1}^N \left[ y_i \log f_{\theta}(\mathbf{x}_i) + (1 - y_i) \log (1 - f_{\theta}(\mathbf{x}_i)) \right] \]

Negative log-likelihood:

\[ -\ell(\theta) = -\sum_{i=1}^N \left[ y_i \log f_{\theta}(\mathbf{x}_i) + (1 - y_i) \log (1 - f_{\theta}(\mathbf{x}_i)) \right] \]

Familiar?

Probabilistic Learning

A note about the Bernoulli/Normal:

MLE estimation under constant variance assumption (or constant functional assumption) makes the variance of the resulting conditional distributions a function of the mean function

Normal: \[ V[X] = \sigma^2 \]

Bernoulli:

\[ V[X] = f(\mathbf x_i) \left[1 - f(\mathbf x_i) \right] \]

This will matter soon!

Probabilistic Learning: Taking Stock

We’ve established that:

• Your choice of loss function implicitly specifies a conditional distribution \(p_{\theta}(y | x)\)
• Minimizing loss = Maximizing log-likelihood
• We fit models by finding \(\theta\) that makes the observed data most probable

The Fundamental Problem

We have two distributions:

Why We Need a Distance Measure

• We want \(p_{\theta}(y|x)\) to be similar to \(\hat{p}(y|x)\)
• But distributions are functions, not points in space!
• We can’t just use Euclidean distance
• We need a measure that respects the probabilistic structure

From Likelihood to Divergence

We’ve been maximizing: \[ \max_{\theta} \sum_{i=1}^N \log p_{\theta}(y_i | x_i) \]

But what are we really doing?

Why Do We Need a Distance Between Distributions?

• We have an empirical distribution \(\hat{p}(y | x)\) (the data)
• We have a model distribution \(p_{\theta}(y | x)\) (our approximation)
• We want to measure how “close” these distributions are
• But distributions are functions, not points!
• We need a proper divergence measure

Kullback-Leibler (KL) Divergence

The KL divergence from distribution \(Q\) to distribution \(P\) is:

\[ D_{KL}(P \| Q) = \sum_{y} P(y) \log \frac{P(y)}{Q(y)} \]

Or for continuous distributions:

\[ D_{KL}(P \| Q) = \int P(y) \log \frac{P(y)}{Q(y)} \, dy \]

Properties:

• Always non-negative: \(D_{KL}(P \| Q) \geq 0\) (Gibbs inequality)
• Equals zero only when \(P = Q\)
• Not symmetric: \(D_{KL}(P \| Q) \neq D_{KL}(Q \| P)\)

KL Divergence: Intuition

\[ D_{KL}(P \| Q) = \mathbb{E}_{y \sim P} \left[ \log \frac{P(y)}{Q(y)} \right] \]

Interpretation:

• Expected difference in log-probability between \(P\) and \(Q\)

• When \(P(y)\) is high but \(Q(y)\) is low → large penalty

• Measures the “information lost” when approximating \(P\) with \(Q\)

Asymmetry matters! We typically minimize \(D_{KL}(\hat{p} \| p_{\theta})\) where \(\hat{p}\) is the true/empirical distribution and \(p_{\theta}\) is our model.

KL Divergence

Key Insight: When means differ, P assigns high probability where Q assigns low probability → large penalty!

KL Divergence

Key Insight: P (narrower) is more concentrated. Q (wider) under-represents the peak → penalty!

KL Divergence

Key Insight: When P ≈ Q, divergence is near zero. Distributions are nearly indistinguishable!

From Empirical to Model Distribution

In supervised learning, our empirical distribution is:

\[ \hat{p}(y | x) = \frac{1}{n} \sum_{i=1}^n \delta(\mathbf x - x_i, y - y_i) \]

(A sum of delta spikes at the observed data points)

We want to find \(\theta\) that minimizes:

\[ D_{KL}(\hat{p} \| p_{\theta}) = \sum_{x,y} \hat{p}(y|x) \log \frac{\hat{p}(y|x)}{p_{\theta}(y|x)} \]

Simplifying the KL Divergence

\[ D_{KL}(\hat{p} \| p_{\theta}) = \sum_{x,y} \hat{p}(y|x) \log \hat{p}(y|x) - \sum_{x,y} \hat{p}(y|x) \log p_{\theta}(y|x) \]

• First term: Entropy of the empirical distribution (constant w.r.t. \(\theta\))
• Second term: Negative expected log-likelihood under the empirical distribution
• Minimizing KL divergence = Maximizing the second term

The Key Result

\[ \min_{\theta} D_{KL}(\hat{p} \| p_{\theta}) \equiv \max_{\theta} \sum_{x,y} \hat{p}(y|x) \log p_{\theta}(y|x) \]

Since \(\hat{p}(y|x) = \frac{1}{n} \sum_{i=1}^n \delta(x - x_i, y - y_i)\):

\[ \max_{\theta} \sum_{x,y} \hat{p}(y|x) \log p_{\theta}(y|x) = \max_{\theta} \frac{1}{n} \sum_{i=1}^n \log p_{\theta}(y_i | x_i) \]

Minimizing KL divergence is equivalent to Maximum Likelihood Estimation!

Why MLE Works

1. We want our model distribution \(p_{\theta}\) to be close to the true distribution
2. KL divergence measures this “closeness”
3. Minimizing KL divergence w.r.t. \(\theta\) drops constant terms
4. What remains is maximizing \(\sum_{i=1}^n \log p_{\theta}(y_i | x_i)\)
5. This is exactly the log-likelihood!

Bottom line: Maximum Likelihood Estimation is the principled way to fit probabilistic models by minimizing distributional distance.

Why Does KL Divergence Matter if MLE Does the Same Thing?

Three key reasons we need to understand KL divergence:

Why Does KL Divergence Matter if MLE Does the Same Thing?

2. Generative Models & Distribution Matching

• GANs, diffusion models, and normalizing flows all try to match a learned distribution to a target
• Understanding forward vs. reverse KL (\(D_{KL}(p \| q)\) vs \(D_{KL}(q \| p)\)) explains why different models behave differently
• Mode-seeking vs. mode-covering behavior comes directly from KL asymmetry

Why Does KL Divergence Matter if MLE Does the Same Thing?

3. Model Comparison & Selection

• Information criteria (AIC, BIC, DIC) are all based on KL divergence
• Cross-entropy loss in neural networks is minimizing KL divergence
• Understanding the distributional basis helps us design better loss functions for specific problems

Coming soon: Bayesian methods, deep generative models, and uncertainty quantification!

Under Pressure

Under Pressure

Maximizing the log-likelihood introduces a pressure on the model.

• Since all we see is the empirical distribution, our goal is to place as much mass as possible at the observed locations

• Respecting the assumed conditional distribution structure and the constraint that the mean of the distribution follows the mean function

For regression - \(f_{\theta}(\mathbf x) = \mathbf x^T \boldsymbol \beta\)

Under Pressure

Under Pressure

One way to increase the flexibility of the linear model is to add polynomial terms.

Question: What’s going to happen to the MLE estimate of \(\sigma^2\) when we add more terms?

\[ \hat{\sigma^2} = \frac{1}{N} \sum \limits_{i = 1}^N (y_i - \mathbf x_i^T \hat{\boldsymbol \beta})^2 \]

Under Pressure

There’s a more unifying way to think about this concept without relying on polynomials alone

• For nested classes of models (moving from the simplest model to the most complex), increasing the curvature budget (e.g. flexibility) necessarily decreases the MLE estimate of \(\sigma^2\).

Under Pressure

Under Pressure

Under Pressure

Under Pressure

Under Pressure

Under Pressure

Under Pressure

As the flexibility of our model increases, the pressure that comes from minimizing the KL divergence with the empirical distribution dominates

• Simple models combat the KL divergence pressure by limiting how complex the mean function can be

• More complex models allow more complex mean functions meaning that there is more opportunity to place all of the mass of the conditional distributions exactly on the observed data points

Does this sound like a familiar concept?

Under Pressure

Finally, a definition of overfitting that makes sense:

Overfitting occurs due to probabilistic mass collapse around the empirical distribution

• We take our empirical distribution a little too literally and end up saying that nothing else can actually happen!

Mitigating Pressure

One thing that is always going to reduce the mass collapse behavior is increasing the size of the training data

• As \(N \rightarrow \infty\), the empirical distribution approaches the true distribution!

Mitigating Pressure

Mitigating Pressure

Mitigating Pressure

Check-in question:

Imagine that our feature space is very high dimensional. What’s going to happen to the number of training examples needed to fill the space to make the empirical distribution \(\approx\) the true distribution?

Mitigating Pressure

Since there are some cases where getting an appropriate \(N\) is not possible, there are other strategies.

One thing that we can do is add a competing penalty to our loss function that will combat the overfitting problem.

Natural for the MSE - add a Ridge penalty:

\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta}{\min} \frac{1}{N} \|\mathbf y - \mathbf X \boldsymbol \beta \|^2 + \lambda \|\boldsymbol \beta\|^2 \]

Mitigating Pressure

With the MSE + Ridge penalty, there is a unique minimizer:

\[ \hat{\boldsymbol \beta} = (\mathbf X^T \mathbf X + \lambda \mathcal I_P)^{-1} \mathbf X^T \mathbf y \]

It turns out that the implied distribution when adding a L2 penalty is still normal! (more next time)

\[ P(y | \mathbf x, \boldsymbol \beta, \lambda) = \mathcal N(y | \mathbf x^T \hat{\boldsymbol \beta}, \hat{\sigma^2}) \]

where \(\hat{\sigma^2}\) is still the MSE.

Mitigating Pressure

Mitigating Pressure

Mitigating Pressure

Mitigating Pressure

Mitigating Pressure

Mitigating Pressure

Regularization works in the probabilistic sense by forcing mass in the conditional distribution away from the observed points.

• Unless the sample size warrants the certainty!

Mitigating Pressure

Turns out any of them could have generated that data

• Probability, baby! Anything can happen.

This is a fundamental problem with attempting to solve inverse problems

• Use generated data to learn about the truth

Why assume simplicity?

Mitigating Pressure

Next time:

• Regularization is an assumption about the way the world really works.

• Balances prior belief about a “good” solution with what the data says

• Fits naturally in our distributional world