DATASCI 447 Lecture 4: What Probabilistic Methods Enable (e.g. solving the most important integral)

Administrative Stuff

A Little Review

Suppose we have a model where we are trying to learn a set of unknowns, \(\boldsymbol \theta\), given a model class, likelihood, and training data.

• What is the difference between the MLE and MAP estimator of the unknowns?

• When are these two estimators equivalent to one another?

• How does the MLE estimator fit with respect to the data? What about the MAP estimate with a non-degenerate prior?

A Little Review

We showed that Ridge regression as you have seen it previously is actually the MAP estimate assuming a prior structure on coefficients.

• What was that prior structure?

• How does that prior structure relate to the hyperparameter in Ridge regression?

• What did we mean by the idea that our prior wasn’t really on the coefficients but on functional form?

A Little Review

We can express Bayesian Linear Regression with Normal Priors as a learning problem:

\[ P(\boldsymbol \beta | \mathbf X , \mathbf y) = \frac{\mathcal N_N(\mathbf y | \mathbf X \boldsymbol \beta, \hat{\sigma}^2 \mathbf I) \mathcal N_P(\boldsymbol \beta | \mathbf 0 , \tau^2 \mathbf I)}{P(\mathbf y | \mathbf X)} \]

• Which part of Bayes’ theorem did we not need to think about when we were just thinking about MAP estimates?

A Little Review

Since \(\boldsymbol \beta\) is a continuous parameter vector, the posterior is over a continuous quantity. Therefore, the more common written version of the denominator is:

\[ P(\boldsymbol \beta | \mathbf X , \mathbf y) = \frac{\mathcal N_N(\mathbf y | \mathbf X \boldsymbol \beta, \hat{\sigma}^2 \mathbf I) \mathcal N_P(\boldsymbol \beta | \mathbf 0 , \tau^2 \mathbf I)}{\int \limits_{\boldsymbol \beta} \mathcal N_N(\mathbf y | \mathbf X \boldsymbol \beta, \hat{\sigma}^2 \mathbf I) \mathcal N_P(\boldsymbol \beta | \mathbf 0 , \tau^2 \mathbf I) d \boldsymbol \beta} \]

• Looks complicated (and it kinda is)

• What is the marginal probability that I would see our observed vector \(\mathbf y\) given \(\mathbf X\) weighted by my prior over \(\boldsymbol \beta\)?

Where We Are

From the past two lectures:

• Likelihood = probabilistic loss

• Priors = probabilistic regularization

• MAP = posterior mode/penalized point estimate

We can sum all of this up using (part of) Bayes’ theorem:

\[ P(\theta | \mathbf X) \propto P(\mathbf X | \theta) P(\theta) \]

Where We Are

\[ \hat{\bbeta} = \min_\bbeta[-\log P(\mathbf y | \bbeta, \mathbf X) - \log P(\bbeta)] \]

If:

\[ P(\bbeta) = \prod \limits_{j = 1}^P \mathcal N(\beta | 0 , \tau^2) \]

Then:

\[ \hat{\bbeta} = \min_\bbeta[-\log P(\mathbf y | \bbeta, \mathbf X) - \lambda \sum \limits_{j = 1}^P \beta_j^2] \]

• Our L2 penalty is a normal prior on coefficients

• Regularization strength is the prior variance

Why Worry?

The probabilistic framework gives us something that hand wavy heuristics cannot:

• Assumptions are explicit, consequences are easy to track

• Explains why regularization works the way it does in a way other than “having a small coefficient budget is good and bias/variance tradeoff”

• Creates explicit methods for understanding the geometry of the hypothesis space. See the prior distribution over functions by drawing from the prior and the learned distributions over functions (functional uncertainty) by taking draws from a posterior.

Why Worry?

This is neat and all

But I really haven’t shown you anything that Bayes’ can do that you couldn’t already do

• Why spend time on Bayes’ in a deep/generative learning class if it just re-describes existing practice?

Why Worry?

The promise:

Bayesian Methods are going to give us a way to unify the procedures we use to optimize to the penalized likelihood given a model and the procedures we use to choose the best model across models.

• It creates a one stop shop

• And it’s going to solve the generalization problem with an analytical solution

Conditional Inference

When we do statistical learning, we have some data and we would like to learn a function of the data that will achieve some goal in a generalizable way

• For prediction: Have low prediction error for unseen feature/outcome pairs

• For generation: Produce high clarity and realism images of frogs that were not in the original training data

Conditional Inference

Conditional Inference

But, MAP/MLE with a L2 penalty gives us:

\[ \hat{\btheta} = \min_\btheta \log P(\mathbf y | \btheta) + \lambda \|\btheta\|^2 \]

Conditional Inference

In statistical learning, what methods do we use to choose values for hyperparameters and select between models?

Conditional Inference

The standard ML recipe:

1. Choose a set of models that you want to test (separate hyperparameters inclusive)

2. For each model, find \(\hat{\btheta}\) conditional on the observed training data

3. Generate an estimate of the true model risk

4. Compare across everything and select the one with the lowest true model risk

Conditional Inference

CV methods provide good estimates of the true risk for a model class!

• They are really good at creating a score for each potential model and ranking them correctly w.r.t. to the true risk

Conditional Inference

Conditional Inference

Model class:

\[ \hat{\bbeta}_{\lambda} = \| y - \mathbf X \bbeta \|^2 + \lambda \|\boldsymbol \beta\|^2 \]

Across models:

Choose \(\lambda\) to minimize generalization score

Conditional Inference

Define the LOOCV score for a model class as:

\[ \text{LOOCV}(\lambda) = \frac{1}{N} \sum \limits_{i = 1}^N - \log P(y_i | \mathbf x_i, \hat{\bbeta}_{-i}) - \log P (\bbeta_{-i}) \]

where

\[ \hat{\bbeta}_{-i} = \min_\bbeta P(\mathbf y_{-i} | \mathbf X_{-i}, \bbeta) P(\bbeta) \]

• The average penalized negative log likelihood for each observation finding the MAP estimate using the other \(N - 1\) observations

Conditional Inference

Conditional Inference

Conditional Inference

Conditional Inference

Conditional Inference

Conditional Inference

Conditional Inference

Conditional Inference