January 15, 2026
We’re going to start with a familiar class of learning methods and use these as the building blocks for new stuff
As with most applied statistics, we’re going to start with the workhorse - linear regression
The linear regression model says:
\[ \hat{y} = g(\mathbf x^T \hat{\boldsymbol \beta}) \]
The predictive goal:
Find values of \(\hat{\boldsymbol \beta} \in \mathbb R^P\) that do a good job of predicting \(y\) for any viable feature combination \(\mathbf x\) that could occur
Linear regression methods represent one specific class of potential predictive functions that could be used
\[ \hat{y} = g(\mathbf x^T \boldsymbol \beta^*) \]
Choose \(\boldsymbol \beta^*\) that makes \(\hat{y} \approx y\)
How do we do that?
Without specification of a loss function, we have no meaningful way to judge how good our predictive model is.
Loss function
A function \(L(\cdot)\) that takes in the truth and a prediction
Returns 0 if truth = prediction
Returns larger positive values as predictions move further from the truth.
Our goal, then, is to find the coefficients that minimize the loss!
Empirical (average) loss
For a training set, \(\{ \mathbf X, \mathbf y\}_{i = 1}^N\), the empirical loss can be computed as:
\[ \frac{1}{N} \sum \limits_{i = 1}^N L(y_i , \hat{y}_i) \]
For linear regression:
\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} \frac{1}{N} \sum \limits_{i = 1}^N L(y_i , \mathbf x_i^T \boldsymbol \beta^*) \]
Conceptual question: Is this the actual loss that we want to minimize when building a good predictive model?
The predictive goal revolves around minimizing the true risk or generalization error
Assume each instance of \(\{ \mathbf x, y\}\) is drawn from the Nature box:
\[ P(X,Y) = \mathcal T \]
The true risk that we want to minimize is:
\[ E_{\mathcal T}[L(y , \mathbf x^T \beta^*)] \]
Can we directly minimize this?
We don’t know the Nature box!
\[ \text{True Risk} = \text{Empirical Risk} + \mathcal{O}\left(\frac{C}{N}\right) \]
For a model with \(N\) training examples and a fixed complexity, the minimum generalization error can be found where the empirical risk is smallest
There exists a balance between empirical risk and complexity (e.g. the Bias/Variance tradeoff)
Larger \(N\) means that the gap between the empirical risk and the true risk is small
\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} \frac{1}{N} \sum \limits_{i = 1}^N L(y_i , \mathbf x_i^T \boldsymbol \beta^*) \]
Is this the actual loss that we want to minimize when building a good predictive model?
Kinda!
Given a class of models, find the one that minimizes the empirical risk
Use methods to approximate the true risk conditional on the empirical risk
What approaches exist for approximating the true risk?
But, the game is still to minimize the empirical loss based on the training data.
So, the last thing we need to do is specify the loss function
The common choices we make depend on the type of outcome we are trying to predict
If \(y\) is a continuous outcome, the most common choice is mean squared error:
\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} \frac{1}{N} \sum \limits_{i = 1}^N (y_i - \theta_i^*)^2 \]
\[ \theta_i^* = \mathbf x_i^T \boldsymbol \beta^* \]
This admits the standard ordinary least squares solution:
\[ \hat{\boldsymbol \beta} = (\mathbf X^T \mathbf X)^{-1} \mathbf X^T \mathbf y \]
This isn’t the only choice - we could fit to minimize mean absolute error:
\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} \frac{1}{N} \sum \limits_{i = 1}^N |y_i - \theta_i^*| \]
\[ \theta_i^* = \mathbf x_i^T \boldsymbol \beta^* \]
Question: What does MAE do that MSE does not? When is this a good loss function to use?
When our outcome is binary (0/1), the most common choice is to use binary cross entropy after transforming the linear predictor via the sigmoid function (e.g. logistic regression):
\[ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta^*}{\min} -\frac{1}{N} \sum \limits_{i = 1}^N \left[ y_i \log \theta_i + (1 - y_i)\log(1 - \theta_i) \right] \]
\[ P(y_i = 1 | \boldsymbol \beta^*, \mathbf x) = \theta_i^* = \sigma(\mathbf x_i^T \boldsymbol \beta^*) \]
When our outcome is categorical ( \(y_i \in (1,2,...,K)\) ), the most common choice is cross entropy after transforming the predictor via the softmax function (e.g. multinomial logistic regression):
\[ \hat{\mathbf W} = \min_{\mathbf{W}} \mathcal{L}(\mathbf{W}) = - \frac{1}{N} \sum_{i=1}^N \sum_{k=1}^K \mathbb{1}(y_i = k) \log \theta_{ik} \]
\[ P(y_i = k | \mathbf W^*, \mathbf x) = \theta_{ik} = \frac{\exp[z_{ik}]}{\sum \limits_{h = 1}^K \exp[z_{ih}]} \]
\[ z_{ik} = \mathbf x_i^T \mathbf w_k \]
where \(\mathbf W\) is a \(P \times K\) matrix where the \(k^{th}\) column contains coefficients for the \(k^{th}\) class.
More broadly, we can define these loss functions as the minimization targets for any function \(f_{\theta}(\mathbf x)\) that maps the inputs to outputs:
| MSE | \(\frac{1}{N} \sum \limits_{i = 1}^N (y_i - f_{\theta}(\mathbf x_i))^2\) |
| MAE | \(\frac{1}{N} \sum \limits_{i = 1}^N |y_i - f_{\theta}(\mathbf x_i)|\) |
| BCE | \(-\frac{1}{N} \sum \limits_{i = 1}^N y_i \log f_{\theta}(\mathbf x_i) + (1- y_i) \log (1 - f_{\theta}(\mathbf x_i))\) |
| CE | \(\frac{1}{N} \sum \limits_{i = 1}^N \sum \limits_{k = 1}^K \mathbb 1(y_i = k) \log f_{\theta}(\mathbf x_i)\) |
These loss functions are the basis of most discriminative machine learning.
From last class, our goal is to learn something about the Nature box
Where do distributions come in?
It turns out that you’ve been doing it the whole time without even realizing it
Key point:
Your choice of loss function implicitly chooses a conditional distribution for the outcome given the inputs
That distribution has:
Width
Tails
Support (viable set of values)
that dictate how your model handles mistakes
Our \(N\) training instances form an empirical distribution from which we are trying to learn a reasonable functional approximation
The empirical conditional distributions that we learn say that:
\[ P(Y = y_i | X = x_i) = 1 \]
Taken at face value, the data we observe is indicative of what we’ll always see!
\[ y = f_{\theta}(\mathbf x) + \epsilon \]
where \(\epsilon\) is a draw from the noise distribution
Therefore
\[ y \neq f_{\theta}(\mathbf x) \]
and we assume that noise follows some structure
\(E[\text{Noise}] = 0\)
\(V[\text{Noise}] = \sigma^2\)
Right?
It turns out that your choice of loss function also tells us what the distribution of that noise is!
To show this, we’re going to start with an assumption and back our way into the identity
Let’s assume that:
\[ f(\epsilon) \sim \mathcal N(\epsilon | 0, \sigma^2) \]
Since adding a constant to a distribution changes only the expectation:
\[ y = f_{\theta}(\mathbf x) + \epsilon \]
implies
\[ y \sim \mathcal N(y | f(\mathbf x) , \sigma^2) \]
The goal: find the normal distributions that are most likely to have generated the observed outcomes!