DATASCI 447 Lecture 5: Why Local Methods Fail (a.k.a. The Curse of Dimensionality)

Administrative Stuff

ML is Geometry

Introductory ML materials present ML as minimizing loss

• Not training error - true loss

• Regularization as a method that creates a tuning knob that allows models to vary between high bias/low variance to low bias/high variance and use CV methods to find the setting that best generalizes (bias/variance tradeoff)

ML is Geometry

Probabilistic ML allows us to begin to think about this process less as a hand-wavy mechanical one and more through the lens of probability theory and model geometry

Minimizing the negative log likelihood (NLL) is equivalent to minimizing the loss.

• If we directly minimize the NLL for a model with infinite capacity (no limit to flexibility), what solution will it find?

Regularization is a way to restrict the space of functions that can be recovered by a learning method by placing priors on the function space

• If we add a Ridge penalty to a loss function, how does the regularization constant \(\lambda\) restrict the function space? How does this relate to restricting model capacity/flexibility?

ML is Geometry

To jointly find \(\mathbf M\) and \(\bbeta\) that are most generalizable over all possible models within our model class:

\[ \{\hat{\mathbf M}, \hat{\bbeta} \} = \max_{\mathbf M} \int \limits_{\btheta} P(\mathbf y | \mathbf X, \bbeta , \mathbf M) P(\bbeta | \mathbf M) d \bbeta \]

\[ \{\hat{\mathbf M}, \hat{\bbeta} \} = \max_{\mathbf M} E_{\bbeta} \left[ P(\mathbf y | \mathbf X, \bbeta , \mathbf M) \right] \]

• Find the arrangement of \(\mathbf M\) and \(\bbeta\) that maximize the likelihood that we would’ve seen our observed data accounting for our prior structural constaints

• Also can think of it as maximize the expected value of the likelihood taken w.r.t. \(\bbeta\)

ML is Geometry

Solving this integral is going to be critical for creating good generative models

• It’s the only way that we’ll be able to train machines that can generate new images!

The Practical Goal of ML

In practice, we want learning methods that:

1. Generalize well to unseen data
2. Can be estimated and tuned with minimal human intervention
3. Scale to problems where hand-tuning is impossible

This isn’t just theoretical preference — it’s practical necessity

The Practical Goal of ML

Medical Imaging: Millions of scans across diverse populations

• Can’t hand-tune a model for each patient
• Need methods that automatically adapt to signal in data

The Practical Goal of ML

Language Models: Trillions of tokens of text

• No human could write rules for grammar, reasoning, knowledge
• Must extract structure directly from data

What Do We Need From a Model Class?

Given this goal, what properties should our model class \(\mathcal{F}\) have?

What Do We Need From a Model Class?

Property 3 has a name: Universal Approximation

Universal Approximation: Formal Definition

Definition: A class of functions \(\mathcal{F}\) is a universal approximator on a compact domain \(\mathcal{X} \subseteq \mathbb{R}^p\) if for any continuous function \(f^*: \mathcal{X} \to \mathbb{R}\) and any \(\epsilon > 0\), there exists \(f \in \mathcal{F}\) such that:

\[\sup_{x \in \mathcal{X}} |f(x) - f^*(x)| < \epsilon\]

Universal Approximation: Formal Definition

What this definition says:

• The function class \(\mathcal{F}\) is “rich enough” to approximate anything continuous
• The approximation is uniform — we can get close everywhere simultaneously
• There are no “blind spots” where our function class fundamentally cannot reach

Formulation 1: Density in Function Space

\(\mathcal{F}\) is a universal approximator if it is dense in the space of continuous functions

Intuition: No matter what continuous function you pick, you can find members of \(\mathcal{F}\) arbitrarily close to it

There are no “gaps” in what \(\mathcal{F}\) can represent

• Pick any target function
• There’s always something in \(\mathcal{F}\) nearby
• Get as close as you want by searching harder in \(\mathcal{F}\)

Formulation 2: The Complexity Path

A model class is a universal approximator if it admits a complexity path — a nested sequence:

\[\mathcal{F}_{\lambda_1} \subseteq \mathcal{F}_{\lambda_2} \subseteq \cdots \subseteq \mathcal{F}\]

such that:

1. Base case: \(\mathcal{F}_{\lambda_1}\) contains simple functions (e.g., constants)

2. Monotonic expressivity: Higher complexity \(\lambda\) = larger function class

3. Limiting completeness: \(\bigcup_\lambda \mathcal{F}_\lambda\) is dense in continuous functions

Intuition: Universal approximators come with a “knob” we can turn from simple to arbitrarily complex

Formulation 3: The Interpolation View

For finite data \(\{(x_i, y_i)\}_{i=1}^n\) with distinct \(x_i\), a universal approximator can achieve:

\[\hat{f}(x_i) = y_i \quad \text{for all } i = 1, \ldots, n\]

That is: zero training error is always achievable with sufficient complexity

Often the most practically relevant formulation: “Can we at least fit the training data?”

Which Methods Are Universal Approximators?

What methods do you know that have this property?

Which Methods Are Universal Approximators?

Implication 1: Expressivity is Not the Bottleneck

If \(\mathcal{F}\) is a universal approximator, then the true function \(f^*\) (assuming it’s continuous) is in the closure of \(\mathcal{F}\)

What this means:

We’re not limiting ourselves by our choice of function class

The “right answer” is in there somewhere

What this doesn’t mean:

We can actually find it

Implication 2: Zero Training Error is Achievable

With sufficient complexity, we can always fit the training data perfectly

This sounds good:

If we can’t even fit the training data, we certainly can’t generalize

But it’s also dangerous:

Interpolating noisy data means memorizing noise

Zero training error \(\neq\) zero test error

Implication 3: We Can Recover the Truth…

Universal approximation combined with consistency results tells us:

\[\hat{f}_n \xrightarrow{P} f^* \quad \text{as } n \to \infty\]

What this says: With infinite data, we can recover the truth

What this leaves open:

• How fast?
• How does the rate depend on the problem?
• What happens when \(n\) is finite (i.e., always)?

The Approximation Guarantee

What universal approximation gives us:

With enough data and capacity, this learner could learn the true function

This is a statement about existence

• There exists a good approximation in our function class
• The answer is “in there”

The Learning Problem

What we actually face:

Given finite, noisy samples \(\{(x_i, y_i)\}_{i=1}^n\) where \(y_i = f^*(x_i) + \epsilon_i\)

Find \(\hat{f} \in \mathcal{F}\) such that \(\hat{f} \approx f^*\)

The Bias-Variance Tradeoff

How do we navigate a universal approximator in practice?

The Bias-Variance Tradeoff

Bias — decreases with complexity: \[\text{Bias}^2 = O\left(\lambda\right)\]

• Simple models (small \(\lambda\)): Can’t capture the true function → high bias
• Complex models (large \(\lambda\)): Can approximate \(f^*\) well → low bias

Variance — increases with complexity, decreases with sample size: \[\text{Variance} = O\left(\frac{\lambda}{N}\right)\]

• More complexity → more parameters to estimate → higher variance
• More samples → better estimates → lower variance
• The ratio \(\lambda / N\) is what matters

The Bias-Variance Tradeoff

Cross-Validation Finds the Best on the Path

How do we find the optimal complexity \(\lambda^*\) in practice?

Cross-Validation Finds the Best on the Path

This is exactly what CV is supposed to do — and it does it well!

When the Path Floats

The ideal scenario: Path dips close to optimal

CV finds the minimum, and it’s close to the best possible

When the Path Floats

The problematic scenario: Entire path floats high

CV finds the minimum of the path — but that minimum is still far from good

Why Does the Path Float?

Is it possible to know how far the path is going to float above the optimum?

• What conditions do you think lead there to be a big gap between the “best” and the best?

Stone’s Minimax Theorem

Theorem (Stone, 1982): For nonparametric regression in \(\mathbb{R}^P\) with \(f^*\) having smoothness \(S\) (the number of continuous derivatives for the function):

\[ \inf_{\hat{f}} \sup_{f^* \in \mathcal{C}^s} \mathbb{E}\|\hat{f} - f^*\|^2 = \mathcal O\left(\frac{1}{N^{\frac{2S}{2S+P}}}\right) \]

What this says:

• The \(\inf\) is over ALL possible estimators (best case over all possible learning methods)
• The \(\sup\) is over all functions with smoothness \(s\) (worst possible true function of smoothness \(s\))
• This is the best achievable rate — no method can do better

Stone’s Minimax Theorem

\[ \mathcal O\left(\frac{1}{N^{\frac{2S}{2S+P}}}\right) \]

Goal: Make as small as possible. What should we make big? What should we make small?

The Curse of Dimensionality

\[\text{Optimal Rate} = O\left(N^{-\frac{2S}{2S+P}}\right)\]

As dimension \(P\) increases:

• The exponent \(\frac{2S}{2S+P}\) gets smaller
• The rate gets slower
• We need more samples to achieve the same error

The Curse of Dimensionality

Same sample size, vastly different achievable error rates

The Curse is Fundamental

Key insight: Stone’s theorem is a lower bound

This isn’t a statement about any particular method:

• Not about KNN being bad
• Not about trees being bad
  
• Not about polynomials being bad

The Floating Path

\[R(\hat{f}_{\lambda^*}) = \sigma^2 + O\left(N^{-\frac{2S}{2S+P}}\right)\]

When \(P\) is small relative to \(N\):

• The rate is fast (e.g., \(N^{-2/3}\) for \(p=1\))
• The path dips close to \(\sigma^2\)
• CV finds a good model

When \(P\) is large relative to \(N\):

• The rate is glacially slow (e.g., \(N^{-1/51}\) for \(p=100\))
• The path floats high above \(\sigma^2\)
• CV finds the best on the path… which is still bad

The Floating Path

A truth:

No learning algorithm can beat this rate!

• Most algorithms don’t even get close to this rate! It requires an essentially optimal learner to get there.

Different Problems, Different Performance

Different Problems, Different Performance

Loading CIFAR-10 dataset...
Dataset size: 50000 images
Image shape: 32×32×3 = 3,072 dimensions
Number of classes: 10

Dimensionality: 3,072 features (32×32×3 RGB pixels)
Training samples: 50000 images

Contender 1: Polynomial Regression

Suppose we have \(P\) features, we can implement a degree \(d\) polynomial in \(P\) dimensions with all interactions

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_1^2 + \beta_3 x_2 + \beta_4 x_2^2 + ... \]

Stone-Weierstrass theorem proves that the path of \(d \in (0,1,2,...,\infty)\) is a universal approximator

• Main gist: If I have \(N\) points, there is always some polynomial expansion that will have at least \(N\) terms and exactly interpolate the data

Even though polynomial regression has the universality property, why do you think that it is quickly going to crash and burn?

Contender 1: Polynomial Regression

The \(P\) to \(N\) ratio is going to get out of control very quickly!

For a degree \(d\) model with \(P\) features, there will be:

\[ \frac{(P + d)!}{P!d!} \]

parameters

Contender 1: Polynomial Regression

I didn’t run this one for CIFAR because it quickly gets computationally unruly

This presents us with the main reason that we avoid explicitly parameterized models:

• The number of coefficients needed to effectively capture nonlinearity and dependencies explodes quickly!

Note: Kernelized methods (like SVMs) get around this issue by using implicit representations of high degree feature spaces. However, they always have \(P = N\). With reasonable \(N\), the gap between the “best” and the best will be quite large. Also, a computational nightmare!

Contender 2: KNN

The Setup

For a test point, \(\mathbf x_0\), compute \(y_0\) as:

• The average of the \(y_i\) associated with the \(K\) nearest \(\mathbf x_i\) in the training set (continuous outcomes)

• The most common \(y_i\) associated with the \(K\) nearest \(\mathbf x_i\) in the training set (categorical outcomes)

Complexity Path

• \(K = N\): Global mean/mode (max bias)
• \(K = 1\): Interpolation (zero training error)

Choose \(K\) that minimizes CV error.

Contender 2: KNN

Why do you think that KNN is going to crash and burn as \(P\) gets larger?

Contender 2: KNN

Contender 2: KNN

KNN quickly degrades in high dimensions

• Performance is quite bad

• Computational nightmare too

Contender 3: Trees

The Setup

Recursively partition feature space with axis-aligned splits

• At each node, find the feature and threshold that best separates the data
• Predict constant value (mean/mode) in each leaf

Complexity Path

• Stump (depth 1): Single split (max bias)
• Full tree: One leaf per observation (interpolation)

Choose depth/leaves that minimizes CV error

Contender 3: Trees

Why do you think that trees are going to crash and burn as \(P\) gets larger?

Contender 3: Trees

But wait — what about Random Forests and XGBoost?

These are the workhorses of modern ML on tabular data!

Random Forests (Bagging):

• Train many trees on bootstrap samples
• Average predictions → reduces variance

Gradient Boosting (XGBoost, LightGBM):

• Sequentially fit trees to residuals
• Each tree corrects errors of the ensemble

Do these escape the curse?

Contender 3: Trees

How ensembles help:

• Variance reduction through averaging (RF) or gradual fitting (boosting)
• Implicit regularization through randomization
• Often the best “off-the-shelf” methods for tabular data

The Common Thread

All these methods share the same failure mode:

They work in the ambient feature space \(\mathbb{R}^P\)

• KNN: Computes distances in \(\mathbb{R}^P\)
• Trees: Partitions \(\mathbb{R}^P\) with axis-aligned splits
• Ensembles: Aggregates over partitions of \(\mathbb{R}^P\)

They rely on locality in that space

• Nearby points should have similar outputs
• Can carve up space into meaningful regions

When \(P\) is large: Locality breaks down, and all methods struggle

The Problem is Not the Methods

Important reframe:

These methods aren’t bad — they’re doing exactly what they should!

• KNN achieves near-optimal rates for its complexity class
• Trees/forests are provably consistent
• XGBoost is extremely well-engineered and consistently is the top performer on tabular data (even relatively high dimensional tabular data)

Reframing the Problem

Reframing the Problem

Instead of trying to make our learning methods better in \(\mathbb R^P\), what if we focused on reducing \(P\)?

• Decrease the dimensionality of the feature space

Reframing the Problem

Goal: Reduce the dimensionality of the feature space from \(P\) to \(K\) where \(K << P\).

But doesn’t that result in information loss?

Reframing the Problem

The Manifold Hypothesis: Real high-dimensional data doesn’t fill the ambient feature space!

Loading MNIST dataset...

MNIST images: 28×28 = 784 dimensions
Random images also: 784 dimensions
But only MNIST lives on a low-dimensional manifold!

Reframing the Problem

The Manifold Hypothesis: Real high-dimensional data doesn’t fill the ambient feature space!

• Random words \(\neq\) sentences

• Thousands of frequency bins, but random combinations aren’t speech

• Thousands of genes, but cells occupy limited phenotypic states

Reframing the Problem

The Hypothesis:

High dimensional data lies on or near a low dimensional manifold \(\mathcal M^K\) embedded in \(\mathbb R^P\)

• An object within the \(P\) space that locally looks like \(\mathbb R^K\)

Reframing the Problem

Reframing the Problem

How do we know that this is the case for images, text, and sound?

Loading MNIST dataset...

Reframing the Problem

Loading MNIST dataset...

Reframing the Problem

Compression is Possible!

• JPEGs reduce image size by 10-20x with minimal loss

• MP3 compresses audio dramatically

If we can compress, then we can represent the object in lower dimensional spaces

Reframing the Problem

Feature dependence is high

• In images: the color of a nearby pixel is largely the same as its surrounding pixels (we’re smooth color collections, not chaos)

• In text: word co-occurrences follow strong patterns

Reframing the Problem

The gap between \(P\) and \(K\) is enormous!

Reframing the Problem

The methods aren’t bad…

• We’re applying \(P\)-dimensional toolkits to a \(K\)-dimensional problem!

If data lies on \(\mathcal M^K \subset \mathbb R^P\):

• There exists a mapping \(\phi\): \(\mathbb R^P \rightarrow \mathbb R^K\) that captures the manifold structure

Working in \(K\) space instead of \(P\) space drastically reduces the gap of our best learners to the optimal solution

Reframing the Problem

\[ \mathcal O\left(N^{- \frac{2s}{2s + k}}\right) \text{ vs. } \mathcal O\left(N^{- \frac{2s}{2s + P}}\right) \]

To get the interior to .1:

• 30 dimensions requires 32k samples (CIFAR 10 has 60k!)

• 3072 requires around \(10^{154}\)!!!!

One is achievable, the other is not.

Next Time

We have a candidate tool for learning a data optimal mapping from \(\mathbb R^P \rightarrow \mathbb R^K\) - PCA

• Finds directions of max variance

• Finds the best \(K\)-dimensional linear subspace