DATASCI 447 Lecture 1: Introduction

Welcome to DATASCI 447!

This is a class that is going to provide an overview of the statistical methods that underpin a number of more advanced machine learning techniques

• The goal of this class is to provide a statistical/mathematical introduction to topics in machine learning that extend on those methods learned in your introductory class

Pre-requisites

For this class, I am expecting that students have taken a class on:

• Linear regression (Like DATASCI 220)

• Machine Learning (like DATASCI 347)

• Calculus 1-3

• Linear algebra

I don’t expect you to remember everything from these classes, but I do expect that when you see certain concepts they aren’t completely foreign.

Lectures

We’re going to meet here on Tuesdays and Thursdays from 2:30 PM - 3:45 PM

• There is no formal attendance requirement for this course, but you should show up

• Be willing to stop me if there’s something that isn’t clear. If it’s not clear to you, then I’m sure there’s someone else in class who is also confused.

• It’s a relatively small class, so I have no problem spending time re-clarifying points that may not have been well made.

Office Hours

I’ll have two office hours periods this semester:

• Wednesdays 2:30 - 3:45

• And one other time that I’ll announce next week

All office hours will occur in my office - PAIS 579

Group Problem Sets

Over the course of the semester, there will be 6-7 problems sets. These will account for 50% of your final grade.

• Implement and extend the materials discussed in class.

• Introduce software and coding

• Derivations and Proofs

Problem Sets will be where you get your applied practice

• Lectures will largely center on the theory of why things work

• Problem sets will relate more to the how

Group Problem Sets

All problem sets should be submitted to the appropriate Canvas assignment as two files:

• The raw notebook file (.qmd, .ipynb)

• A rendered version of the notebook (.html or .pdf)

Each problem set will be posted in a variety of different formats, so feel free to use this as a template for your final solutions.

Group Problem Sets

All problem sets in this class can be completed in groups of, at most, 3 students

• Each student should turn in a copy of the solutions, but can be identical to group’s solutions.

• All collaborators must be outlined at the top in the by-line

• I recommend the same group each time - repeated vs. one-shot games of trust.

Since it is an upper level course, I’m creating a mechanism for getting rid of shirkers

• If they don’t contribute equally, get rid of ’em.

You can complete assignments individually.

I can also assign you a group, if you’d like. Just email me and I’ll put people together who want a group.

Final Project

The other 50% of your final grade will be determined by a final project

• A significant project that applies methods discussed in this class

• Up to you (and your group) exactly what this is

Do something interesting!

• Projects that are too simple will not be accepted

• No basic comparisons of methods

• Really try to answer a question that’s interesting to you

Final Project

Three checkpoints:

• On March 26th, teams will present a single slide outlining their final project (10%)

• On April 25th, teams will present project posters at QTM’s end of semester showcase (20%)

• By May 5th, each student should submit a final paper about the final project. A scientific-styled paper no more than 15 pages and a nicely formatted Github codebase (10% for deliverable, 10% for Github)

Final Project

The goal of this final project is to give you something to include in your portfolio as you apply for jobs or grad school

• Maybe start something that turns into a bigger project later

Coding

I’ll be using Python for this class.

Coding

My setup:

• Python 3.10

• VSCode

• Jupyter Notebooks/Quarto

Coding

As we progress through this class, we’re going to get to methods that are computationally demanding

• Often too demanding for your laptop

If you don’t have a machine with a discrete GPU, you’ll be using Google Colab

• I highly recommend paying $10 a month for Colab Pro during this class

• Gives access to better GPUs and priority time on them

• If $10 a month presents a problem, let me know.

Coding

We’ll be using PyTorch throughout this class

• Software for general purpose optimization

• The dominant software for deep learning

Using nice utility functions from PyTorch Lightning

• A really nice library that sits over PyTorch that organizes code

• Automatically detects and uses GPUs and other accelerators when available

Coding

Please use Github Copilot/Codex/Etc.!

• This is a machine learning course an these LLMs are a modern marvel!

So many of the annoyances of coding with Python are gone when you use code LLMs

• Matplotlib syntax

• Documentation searches

• scikit-learn nuances

• Pytorch modules and training loops

If you have a Github account, get it student certified and you’ll be able to use Copilot for free

• I think the $20 ChatGPT pro license/Google AI license is money well spent

The Goals of this Class

The topics I hope to cover this semester are outlined on the syllabus.

• The pace is ambitious.

• If we need to take more time on topics, we will adjust.

The Goals of this Class

Quick note: I’m trying something new this semester and changing the lens through which I’m presenting the materials in this class to make generative machine learning a common thread

• Not your typical deep learning class

A part of this is that optimization doesn’t fit in to our lecture flow all that well

The Goals of this Class

Any questions?

Machine Learning

What is a machine learning algorithm?

Tasks

Machine learning is so popular because it can be used to address many different questions/achieve many different tasks.

Two basic supervised starting points:

Regression - given a vector of input features, \(\mathbf X\), and an outcome vector, \(\mathbf y \in \mathbb R\), output a function \(\hat{f}(\mathbf x)\) that accurately predicts the value of the outcome given \(\mathbf x\).

• Use inputs to predict the value of a continuous outcome.

Tasks

Machine learning is so popular because it can be used to address many different questions/achieve many different tasks.

Two basic supervised starting points:

Classification - given a vector of input features, \(\mathbf X\), and an outcome vector, \(\mathbf y \in (1,2,...,K)\), output a function \(\hat{f}(\mathbf x)\) that accurately predicts the class given \(\mathbf x\).

• Use inputs to predict the value of a countable and finite set of outcomes.

Tasks

These two basic tasks largely describe what you’ve seen previously.

Tasks

How do we get from there to here?

Tasks

Or here?

Which Face is Real?

Tasks

Or here?

Tasks

Or here?

The World is A Simulation?

Unfortunately, this toolkit will only get us so far

• We need something else

The World is A Simulation?

Imagine a Nature Box that outputs data points

• Collections of pixels in an image and the label

• Words in the written English language

The World is A Simulation?

We don’t know \(P(X,Y)\)

• But we can take samples

• Pull the lever and one instance drops out!

The World is A Simulation?

The joint distribution \(P(X,Y)\) is the holy grail

• If I knew this, I could answer any question about the data (at least up to probability)

  • How does pixel 1 correlate with pixel 277?

  • If I say “The dog is”, what is the probability that “barking” is the next word?

  • What is the distribution of pixels in an image that contains a cat?

The problem is that this distribution is often high dimensional and scary looking

The Fork in the Road

\[ P(X,Y) = P(Y | X) \cdot P(X) \]

• Path A (The Discriminative Path): We only care about \(P(Y | X)\) and we throw away \(P(X)\)

  • Shortcuts the problem. Perfectly fine if our end goal is predicting \(y\) given some set of input features.

  • Useless for simulation

• Path B (The Generative Path): We model \(P(X)\) directly

  • Harder

  • Requires modeling the geometry of the data manifold

Black and White Images

Grayscale images are just a matrix of pixel values

• Each pixel is between 0 (black) and 255 (white)

• Often renormalized between 0 and 1

RGB Images

RGB Images are three matrixes stacked on each other

• Each pixel is a combination of three pixel values between 0 and 255

RGB Images

CIFAR-10 Images: 32 x 32 RGB Images

RGB Images

A Linear Discriminator

\[P(y = 1 | \mathbf x) = \theta = \sigma(\mathbf w^T \mathbf x)\]

where:

\[\sigma(z) = \frac{1}{1 + \exp[-z]}\]

To finish out the simulator:

\[y \sim \text{Bernoulli}(\theta)\]

A Linear Discriminator

Training Logistic Regression on 3072 features...
Logistic Regression Accuracy (Plane vs Frog): 0.8880

A Linear Discriminator

This is an example of a very good linear classifier

• 88% accuracy on the held out validation set
• Very generalizable

It can classify quite well!

Making Frogs

Let’s find the most “froggy” of all the frogs

• Maximize the probability that it belongs to the frog class

We renormalized the pixel values to be between 0 and 1

• Make any pixels that contribute positively to the probability a 1 and make those that contribute negatively a 0

• Or translate the weight vector to RGB to get the expected frog

Making Frogs

The expected frog:

Making Frogs

The most froggy of all the frogs:

Making Frogs

Neither of those look like a frog at all…

Can anyone tell me what’s going on here?

Making Frogs

Bayes’ Theorem, baby!

\[P(\mathbf x | y = 1) = \frac{P(y = 1 | \mathbf x) P(\mathbf x)}{\int \limits_{\mathbf x} P(y = 1 | \mathbf x) P(\mathbf x) d\mathbf x}\]

High Dimensional Distributions

We have 3072 pixels and we need to come up with some method to approximate the distribution that generated these pixels

For now, we’re going to take a shortcut and try to learn:

\[P(\mathbf x | y = \text{Frog})\]

directly using the CIFAR data set.

High Dimensional Distributions

When in doubt, use the workhorse of multidimensional statistics - The Multivariate Normal Distribution

\[f(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^k |\boldsymbol{\Sigma}|}} \exp\left(-\frac{1}{2}(\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})\right)\]

where:

• \(\mathbf{x} \in \mathbb{R}^k\) is the input vector (3072 pixels).
• \(\boldsymbol{\mu} \in \mathbb{R}^k\) is the mean vector (3072 values).
• \(\boldsymbol{\Sigma} \in \mathbb{R}^{k \times k}\) is the covariance matrix (a 3072 x 3072 matrix).
• \(k\) is the dimensionality (3072).

High Dimensional Distributions

Given 1000 samples of frogs in the CIFAR dataset, we can use maximum likelihood estimation to figure out the parameters of the best MVN approximation given our empirical distribution

\[ \hat{\boldsymbol{\mu}} = \frac{1}{N} \sum_{i=1}^N \mathbf{x}_i \]

\[ \hat{\boldsymbol{\Sigma}} = \frac{1}{N} \sum_{i=1}^N (\mathbf{x}_i - \hat{\boldsymbol{\mu}})(\mathbf{x}_i - \hat{\boldsymbol{\mu}})^T \]

High Dimensional Distributions

The distribution mode:

High Dimensional Distributions

9 draws from our distribution:

High Dimensional Distributions

Those don’t look like frogs at all!

• If you squint and turn your head to the right - kinda…

This is an incredibly high dimensional problem that leads to a number of “traps”

The Emptiness Inside

Trap 1: Parameter Explosion and Emptiness

We have 5000 frog images

The Emptiness Inside

Most pixels show around zero covariance with other pixels!

The Emptiness Inside

The issue here is that our 3072 dimension feature space is a big empty box.

• 5,000 samples of frogs does nothing to tell us anything about the true distribution of pixels for frogs

This is referred to as the curse of dimensionality

The Emptiness Inside

The Emptiness Inside

The Emptiness Inside

At 3,072 dimensions, 5,000 images is like dropping 5,000 grains of sand into the Pacific Ocean and trying to map the sea floor!

The Emptiness Inside

A truth:

As the number of samples increases, our ability to use the empirical distribution to approximate the true distribution increases

The Emptiness Inside

The Emptiness Inside

Our training data is just a collection of small spikes in an ocean of possibilities

• The empirical distribution puts 1/N probability on exactly images we saw and 0 probability everywhere else!

• What we saw is all there is and any slight variation is impossible…

The Emptiness Inside

To fix this, we need to accept that our data points are just samples - not the whole truth

• We need to smooth the distribution

• A slightly less green frog is still a frog…

The act of smoothing or filling in the gaps based on assumptions is called Regularization

The Emptiness Inside

Prior Beliefs

In regression, you added a penalty term \(\lambda \|\boldsymbol \beta\|^2\) to prefent overfitting

• You probably thought of this as shinking coefficients.

The Linearity Trap

Let’s suppose that we had wayyyyy more images of frogs (think billions upon billions)

The Linearity Trap

The Linearity Trap

The frog manifold:

The Linearity Trap

There is a very small segment of the pixel space where all the pixels align in a way that is just right to give us a frog!

Difficulties

This is a tough problem.

• But doable.

Roadmap

1. A bit of distributional fitting theory
2. Bayesian Inference
3. Extending PCA
4. Deep Learning
5. Basic Generation with VAEs
6. CNNs
7. Transformers
8. Modern Generative Methods