Course contentsShow

Basics/Lesson 3 of 11

Linear Regression and Maximum Likelihood

A Probabilistic View

In the previous lesson, we chose squared error as the objective for linear regression because errors can be positive or negative.

In this lesson, we look at linear regression from a probabilistic point of view. We'll see that the same squared error objective comes out of a statistical method called maximum likelihood estimation.

Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) is a statistical method used in machine learning. It works by maximizing the likelihood function, which represents the probability of observing the data given specific parameter values.

Our dataset gives us both the inputs and the labeled outputs. So we try to maximize the likelihood of the actual output given the input.

Here is a small example. Suppose you want to estimate a coin's bias, so you flip it 100 times and get 60 heads.

MLE says the most likely probability of heads is $0.6$ , because that is the value that best explains your data.

The formula for the likelihood depends on the distribution the data comes from. Next, we'll apply this idea to linear regression.

Recap: The Linear Regression Model

We'll use the house price problem from the previous lesson. The inputs are the house area, number of bedrooms, number of toilets, and number of balconies. The output is the house price.

We have $n$ data samples:

(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)

Each $x_i$ is a $d$ -dimensional input, and each $y_i$ is a real value. The predicted output is:

f(x_i) = w_0 + \sum_{k=1}^{d} w_k x_i^{(k)}

As before, we collect the weights into a vector $w$ of $d + 1$ values, and add a 1 to each $x_i$ so it has $d + 1$ values too. The prediction then becomes:

f(x_i) = w^T x_i = x_i^T w

The error for the $i$ -th sample is the difference between the actual output and the predicted output:

e_i = y_i - f(x_i) = y_i - w^T x_i

On a plot, these errors are the distances from the data points to the line. Our objective so far has been to minimize the sum of the squared errors.

Assuming Gaussian Errors

Many lines are possible, but only the best one matches our data. If we have the right line, two things should be true about the errors:

They are symmetric around the line, so points are equally likely to fall above or below it.
Small errors are more likely than large ones.

This lets us assume the errors come from a Gaussian (normal) distribution with mean zero and variance $\sigma^2$ :

e_i \sim \mathcal{N}(0, \sigma^2)

The Distribution of the Output

Each actual output is the predicted output plus the error:

y_i = w^T x_i + e_i

For all $n$ samples together, $X$ is a matrix because each $x_i$ is a vector, while $y$ and $e$ are vectors of the outputs and errors:

y = Xw + e

The output $y_i$ depends on the error $e_i$ , which is Gaussian. So, given the input, $y_i$ is also Gaussian. Adding $w^T x_i$ shifts the mean, and the variance stays the same:

y_i \mid x_i \sim \mathcal{N}(w^T x_i, \sigma^2)

The Likelihood

Since the actual output follows a Gaussian distribution, we can use the Gaussian formula to write the likelihood of the actual output given the input:

p(y_i \mid x_i; w) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_i - w^T x_i)^2}{2\sigma^2}\right)

This is the likelihood for one sample. To cover the whole dataset, we also assume the errors are independent of each other, so the likelihood of all $n$ outputs is the product of the individual likelihoods:

L(w) = \prod_{i=1}^{n} p(y_i \mid x_i; w)

We want to find the $w$ that maximizes this likelihood.

The Log-Likelihood

For ease of computation, we take the logarithm. Maximizing the log of a function gives the same $w$ as maximizing the function itself, and the log turns the product into a sum:

\log L(w) = \sum_{i=1}^{n} \log \frac{1}{\sqrt{2\pi\sigma^2}} - \sum_{i=1}^{n} \frac{(y_i - w^T x_i)^2}{2\sigma^2}

From Likelihood to Squared Error

Now we simplify. Remember that our variable is $w$ .

The first term doesn't contain $w$ , so it's a constant and we can ignore it. We're left with maximizing:

-\sum_{i=1}^{n} \frac{(y_i - w^T x_i)^2}{2\sigma^2}

Maximizing a negative quantity is the same as minimizing the positive quantity:

\min_{w} \; \sum_{i=1}^{n} \frac{(y_i - w^T x_i)^2}{2\sigma^2}

The $2\sigma^2$ is also a constant, so we can remove it too:

\min_{w} \; \sum_{i=1}^{n} (y_i - w^T x_i)^2

This is the squared error objective from the previous lesson.

Summary

The key ideas are:

Maximum likelihood estimation finds the parameter values that make the observed data most likely.
If the errors are Gaussian with mean zero, each output is Gaussian with mean $w^T x_i$ and variance $\sigma^2$ .
Maximizing the log-likelihood of the outputs is the same as minimizing the squared error.

So squared error isn't an arbitrary choice. It is what maximum likelihood gives us when we assume Gaussian errors.

Free preview

That was a free preview lesson.

Enroll to access the full course