Logistic Regression

What Is Logistic Regression?

Logistic regression is a supervised classification algorithm, used mostly for binary classification. It predicts the probability that an outcome belongs to a certain class.

For example, suppose we know whether a person smokes, their age, and their blood pressure and cholesterol. From this information, we want to predict whether the person could have heart disease. This is a binary classification problem.

Logistic regression can be used for many other classification problems:

Medical diagnosis: predicting whether a patient has a disease, such as diabetes, from symptoms, test results, or genetic data
Credit scoring: predicting whether a person will default on a loan, based on prior data
Customer behavior: predicting whether a customer will buy a product from their age, income, and past purchases, or whether they will click on it

The Problem

In this lesson, we predict heart disease from age, blood pressure, cholesterol, and whether someone is a smoker. Our labeled training dataset has $n$ samples:

Age	Blood Pressure	Cholesterol	Smoker	Heart Disease
45	130	230	1	1
54	140	250	0	1
62	150	240	1	1
50	120	210	0	0
38	110	190	0	0
65	155	260	1	1
59	145	255	1	1
48	125	200	0	0
53	135	245	1	1
70	160	270	1	1

(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)

Each $x_i$ is the input, and each $y_i$ is the output label. The input is $d$ -dimensional. We have four attributes here, so $d = 4$ :

x_i = (x_{i1}, x_{i2}, \dots, x_{id})

We want a model that takes any input $x$ and predicts the output $y$ , which is either 1 (heart disease) or 0 (no heart disease).

The Bernoulli Distribution

Since the output is a yes or no answer, we need a Bernoulli distribution. It is used when there are exactly two possible outcomes, success or failure. We usually call success 1 and failure 0.

Here are some examples:

Flipping a coin, where heads is 1 and tails is 0
Whether an email is spam or not
Whether a customer buys a product or not
Whether something is present or absent

Let $p$ be the probability of success, which is always between 0 and 1. The probability of failure is then $1 - p$ :

P(Y = 1) = p \qquad P(Y = 0) = 1 - p

We can write both cases in a single formula:

P(Y = y) = p^{y} (1 - p)^{1 - y}

This works because $y$ can only be 0 or 1:

If $y = 1$ , the formula becomes $p^{1} (1 - p)^{0} = p$ .
If $y = 0$ , the formula becomes .

From Linear Regression to Classification

Can we adapt linear regression to give us a class instead of a continuous value? This is the idea behind logistic regression:

Use a linear regression model to predict a continuous value.
Transform that continuous value into a probability $p$ of success, between 0 and 1.
Treat a value below 0.5 as failure (0), and a value above 0.5 as success (1).

In linear regression, the predicted output was:

f(x_i) = w_0 + w_1 x_{i1} + w_2 x_{i2} + \dots + w_d x_{id} = w^T x_i

This can be any continuous value, but $p$ has to lie between 0 and 1. To convert it, we use the sigmoid function:

p = \sigma(w^T x_i)

The Sigmoid Function

For any continuous value $a$ , the sigmoid function is:

\sigma(a) = \frac{1}{1 + e^{-a}}

Let's see how it behaves, using the fact that $e^{\infty} \to \infty$ and $e^{-\infty} \to 0$ :

As $a \to -\infty$ , $\sigma(a) \to \frac{1}{1 + \infty} \to 0$ .
As , .

So all sigmoid values lie between 0 and 1, with 0.5 as the midpoint. Half of the curve is below 0.5 and half is above it. This is why the sigmoid can convert any continuous value into a value between 0 and 1.

Prediction

For any input $x_i$ , the predicted probability of class 1 is $\sigma(w^T x_i)$ , and the probability of class 0 is what's left over:

P(y_i = 1 \mid x_i) = \sigma(w^T x_i)

P(y_i = 0 \mid x_i) = 1 - \sigma(w^T x_i)

At test time, we get a class by thresholding:

If $\sigma(w^T x_i)$ is below 0.5, we predict class 0, the failure class.
If $\sigma(w^T x_i)$ is above 0.5, we predict class 1, the success class.

The Likelihood

The likelihood is the probability of getting the actual output for a given input. It is $\sigma(w^T x_i)$ when $y_i = 1$ , and when .

Using the same formula as the Bernoulli distribution, we can write both cases together:

P(y_i \mid x_i) = \sigma(w^T x_i)^{y_i} \, \big(1 - \sigma(w^T x_i)\big)^{1 - y_i}

As before, this works because $y_i$ is only 0 or 1:

If $y_i = 0$ , the first factor becomes 1, leaving $1 - \sigma(w^T x_i)$ .

The Loss Function

We want to maximize the likelihood. As in the maximum likelihood lesson, we take the log, which gives us this for sample $i$ :

y_i \log \sigma(w^T x_i) + (1 - y_i) \log\big(1 - \sigma(w^T x_i)\big)

Instead of maximizing the log-likelihood, we can minimize the negative log-likelihood:

-y_i \log \sigma(w^T x_i) - (1 - y_i) \log\big(1 - \sigma(w^T x_i)\big)

This is the loss function for logistic regression, and it is also called the binary cross-entropy.

Why "Binary Cross-Entropy"?

For a binary random variable with probability $p$ , the entropy measures the randomness of the variable:

H(p) = -p \log p - (1 - p) \log(1 - p)

Now look at our loss function again, and let $p = y_i$ and $q = \sigma(w^T x_i)$ . The loss then becomes:

-p \log q - (1 - p) \log(1 - q)

This looks similar to entropy, but with two different terms, $p$ and $q$ , which come from separate Bernoulli distributions. That's why it is called binary cross-entropy.

The Derivative of the Sigmoid

To train the model, we use gradient descent, with the update rule from the gradient descent lesson:

w^{(t+1)} = w^{(t)} - \alpha \nabla L(w^{(t)})

We need the gradient of the loss, and the loss contains sigmoids. So first, let's find the derivative of the sigmoid:

\sigma'(a) = \frac{d}{da} \left(1 + e^{-a}\right)^{-1} = -\frac{1}{(1 + e^{-a})^2} \cdot \left(-e^{-a}\right) = \frac{e^{-a}}{(1 + e^{-a})^2}

We can split this into two fractions:

\sigma'(a) = \frac{1}{1 + e^{-a}} \cdot \frac{e^{-a}}{1 + e^{-a}}

The first fraction is $\sigma(a)$ , and the second is $1 - \sigma(a)$ . So:

\sigma'(a) = \sigma(a) \big(1 - \sigma(a)\big)

The Gradient of the Loss

Over all $n$ samples, the loss to minimize is the average:

L(w) = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log \sigma(w^T x_i) + (1 - y_i) \log\big(1 - \sigma(w^T x_i)\big) \Big]

To shorten the next few lines, we'll write $\sigma_i$ for $\sigma(w^T x_i)$ .

Taking the gradient with respect to $w$ , we use the derivative of the log, the sigmoid derivative above, and the fact that the gradient of $w^T x_i$ is $x_i$ :

\nabla L(w) = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \frac{1}{\sigma_i} \, \sigma_i (1 - \sigma_i) \, x_i + (1 - y_i) \frac{1}{1 - \sigma_i} \, \big(-\sigma_i (1 - \sigma_i)\big) \, x_i \right]

The $\sigma_i$ terms cancel in the first part, and the $1 - \sigma_i$ terms cancel in the second:

\nabla L(w) = -\frac{1}{n} \sum_{i=1}^{n} \big[ y_i (1 - \sigma_i) x_i + (y_i - 1) \sigma_i x_i \big]

Expanding the brackets gives:

\nabla L(w) = -\frac{1}{n} \sum_{i=1}^{n} \big[ y_i x_i - y_i \sigma_i x_i + y_i \sigma_i x_i - \sigma_i x_i \big]

The two middle terms cancel, which leaves:

\nabla L(w) = -\frac{1}{n} \sum_{i=1}^{n} (y_i - \sigma_i) x_i

Moving the minus sign inside reverses the order. Since $\sigma_i = \sigma(w^T x_i)$ is the computed output, we can write it as $\hat{y}_i$ :

\nabla L(w) = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i) \, x_i

So the gradient is built from the difference between the computed output $\hat{y}_i$ and the actual output $y_i$ . We use this gradient in gradient descent to optimize the weights.

Summary

The key ideas are:

Logistic regression predicts the probability of a class, and is mostly used for binary classification.
It applies the sigmoid function to a linear model's output, giving a value between 0 and 1.
A threshold of 0.5 turns that probability into a class.
The likelihood comes from the Bernoulli distribution, and its negative log is the binary cross-entropy loss.
The gradient is $\frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i) \, x_i$ , and we use it to train the model with gradient descent.