Multiclass Classification

What Is Multiclass Classification?

So far we have looked at binary classification, where there are only two classes. In multiclass classification, a model predicts one class out of many.

Here are some well-known applications:

Handwritten digit recognition: classifying images of digits from the MNIST dataset into one of ten classes, 0 to 9
Animal image classification: classifying a photo as a dog, cat, lion, elephant, bird, or another animal
Email categorization: classifying an email as work, personal, promotions, or spam
Fruit recognition: classifying a photo as an apple, banana, grapes, orange, or pineapple
Language detection: classifying a text message as English, Spanish, French, German, or Hindi
Sentiment with multiple emotions: classifying a product review as happy, neutral, angry, or sad

The Problem

Suppose we have a training dataset of $n$ samples:

(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)

Each $x_i$ is a $d$ -dimensional input. Each output label $y_i$ belongs to one of $K$ classes, $C_{0}, C_{1}, \dots,$ . We use zero indexing for the classes.

We want to estimate $P(y_i \mid x_i)$ from the training data, so that we can predict the class of any new input.

To do this, we need a probability for each class, $p_0, p_1, \dots, p_{K-1}$ . Here $p_0$ is the probability of class , is the probability of class , and so on. These probabilities must add up to 1:

\sum_{k=0}^{K-1} p_k = 1

Before we see how to calculate them, let's look at the main approaches to multiclass classification.

One-Versus-Rest

In the one-versus-rest approach, we train multiple binary classifiers using a method we already know, such as logistic regression. Each classifier separates one class from all the others.

For a dataset with three classes, A, B, and C, we train three binary classifiers:

Classifier 1: class A versus the rest (B and C)

Classifier 2: class B versus the rest (A and C)

Classifier 3: class C versus the rest (A and B)

In each classifier, the chosen class is the success, and every other class is the failure.

The final class comes from the classifier with the highest score. If classifier 2 scores highest, the output is class B.

The Deep Learning Approach

The deep learning approach is the most widely used today. Instead of many classifiers, it uses a single, more complex model, with an input layer, several hidden layers, and an output layer.

The output layer has one neuron per class. With three classes, there are three output neurons, and each one outputs the probability of its class. The model uses a function called softmax to produce this probability distribution.

We'll cover softmax in this lesson, and leave the details of neural networks for the deep learning course.

Representing the Output

There are two ways to represent an output class.

Integer Representation

We can use the class index as the label. Suppose we have three categories, cat, dog, and cow. With zero indexing, cat is 0, dog is 1, and cow is 2. If the correct label is dog, the integer representation is 1.

One-Hot Representation

We can also represent the output as a one-hot vector, which is mostly what we use for model training. This is a $K$ -dimensional vector with a 1 in one position and zeros everywhere else. The position of the 1 gives the class.

For example, with four classes, 0 to 3, class 2 is represented as:

\begin{bmatrix} 0 & 0 & 1 & 0 \end{bmatrix}

We write the one-hot form of $y_i$ as $(y_{i0}, y_{i1}, \dots, y_{i,K-1})$ . Exactly one of these values is 1, and the rest are 0.

Calculating the Class Probabilities

In binary classification, we train one weight vector $w$ , which has $d + 1$ values. For $K$ classes, we need $K$ weight vectors, one per class:

w^{(0)}, w^{(1)}, \dots, w^{(K-1)}

The superscript here is the class. So $w^{(1)}$ is the weight vector for class 1, with values $w_0^{(1)}, w_1^{(1)}, \dots, w_d^{(1)}$ .

As in linear regression, each input $x_i$ has a 1 added at the start, so it also has $d + 1$ values. For any input $x_i$ , we can then calculate one linear output per class:

a_{ik} = w^{(k)T} x_i \quad \text{for } k = 0, 1, \dots, K - 1

These values can be negative or any other continuous value, so they aren't probabilities. We need to convert them.

The Softmax Function

Given $C$ values $r_0, r_1, \dots, r_{C-1}$ , the of the -th value is:

\text{softmax}(r_i) = \frac{e^{r_i}}{\sum_{j=0}^{C-1} e^{r_j}}

Here, $e$ is Euler's number. The denominator adds up $e^{r_j}$ over all $C$ values, so the softmax of each value depends on all the other values too.

Applying this to our linear outputs, the probability of class $k$ for input $x_i$ is:

z_{ik} = \text{softmax}(a_{ik}) = \frac{e^{a_{ik}}}{\sum_{j=0}^{K-1} e^{a_{ij}}}

This gives us a vector of probabilities, one per class:

z_i = (z_{i0}, z_{i1}, \dots, z_{i,K-1})

The probabilities in this vector add up to 1.

Why Softmax?

There are two main reasons to use softmax.

It squeezes values between 0 and 1. The value of $e$ raised to any power is always positive. The numerator is just one of the positive terms in the denominator, so the result always lies between 0 and 1.

It amplifies differences. The exponential magnifies larger values. If $a_{ik} > a_{il}$ , then $e^{a_{ik}} > e^{a_{il}}$ , and the gap grows quickly as the difference increases. This helps the model assign higher confidence to the correct class.

The Loss Function

Now we need a loss function to train the weights. Let's start with the probability assigned to the correct class:

If the actual class is 0, so $y_{i0} = 1$ , the probability is $z_{i0}$ .
If the actual class is 1, so $y_{i1} = 1$ , the probability is .

We can write all of these cases as a single formula, called the likelihood:

P(y_i \mid x_i) = z_{i0}^{\,y_{i0}} \; z_{i1}^{\,y_{i1}} \cdots z_{i,K-1}^{\,y_{i,K-1}} = \prod_{k=0}^{K-1} z_{ik}^{\,y_{ik}}

This works because of the one-hot vector. Only one $y_{ik}$ is 1, and anything raised to the power 0 is 1. So every factor becomes 1 except the one for the correct class, which leaves just the probability of the correct class.

We want to maximize the likelihood, so we maximize its log:

\log P(y_i \mid x_i) = \sum_{k=0}^{K-1} y_{ik} \log z_{ik}

Instead, we can minimize the negative log-likelihood. This gives us the loss for sample $i$ :

E_i = -\sum_{k=0}^{K-1} y_{ik} \log z_{ik}

This is called the cross-entropy loss, and it is the loss function for multiclass classification.

Why "Cross-Entropy"?

For probabilities $p_0, p_1, \dots, p_{K-1}$ over $K$ classes, the entropy is:

H = -\sum_{k=0}^{K-1} p_k \log p_k

In entropy, the same probability appears in both places. In our loss, there are two different terms, the actual output $y_{ik}$ and the computed output $z_{ik}$ . This is why it is called cross-entropy.

The Total Loss

The total loss over all $n$ samples adds up the loss for each one:

E(W) = -\sum_{i=1}^{n} \sum_{k=0}^{K-1} y_{ik} \log z_{ik}

Here, $W$ stands for all the weight vectors. We take the gradient of this loss, and use gradient descent to optimize the weights.

Categorical and Sparse Categorical Cross-Entropy

There are two commonly used forms of the cross-entropy loss. They differ in how the target labels are stored:

Categorical cross-entropy requires the target labels to be in one-hot format.
Sparse categorical cross-entropy is designed for labels stored as integers, or class indices.

Both compute the same loss. In the one-hot formula, only the correct class has a non-zero term, so the sum reduces to the negative log of the probability of the correct class. Sparse categorical cross-entropy uses the integer label to pick out that probability directly:

E_i = -\log z_{i, y_i}

Since there is no need to store a one-hot vector for each sample, this reduces memory usage and computational overhead, especially for datasets with a large number of classes.

Inference

During test time, we use the optimized weight vectors to predict the class of a new input $x$ .

First, we calculate the probability of each class:

(z_0, z_1, \dots, z_{K-1}) = \big(\text{softmax}(w^{(0)T} x), \dots, \text{softmax}(w^{(K-1)T} x)\big)

The output class is the one with the highest probability:

k^* = \arg\max_{k} \; z_k

Summary

The key ideas are:

Multiclass classification predicts one class out of $K$ classes.
One-versus-rest trains one binary classifier per class, while deep learning uses a single model with one output neuron per class.
Outputs can be represented as integers or as one-hot vectors.
Softmax converts the linear outputs into probabilities that add up to 1.
The loss function is cross-entropy, which is the negative log-likelihood of the correct class.
Categorical cross-entropy uses one-hot labels, and sparse categorical cross-entropy uses integer labels.
To make a prediction, we take the class with the highest probability.