Image Classification

Image classification is one of the most fundamental problems in computer vision.

The objective is simple:

Given an image, predict the category to which the entire image belongs.

For example, a classification model may receive an image and predict one of the following classes:

The model does not normally identify the position of the object. It assigns one or more labels to the complete image.

What Is the Input?

The input is an image represented as numerical pixel values.

A greyscale image will have height $H$ and width $W$.

An RGB image with height $H$ and width $W$ has the shape:

The three channels represent red, green and blue.

For example, an RGB image resized to $224 \times 224$ has the shape:

In a deep-learning framework, multiple images are processed together in a batch. The complete input may therefore have the shape:

where:

• $B$ is the batch size,
• $C$ is the number of channels,
• $H$ is the image height, and
• $W$ is the image width.

For a batch of 32 RGB images:

The exact channel position depends on the deep-learning framework.

What Is the Output?

Suppose the model recognizes four classes:

Cat, Dog, Horse, Bird

The final layer produces one score for each class:

These raw scores are called logits.

We use Softmax to convert the logits into probabilities:

The probabilities add up to one.

For example:

The predicted class is Dog because it has the highest probability.

Types of Image Classification

Binary Classification

Binary classification chooses between two classes.

Examples include:

• normal versus abnormal scan,
• defective versus non-defective product,
• cat versus dog, and
• authentic versus counterfeit document.

A binary classifier commonly produces one probability:

The sigmoid function is often used:

If the probability is greater than a selected threshold, the positive class is predicted.

Multiclass Classification

Multiclass classification selects exactly one class from several mutually exclusive classes.

Examples include:

• classifying an image as a cat, dog, horse or bird,
• recognizing one handwritten digit from 0 to 9, and
• identifying one plant species from many possible species.

The final layer normally uses softmax.

Multilabel Classification

In multilabel classification, one image can have multiple correct labels.

For example, an image may contain:

Beach
Sunset
Ocean
Person

Each label is treated independently, so the model normally uses one sigmoid output per class rather than softmax.

What Annotation Is Required?

Image classification requires an image-level label.

For example:

For a folder-based dataset, images may be organized as:

dataset/
├── train/
│   ├── cat/
│   ├── dog/
│   └── bird/
├── validation/
│   ├── cat/
│   ├── dog/
│   └── bird/
└── test/
    ├── cat/
    ├── dog/
    └── bird/

The folder name acts as the class label.

How Does a CNN Classify an Image?

A CNN normally contains two broad parts:

Input image
     ↓
Feature-extraction network
     ↓
Learned visual representation
     ↓
Classification head
     ↓
Class probabilities

Feature Extraction

Convolutional layers learn filters that detect useful visual patterns.

Early layers commonly learn:

• edges,
• colour changes,
• corners, and
• simple textures.

Middle layers combine them into:

• curves,
• repeated patterns,
• shapes, and
• object parts.

Deeper layers may respond to:

• faces,
• wheels,
• eyes,
• wings, and
• complete semantic structures.

The progression can be understood as:

Pixels
  ↓
Edges
  ↓
Textures and curves
  ↓
Object parts
  ↓
High-level representation

Classification Head

The learned representation is passed to a classification head.

Older networks such as AlexNet and VGG use large fully connected layers. Many newer networks use global average pooling followed by a small final layer.

For $K$ classes, the final layer produces $K$ logits:

Softmax or sigmoid then converts these logits into class probabilities.

Training the Classifier

During training, the model receives:

1. an input image,
2. the correct class label, and
3. a loss function that measures prediction error.

The learning process is:

Input image
    ↓
Forward pass
    ↓
Predicted probabilities
    ↓
Compare with the correct label
    ↓
Calculate loss
    ↓
Backpropagation
    ↓
Update filters and weights

At the beginning, the filters contain random values. The predictions are therefore mostly incorrect.

Backpropagation calculates how much every parameter contributed to the error. An optimizer then updates the parameters so that future predictions become more accurate.

Cross-Entropy Loss

For multiclass classification, cross-entropy loss is commonly used:

where:

• $K$ is the number of classes,
• $y_k$ is the true label representation, and
• $p_k$ is the predicted probability for class $k$.

Because only the correct class has $y_k=1$, this simplifies to:

If the model gives a high probability to the correct class, the loss is small. If it gives the correct class a low probability, the loss is large.

For binary and multilabel classification, binary cross-entropy is commonly used.

Image Preprocessing

Images in a dataset may have different sizes, colour ranges and aspect ratios. Before being passed to a model, they are usually preprocessed.

Common operations include:

• resizing,
• centre cropping or random cropping,
• converting pixels to floating-point values,
• scaling values to a standard range, and
• normalizing each colour channel.

For example, pixel values between 0 and 255 may be scaled to the range 0 to 1:

When using a pretrained model, its expected image size and normalization values should be used.

Data Augmentation

Data augmentation creates different versions of training images without changing their labels.

Common transformations include:

• random cropping,
• horizontal flipping,
• small rotations,
• brightness adjustment,
• contrast adjustment,
• colour variation, and
• random erasing.

Data augmentation helps the model learn that the class should remain unchanged under reasonable visual variations.

For example, a dog remains a dog when the image is slightly cropped, brightened or flipped.

Augmentation should reflect realistic variations. An inappropriate transformation can change the meaning of the image. For example, vertical flipping may be unsuitable for medical scans or street scenes.

Evaluating an Image Classifier

Accuracy

Accuracy measures the fraction of correct predictions:

Accuracy works well when classes are reasonably balanced.

Top-1 Accuracy

Top-1 accuracy counts a prediction as correct only when the highest-probability class is the correct class.

Top-5 Accuracy

Top-5 accuracy counts a prediction as correct when the correct label appears among the five highest-probability classes.

Top-5 accuracy is commonly reported for datasets with many classes, such as ImageNet.

Precision

Precision asks:

Of all images predicted as a class, how many actually belong to that class?

Recall

Recall asks:

Of all images that truly belong to a class, how many did the model find?

F1 Score

The F1 score balances precision and recall:

Precision, recall and F1 are particularly useful when the class distribution is imbalanced.

Confusion Matrix

A confusion matrix shows how often each true class is predicted as every possible class.

It helps answer questions such as:

• Is the model confusing cats with dogs?
• Is one class almost never recognized?
• Are predictions biased toward a common class?

Challenges in Image Classification

Changes in Object Position

The same object can appear at different positions within an image.

Convolution and pooling provide some tolerance to translation, while augmentation exposes the model to more positional variation.

Changes in Scale

An object can occupy a small part of an image or almost the complete image.

Models must learn representations that remain useful across scale changes.

Changes in Viewpoint

An object may look very different from the front, side, top or back.

Illumination

Lighting can change colour, contrast and the visibility of details.

Occlusion

Part of an object may be hidden behind another object.

Background Correlation

A model may learn the background instead of the object.

For example, if most boat images contain water, the network may incorrectly associate water with the boat class.

Fine-Grained Differences

Some classes differ only in subtle details, such as similar bird species or visually similar medical conditions.

Class Imbalance

If one class has far more examples, the model may become biased toward it.

Famous Image-Classification Models

Image-classification architectures evolved by addressing different limitations: scale, depth, gradient flow, efficiency and global context.

• LeNet-5
• AlexNet
• Network in Network
• VGGNet
• GoogLeNet and Inception
• ResNet
• DenseNet
• MobileNet
• EfficientNet
• ConvNeXt
• Vision Transformer

Common Classification Mistakes

Data Leakage

Very similar or duplicate images must not appear in both training and test sets. Otherwise, test performance may look unrealistically high.

Incorrect Splitting

When several images come from the same patient, person, video or physical object, they should normally remain in the same split.

Using Accuracy Alone

Accuracy can hide poor performance on rare classes. Examine per-class precision, recall, F1 and the confusion matrix.

Ignoring Dataset Bias

A model may learn shortcuts such as backgrounds, watermarks, camera types or acquisition settings instead of meaningful object features.

Different Training and Inference Preprocessing

The resizing and normalization used during inference must match the model's expected preprocessing.

Training From Scratch Without Enough Data

Large networks can overfit small datasets. Transfer learning is usually a better starting point.

Key Takeaways

• Image classification assigns one or more labels to an entire image.
• It does not identify the position of objects.
• Classification requires image-level annotations rather than bounding boxes or masks.
• Binary classification usually uses sigmoid and binary cross-entropy.
• Multiclass classification normally uses softmax and categorical cross-entropy.
• Multilabel classification uses one independent sigmoid output per class.
• CNNs learn a hierarchy from edges to high-level object representations.
• Accuracy is useful, but imbalanced datasets also require precision, recall and F1.

Image classification is often the first major task studied in computer vision because it clearly demonstrates how a model transforms raw pixels into a semantic decision.