Forward Propagation

How Neural Networks Transform Data

We already learned about neural network with weights and activation functions. But, how does it actually make a prediction? Forward propagation is the answer—it's the journey of data from input to output, layer by layer.

Just as we decomposed XOR into hidden neurons in the motivation chapter, forward propagation is the mechanism that computes those intermediate values ($h_1$, $h_2$) and finally the output. It's the first step of every training iteration.

Think of it like water flowing through pipes:

• Input enters at one end
• Pipes (weights) control how much flows where
• Junctions (activation functions) decide what gets passed forward
• Output emerges at the other end

The Single Neuron Computation

The Single Neuron Computation

Before understanding an entire neural network, let us begin with a single neuron.

A neuron performs two main operations:

1. Computes a linear combination of its inputs.
2. Applies an activation function to the result.

Step 1: Linear Combination

The neuron multiplies every input by its corresponding weight and adds a bias:

Equivalently:

In the diagram, the bias is represented using an additional input:

with weight $w_0$. Therefore:

and $w_0$ acts as the bias term.

The complete linear operation can then be written as:

where:

• $x_i$ represents an input value.
• $w_i$ represents its corresponding learned weight.
• $x_0=1$ is the constant bias input.
• $w_0$ is the learned bias parameter.

Step 2: Activation Function

The neuron passes the linear result through an activation function:

Depending on the network, the activation function could be:

or:

The selected activation function determines how the neuron responds to its weighted input.

For a single neuron, forward propagation can therefore be summarized as:

Layer-by-Layer Propagation

Now let's scale this to multiple neurons organized in layers. A neural network has three types of layers:

Input Layer

The input layer receives raw features and passes them forward without computation:

Where $\mathbf{x}$ is the vector of input features (e.g., $[x_1, x_2]$ for our XOR example).

Hidden Layers

Each hidden layer transforms the representation from the previous layer. For layer $l$:

Linear step:

Activation step:

Where:

• $\mathbf{W}^{(l)}$ is the weight matrix connecting layer $l-1$ to layer $l$
• $\mathbf{b}^{(l)}$ is the bias vector for layer $l$
• $g^{(l)}$ is the activation function (ReLU, tanh, sigmoid, etc.)
• $\mathbf{a}^{(l)}$ is the activation output of layer $l$

Output Layer

The final layer produces the network's prediction:

Where $L$ is the total number of layers. The activation function here depends on the task:

• Binary classification: Sigmoid (outputs probability between 0 and 1)
• Multi-class classification: Softmax (outputs probability distribution)
• Regression: Linear (no activation, or ReLU for non-negative outputs)

Worked Example: Solving XOR with Forward Propagation

Let's walk through the XOR network we built in the motivation chapter. This network uses a hidden layer with two neurons (one approximating OR, one approximating AND) and an output layer that combines them.

Network Architecture

• Input layer: 2 neurons ($x_1$, $x_2$)
• Hidden layer: 2 neurons with sigmoid activation
• Output layer: 1 neuron with sigmoid activation

The Learned Weights

From our earlier construction:

Hidden layer:

• Neuron 1 (OR behavior): $w_1 = [20, 20]$, $b_1 = -10$
• Neuron 2 (AND behavior): $w_2 = [20, 20]$, $b_2 = -30$

Output layer:

• $w = [20, -20]$, $b = -10$

Forward Pass for Input $(1, 0)$

Let's compute step by step:

Hidden layer - Neuron 1 (OR):

Hidden layer - Neuron 2 (AND):

Output layer:

Result: For input $(1, 0)$, the network predicts $1$. Since $1 \oplus 0 = 1$, this is correct.

Complete Truth Table

Let's verify all four input combinations:

The network correctly computes XOR by:

1. First layer: Learning intermediate concepts (OR and AND)
2. Second layer: Combining them to produce the final output

This is the power of depth—each layer builds on the previous one.

Vectorization: Processing Multiple Samples

Processing one sample at a time is inefficient. Modern hardware (GPUs) excels at performing the same operation on many samples simultaneously. Vectorization allows us to process an entire mini-batch in one go.

Single Sample (Slow)

For one sample, layer $l$ computes:

Here $\mathbf{a}^{(l-1)}$ is a column vector of shape $(n^{(l-1)}, 1)$.

Mini-Batch of $m$ Samples (Fast)

For $m$ samples, we stack them as columns:

Where:

• $\mathbf{A}^{(l-1)}$ has shape $(n^{(l-1)}, m)$ — each column is one sample
• $\mathbf{Z}^{(l)}$ has shape $(n^{(l)}, m)$ — each column is the linear output for one sample
• $\mathbf{b}^{(l)}$ is broadcasted across all columns

Why this matters:

• A batch of 128 samples takes roughly the same time as 1 sample on a GPU
• Matrix multiplication is highly optimized
• Training becomes dramatically faster

Why the $w_{ji}^{(l)}$ Notation Matters?

In the notation $w_{ji}^{(l)}$:

• $l$ identifies the current layer.
• $j$ identifies the destination neuron in layer $l$.
• $i$ identifies the source neuron in layer $l-1$.

Therefore, $w_{ji}^{(l)}$ represents:

The weight connecting source neuron $i$ in layer $l-1$ to destination neuron $j$ in layer $l$.

This notation may initially appear backward, but it makes matrix operations clean and consistent.

It Reads Like an Address

The symbol:

can be interpreted as:

The weight going from source neuron $i$ in layer $l-1$ to destination neuron $j$ in layer $l$.

For example:

represents the weight connecting neuron $2$ in layer $l-1$ to neuron $3$ in layer $l$.

It Organizes Weights into Rows

Because the destination index $j$ appears first, all weights entering the same neuron are stored in the same row of the weight matrix.

The weight matrix for layer $l$ is:

where:

• $n_{l-1}$ is the number of neurons in the previous layer.
• $n_l$ is the number of neurons in the current layer.
• Each row stores the weights entering one destination neuron.
• Each column corresponds to one source neuron in the previous layer.

For example:

• Row 1 contains all the weights entering Neuron 1 in layer $l$.
• Row 2 contains all the weights entering Neuron 2 in layer $l$.
• Row $j$ contains all the weights entering Neuron $j$ in layer $l$.

It Fits Naturally into Matrix Multiplication

The weight matrix for layer $l$ has dimensions:

For a batch containing $m$ samples, the activation matrix from the previous layer has dimensions:

The linear transformation is:

The matrix dimensions align as follows:

The two inner dimensions, $n_{l-1}$, match.

Therefore, the weight matrix and activation matrix can be multiplied directly without transposing the weight matrix.

The resulting matrix contains the linear outputs of every neuron in layer $l$ for all $m$ samples.

After applying the activation function:

where $g^{(l)}$ is the activation function used in layer $l$.

Vectorization allows the network to process multiple neurons and samples simultaneously. This reduces slow programmatic loops, accelerates training, and makes efficient use of modern hardware.

Bridging the Gap: One-Hot Encoding

Before passing network outputs into a loss function, categorical labels such as Dog, Cat, and Bird must be converted into numerical representations.

This is necessary for multiclass classification.

We should not simply assign sequential integers:

This would create an artificial ordinal relationship.

It would incorrectly suggest that Bird is numerically greater than Dog or that some classes are closer to one another than others.

One-Hot Representation

One-hot encoding transforms each categorical label into a binary vector of length $K$, where $K$ is the total number of classes.

Every element is zero except for the position representing the correct class.

For three classes:

Dog

Cat

Bird

For a dataset containing $m$ samples, the individual target vectors are stacked horizontally to form the ground-truth matrix:

For example:

The network's Softmax predictions have the same dimensions:

For example:

Therefore:

This creates element-wise alignment between the true labels and the network's predicted probabilities.

Measuring Error: Loss Functions

After forward propagation, the network produces predictions $\hat{y}$.

We need a quantitative way to measure the difference between these predictions and the actual targets $y$.

This calculation is performed by the loss function.

Different machine-learning tasks require different loss functions.

Binary Cross-Entropy Loss

Binary Cross-Entropy is commonly used for binary classification when the output neuron uses a Sigmoid activation function:

This loss heavily penalizes confident but incorrect predictions.

For example, if:

then:

As a result, the loss becomes very large and generates a strong learning signal during optimization.

Categorical Cross-Entropy Loss

Categorical Cross-Entropy is commonly used for multiclass classification when the output layer uses Softmax:

Because the target vector $\mathbf{y}$ is one-hot encoded, only the term corresponding to the correct class contributes to the loss.

If the correct class is $c$, then:

while all other target values are zero.

Therefore, the loss simplifies to:

The model receives a small loss when it assigns a high probability to the correct class and a large loss when it assigns a low probability to the correct class.

Mean Squared Error

Mean Squared Error is commonly used for regression tasks involving continuous numerical values:

MSE squares the difference between predictions and ground-truth values.

This penalizes large errors more strongly and encourages the model to learn the overall numerical trend.

Key Takeaways

• Forward propagation transforms raw input features into final network predictions.
• Every neuron performs two operations: a linear combination followed by a nonlinear activation function.
• Hidden layers transform input features into increasingly abstract representations.
• In $w_{ji}^{(l)}$, $l$ identifies the current layer, $j$ identifies the destination neuron, and $i$ identifies the source neuron.
• The weight matrix $\mathbf{W}^{(l)}$ has dimensions $n_l\times n_{l-1}$.
• The activation matrix $\mathbf{A}^{(l-1)}$ has dimensions $n_{l-1}\times m$.
• Vectorization processes multiple neurons and samples simultaneously using matrix operations.
• One-hot encoding converts categorical labels into binary target vectors without creating an artificial ordering.
• Loss functions compare predictions with ground-truth targets and provide the objective required for learning.

Now that forward propagation has transformed the inputs into predictions and produced a loss value, the next step is to understand how the network learns from its mistakes.