Weight Initialization

Starting Model Training with Stable Weights and Gradients

Training a neural network begins with one simple question:

What values should we assign to the weights before learning starts?

At first glance, initialization may seem unimportant because the network will eventually learn suitable weights through gradient descent.

However, poor initialization can slow down learning or prevent it completely.

Weight initialization plays a critical role in maintaining stable activations and gradient flow during training.

Why Initialization Matters

Neural networks learn by updating weights through backpropagation.

For a weight parameter $w$:

where:

• $E$ is the loss.
• $\alpha$ is the learning rate.
• $\partial E / \partial w$ is the gradient of the loss with respect to the weight.

The gradient at one layer depends on activations, weights, and gradients from other layers.

If weights are initialized poorly, activations and gradients can become unstable.

This may lead to:

• Vanishing gradients
• Exploding gradients
• Slow convergence
• Failed training

Training stability therefore begins before the first update step.

Why Not Initialize Every Weight to Zero?

A natural idea might be to set every weight to zero:

However, this creates a serious problem called symmetry.

Suppose multiple neurons in the same layer start with identical weights.

During forward propagation:

• All neurons receive the same inputs.
• All neurons produce the same output.

During backpropagation:

• All neurons receive identical gradients.
• Their weights receive identical updates.

As a result, all neurons continue learning exactly the same feature.

Instead of learning diverse representations, the entire layer behaves like a single neuron.

This prevents effective learning.

Important: Zero initialization is problematic for the weights of neurons in the same layer. Bias parameters can generally be initialized to zero because randomly initialized weights already break the symmetry.

Random Initialization

To break symmetry, weights are initialized randomly.

For example, weights may be sampled from a uniform distribution:

or a normal distribution:

Now neurons begin with different weights and can learn different features.

However, naive random initialization introduces another problem.

If the weights are too large:

• Activations can grow rapidly.
• Gradients can explode.

If the weights are too small:

• Activations can shrink rapidly.
• Gradients can vanish.

Randomness breaks symmetry, but the scale of that randomness determines whether signals remain stable.

Fan-In and Fan-Out

Initialization formulas often use two quantities:

• $n_{\text{in}}$: the number of input connections to a layer, also called fan-in.
• $n_{\text{out}}$: the number of output connections from a layer, also called fan-out.

These values help determine an appropriate variance for the initial weights.

Xavier Initialization

For activation functions such as sigmoid and tanh, Xavier initialization helps maintain stable activation variance.

It is also called Glorot initialization.

Normal Distribution

Weights may be sampled from:

Here, the second argument represents the variance.

A commonly used Glorot-normal variant balances fan-in and fan-out:

Uniform Distribution

Weights may instead be sampled from:

Xavier initialization helps prevent activations from becoming too large or too small as they pass through the network.

It is commonly used with:

• Sigmoid
• Tanh
• Linear activations

He (Kaiming) Initialization

For ReLU-based networks, Xavier initialization may not preserve variance as effectively.

ReLU removes negative activations:

Because many negative pre-activations become zero, the variance of the signal is reduced.

He initialization compensates for this effect.

It is also called Kaiming initialization.

Normal Distribution

Weights are commonly sampled from:

Equivalently, the standard deviation is:

Uniform Distribution

A corresponding uniform form is:

He initialization preserves activation variance more effectively in deep ReLU networks.

It is commonly used with:

• ReLU
• Leaky ReLU
• Other ReLU-family activations
• Modern deep networks with rectified activations

Xavier vs He Initialization

The activation function and initialization method should be chosen together.

Intuition Behind Good Initialization

A good initialization ensures that signals remain stable as they move forward and backward through the network.

During the Forward Pass

• Activations remain well-scaled.
• Information does not disappear.
• Values do not grow uncontrollably.

During the Backward Pass

• Gradients remain stable.
• Earlier layers continue receiving useful learning signals.
• Optimization remains efficient.

The goal is simple:

Forward pass: Stable activations

Backward pass: Stable gradients

Poor initialization can break both.

Connection to Stable Training

Weight initialization directly affects gradient behavior.

Poor initialization can cause:

• Vanishing gradients
• Exploding gradients

Good initialization allows gradients to flow more smoothly across deep networks.

This makes optimization significantly easier.

Modern deep learning became far more practical as initialization strategies, activation functions, normalization methods, and optimizers improved.

Key Takeaways

• Weight initialization strongly affects training stability.
• Initializing every weight to zero preserves symmetry and prevents neurons from learning diverse features.
• Random initialization breaks symmetry but must be scaled carefully.
• Xavier initialization is commonly used with sigmoid and tanh networks.
• He initialization is preferred for ReLU-based deep networks.
• Good initialization helps control vanishing and exploding gradients.
• Biases can usually be initialized to zero when weights are initialized independently.

Final Insight

Learning does not begin with gradient descent.

It begins with choosing the right starting point.

A well-initialized network learns faster, remains more stable, and trains far more effectively.