Optimizers

Understanding the Need for Better Optimizers

So far, we have studied Gradient Descent and the different variants used to train neural networks.

These include:

• Batch Gradient Descent — uses the entire training dataset for each gradient update.
• Stochastic Gradient Descent (SGD) — uses a single training example for each update.
• Mini-Batch Gradient Descent — uses a small subset of the training data for each update.

Although these methods differ in how much data they use during each update, they rely on the same fundamental mechanism.

If the loss function is represented by:

then the weight update rule is:

where:

• $\mathbf{w}$ represents the model parameters.
• $\alpha$ is the learning rate.
• $\nabla_{\mathbf{w}}E(\mathbf{w})$ is the gradient of the loss with respect to the model parameters.

Regardless of whether the gradient is estimated using the full dataset, one example, or a mini-batch, the underlying parameter-update mechanism remains the same.

At first glance, this may seem sufficient.

However, deep neural networks introduce optimization challenges that can make this basic update rule inefficient.

In practice, relying only on basic gradient descent can make training:

• Slow
• Unstable
• Sensitive to the learning rate
• Ineffective in difficult regions of the loss landscape

This chapter explains why these limitations occur and why modern deep learning uses more advanced optimization algorithms.

The Core Limitation: A Uniform Learning Rate

One fundamental limitation of standard gradient descent is that every parameter uses the same global learning rate.

A deep neural network may contain millions or billions of parameters distributed across many layers.

Different parameters can behave differently during training:

• Some may require relatively large adjustments.
• Others may require only small updates.
• Some may receive gradients frequently.
• Others may receive gradients only occasionally.

Yet standard gradient descent uses the same learning-rate multiplier, $\alpha$, for every parameter.

The gradient magnitude still changes each individual update, but the optimizer does not adapt the learning-rate scale separately for each parameter.

This creates conflicting behavior across the network.

Problem 1: Overshooting in Steep Directions

Some directions lie in regions where the loss changes rapidly.

These directions have steep gradients.

If the learning rate is too large, the optimizer may take excessively large steps.

Instead of approaching the minimum smoothly, it may repeatedly overshoot.

This can lead to:

• Oscillations
• Unstable learning
• Increasing loss
• Failure to converge

Problem 2: Slow Learning in Flat Directions

Other directions may have very small gradients.

The same learning rate that was too large for a steep direction may produce tiny updates in a flat direction.

As a result:

• Weight updates become very small.
• Learning slows dramatically.
• Some parameters barely change.

A single global learning rate can therefore be too large in one direction and too small in another.

The Geometry of the Loss Surface

To understand optimization difficulties, we must understand the shape of the loss function.

A neural network with many parameters creates a highly complex loss landscape.

Unlike a simple convex optimization problem, a deep-learning loss surface can contain:

• Narrow valleys
• Flat plateaus
• Sharp curvature
• Local minima
• Saddle points

Moving through this landscape efficiently is much harder than the basic gradient update suggests.

Problem 3: Ravines and Oscillation

One common difficulty is a structure called a ravine.

A ravine is a region where the loss changes very differently in different directions.

In one direction:

• The slope is steep.

In another direction:

• The slope changes slowly.

This creates a long, narrow valley.

Batch Gradient Descent in Ravines

Batch Gradient Descent computes the gradient of the empirical loss using the entire training dataset.

Because the steep direction can dominate the gradient, the optimizer may repeatedly move toward one wall of the ravine and then the other.

It keeps bouncing sideways.

As a result:

• Large sideways oscillations occur.
• Little forward progress is made along the valley.
• Training becomes slow.

Stochastic Gradient Descent in Ravines

Stochastic Gradient Descent uses one training example per update.

This introduces high variance into the gradient estimate.

Instead of moving steadily through the ravine, the optimizer may jump in noisy directions.

The noise can sometimes help exploration, but it can also make convergence unstable.

Mini-Batch Gradient Descent in Ravines

Mini-batch Gradient Descent reduces some of the noise found in single-example SGD.

However, the underlying geometry remains unchanged.

The steep direction can still dominate, causing unnecessary oscillation.

Mini-batches provide a practical compromise, but changing the batch size alone does not solve the ravine problem.

Problem 4: Saddle Points

Deep neural networks frequently encounter saddle points.

A saddle point curves upward in some directions and downward in others.

Near its center, the gradient can become very small:

Gradient descent updates parameters using:

When the gradient is close to zero:

The optimizer can slow down dramatically even though lower-loss solutions exist nearby.

Large neural networks can contain many flat or nearly flat directions, making these regions difficult to traverse.

Problem 5: Different Features Learn at Different Speeds

Neural networks often learn from features that occur at different frequencies.

Some features appear frequently. Other important features may be rare.

Consider a language model.

Common words such as:

• the
• is
• and

appear constantly, while rare words appear much less frequently.

Standard gradient descent uses the same global learning-rate multiplier for parameters associated with both common and rare features.

This creates an imbalance:

• Parameters associated with common features update frequently.
• Parameters associated with rare features update infrequently.

As a result:

• Common patterns can dominate learning.
• Rare patterns may remain undertrained.

Adaptive optimizers address this problem by scaling updates separately for individual parameters.

Why Batch Size Alone Cannot Solve These Problems

It may seem that changing the batch size should solve these optimization issues.

However, whether we use:

• Batch Gradient Descent
• Stochastic Gradient Descent
• Mini-Batch Gradient Descent

the basic update remains:

Changing the batch size changes how the gradient is estimated.

It affects:

• Gradient noise
• Update frequency
• Computational efficiency
• Memory requirements

But it does not by itself give the optimizer:

• Memory of past gradients
• Per-parameter adaptive learning rates
• Explicit control over oscillation

The deeper optimization problem remains.

What We Need Instead

To train deep neural networks efficiently, we need better optimization strategies.

Modern optimizers build on two major ideas.

Strategy 1: Using Past Information

Instead of relying only on the current gradient, an optimizer can remember previous gradients or updates.

If gradients consistently point in one direction, the optimizer can build speed in that direction.

If gradients alternate in a steep direction, their effects can partially cancel.

This can:

• Reduce unnecessary oscillations
• Accelerate progress in consistent directions
• Help move through shallow or noisy regions

This idea leads to Momentum Optimization.

Strategy 2: Adaptive Learning Rates

Instead of using the same effective learning-rate scale for every parameter, an optimizer can adapt the scale separately.

Parameters receiving consistently large gradients can slow down.

Parameters receiving small or infrequent gradients can retain larger effective learning rates.

This idea leads to optimizers such as:

• Adagrad
• RMSProp
• Adam

Summary of Gradient Descent Variants

How Modern Optimizers Improve Learning

Transition to Modern Optimizers

Basic gradient descent gives us the foundation of learning.

But deep neural networks create optimization challenges beyond what a memoryless, globally scaled update can handle efficiently.

To learn more effectively, an optimizer should:

• Remember past movement.
• Reduce unnecessary oscillations.
• Move through difficult regions more efficiently.
• Adapt update scales for individual parameters.

The first major improvement comes from adding memory to gradient descent.

This idea gives rise to Momentum Optimization.

In the next chapter, we will study how Momentum uses previous gradients to guide future updates and accelerate learning.

Key Takeaways

• Gradient descent forms the foundation of neural-network learning.
• Batch, stochastic, and mini-batch variants mainly differ in how they estimate gradients.
• A single global learning rate can behave poorly across directions and parameters with different gradient scales.
• Ravines, plateaus, saddle points, and sparse features make optimization difficult.
• Changing batch size alone does not add memory or per-parameter adaptation.
• Momentum uses past information to reduce oscillation and accelerate consistent movement.
• Adagrad, RMSProp, and Adam adapt update scales for individual parameters.
• These limitations explain why advanced optimizers are essential for training deep neural networks effectively.