Optimization & Learning
JUN 2026

AdaGrad

Why should every parameter share the same learning rate? Discover how giving each dimension its own adaptive step size solves the zigzagging problem.

Read Stochastic Gradient Descent Primer

Historical Context

In 2011, John Duchi, Elad Hazan, and Yoram Singer published Adaptive Subgradient Methods for Online Learning and Stochastic Optimization (AdaGrad). Up until this point, algorithms largely applied a single, global learning rate to every parameter in a model.

AdaGrad asked a simple question: What if a parameter is associated with a very rare feature? With a global learning rate, rare features barely get updated, while frequent features get updated constantly. AdaGrad introduced the idea of adaptive, per-parameter learning rates.

By keeping a running sum of the squared gradients for each parameter, AdaGrad automatically shrinks the learning rate for frequently updated parameters and keeps it high for rarely updated ones. This was a massive conceptual leap that paved the way for modern optimizers like RMSProp and Adam.

Layer 1: The Anisotropic Landscape

In most real-world problems, loss landscapes are anisotropic—meaning they don't slope equally in all directions. Imagine a stretched valley that is very steep along the Y-axis but incredibly flat along the X-axis.

Standard Gradient Descent struggles here. If you set the learning rate low enough to avoid exploding on the steep Y-axis, the optimizer crawls at a snail's pace along the flat X-axis. If you increase the learning rate to move faster along X, it wildly overshoots and diverges along Y.

We need an algorithm that can take small steps in steep directions and large steps in flat directions, automatically.

Notice how Vanilla SGD (left) points either straight down or up the steep walls of the valley, mostly ignoring the path to the global minimum. It gets trapped bouncing back and forth across the Y-axis.

AdaGrad (right) tracks the history of the gradients. Because the Y gradients are huge, its accumulator grows rapidly, shrinking the effective learning rate in the Y direction. This acts as a brake and normalizes the step size, allowing it to point correctly down the center of the valley!

Vanilla SGD

xn+1=xn−η∇f(xn)x_{n+1} = x_n - \eta \nabla f(x_n)

AdaGrad

xn+1=xn−ηGn+ϵ⊙∇f(xn)x_{n+1} = x_n - \frac{\eta}{\sqrt{G_n + \epsilon}} \odot \nabla f(x_n)
where:
  • xn+1x_{n+1}= Next position
  • xnx_n= Current position
  • η\eta= Base learning rate
  • ∇f(xn)\nabla f(x_n)= Gradient of the objective function
  • GnG_n= Sum of squares of past gradients
  • ϵ\epsilon= Small constant for numerical stability
  • ⊙\odot= Element-wise multiplication

Vanilla SGD

READY. PRESS PLAY TO START.
Iteration 0
Pos: [-4.0000, 4.0000]
f(x,y): 176.0000
∇f: [-8.0000, 80.0000]
step: [0.4000, -4.0000]
Click anywhere to set start point
Global Min

AdaGrad

READY. PRESS PLAY TO START.
Iteration 0
Pos: [-4.0000, 4.0000]
f(x,y): 176.0000
∇f: [-8.0000, 80.0000]
step: [0.5000, -0.5000]
Click anywhere to set start point
Global Min

Layer 2: The Gradient Accumulator

AdaGrad solves this by giving every parameter its own personal learning rate. It does this by keeping a memory of every gradient it has ever seen.

For each coordinate, AdaGrad tracks the sum of squared gradients (let's call it G). The effective learning rate for that coordinate is the base learning rate divided by the square root of G.

If a direction is steep, its gradients are large, so G grows rapidly. Dividing by a large number shrinks the step size, acting as an automatic brake. If a direction is flat, G stays small, keeping the learning rate high.

Notice how the step ellipse adapts below: it squashes in the steep direction to prevent overshooting, and stretches in the flat direction to accelerate progress.

Gn=∑i=1ngi⊙giG_n = \sum_{i=1}^n g_i \odot g_i
ηeffective=ηGn+ϵ\eta_{effective} = \frac{\eta}{\sqrt{G_n + \epsilon}}
where:
  • GnG_n= Accumulated squared gradients
  • gig_i= Gradient at step i
  • ⊙\odot= Element-wise multiplication
  • ηeffective\eta_{effective}= Adaptive learning rate
  • η\eta= Base learning rate
  • ϵ\epsilon= Small constant for numerical stability
READY. PRESS PLAY TO START.
Iteration 0
Pos: [4.0000, 4.0000]
f(x,y): 176.0000
∇f: [8.0000, 80.0000]
step: [-0.5000, -0.5000]
Click anywhere to set start point
Global Min

Layer 3: The Fatal Flaw & Sparse Gradients

Look closely at the formula. G is a sum of squares, which means it can only ever increase. It never forgets the past.

Because we divide by this ever-growing sum, the learning rate for every parameter shrinks continuously over time. Eventually, the learning rates become infinitesimally small, and the algorithm completely freezes, often long before reaching the minimum.

Why use it then? Because for sparse data (like words in a massive vocabulary), most parameters get a gradient of zero on most steps. Their G barely grows, preserving their learning rate for when they finally appear in a batch. AdaGrad remains incredibly effective for sparse problems, even if its freezing flaw required the invention of RMSProp and Adam for dense deep learning.

Simulated Sparse Gradients

if (random > sparsity)  ⟹  gy=∂f∂y\text{if (random > sparsity)} \implies g_y = \frac{\partial f}{\partial y}
else  ⟹  gy=0\text{else} \implies g_y = 0
where:
  • gyg_y= Gradient in the y direction
  • sparsity\text{sparsity}= Probability of zeroing the y gradient
  • ∂f∂y\frac{\partial f}{\partial y}= True partial derivative
READY. PRESS PLAY TO START.
Iteration 0
Pos: [-4.0000, -4.0000]
f(x,y): 3.2000
∇f: [-0.8000, -0.8000]
step: [1.0000, 1.0000]
Click anywhere to set start point
Global Min

AdaGrad Implementation

Placeholder description.

Output
Output will appear here after running the code

AdaGrad proved that per-parameter learning rates were the key to navigating complex, stretched landscapes.

But its infinite memory causes it to freeze prematurely. What if we allowed it to gradually forget the distant past?

Fixing the Accumulator

Enter RMSProp