Interactive Algorithms
AUG 2026

The Algorithms Lab

Deconstructing the mathematical engines of modern computation. Interactive explorations of foundational algorithms, stripped of abstraction.

Volume I — Roots & Convergence

Algorithms emerged naturally from the historical need to solve concrete problems. We begin with the oldest known numerical method—designed simply to approximate a root—and trace its mathematical evolution all the way to fixed-point iteration. These are the foundational engines behind inference and optimization; the exact same mathematics, regardless of what you call the discipline.

ALG_01
Ref: 2026-B

Babylonian Method

Heron's Root Finding

The world's oldest known numerical algorithm. Computes square roots by iteratively averaging a guess with its complement, demonstrating quadratic convergence.

xn+1=12(xn+Sxn)x_{n+1} = \frac{1}{2}\left(x_n + \frac{S}{x_n}\right)
ALG_02
Ref: 2026-N

Newton-Raphson

Calculus-based Root Finding

A powerful numerical method that iteratively finds roots using the first derivative (tangent lines). Exhibits rapid quadratic convergence for well-behaved functions.

xn+1=xn−f(xn)f′(xn)x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
ALG_03
Ref: 2026-S

Secant Method

Finite-Difference Roots

A root-finding algorithm that uses a succession of roots of secant lines to better approximate a root of a function f.

xn+1=xn−f(xn)xn−xn−1f(xn)−f(xn−1)x_{n+1} = x_n - f(x_n)\frac{x_n - x_{n-1}}{f(x_n) - f(x_{n-1})}
ALG_04
Ref: 2026-I

Bisection Method

Root Finding & Interval Halving

A fundamental numerical method for finding the roots of continuous functions. The conceptual ancestor of binary search and gradient descent.

c=a+b2c = \frac{a+b}{2}
ALG_05
Ref: 2026-F

Fixed Point Iteration

Cobweb Plot Convergence

A method that rewrites f(x)=0 as x=g(x). By applying g repeatedly, a sequence is generated that can converge to a fixed point, visualized through cobweb plots.

xn+1=g(xn)x_{n+1} = g(x_n)

Volume II — Optimization & Learning

A direct continuation of the first volume. Here, we bridge the gap between root-finding and gradient descent, shifting our objective from finding a zero to finding a minimum. We follow the progression through stochasticity, adaptive learning rates, and momentum, ultimately culminating in Adam—the workhorse of modern machine learning.

ALG_06
Ref: 2026-GD

Gradient Descent

First-Order Optimization

The workhorse of machine learning. A first-order optimization algorithm that takes steps proportional to the negative of the gradient of a function at the current point.

θt+1=θt−η∇J(θt)\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)
ALG_07
Ref: 2026-SGD

Stochastic Gradient Descent

Noisy Optimization

An iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable).

θt+1=θt−η∇Ji(θt)\theta_{t+1} = \theta_t - \eta \nabla J_i(\theta_t)
ALG_08
Ref: 2026-AG

AdaGrad

Adaptive Gradients

An optimization algorithm with parameter-specific learning rates, adapting to the geometry of the data. Ideal for sparse features, but suffers from premature freezing.

θt+1=θt−ηGt+ϵgt\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t + \epsilon}} g_t
ALG_09

RMSProp

Root Mean Square Propagation

An optimization algorithm that resolves AdaGrad's radically diminishing learning rates by using a moving average of the squared gradients.

E[g2]t=βE[g2]t−1+(1−β)gt2E[g^2]_t = \beta E[g^2]_{t-1} + (1-\beta)g_t^2
ALG_10
Ref: 2026-A

Adam Optimizer

Adaptive Moment Estimation

An optimization algorithm that combines the best properties of the AdaGrad and RMSProp algorithms to provide an optimization heuristic for noisy, sparse gradients.

θt+1=θt−ηv^t+ϵm^t\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t