The Algorithms Lab
Deconstructing the mathematical engines of modern computation. Interactive explorations of foundational algorithms, stripped of abstraction.
Volume I — Roots & Convergence
Babylonian Method
Heron's Root Finding
The world's oldest known numerical algorithm. Computes square roots by iteratively averaging a guess with its complement, demonstrating quadratic convergence.
Newton-Raphson
Calculus-based Root Finding
A powerful numerical method that iteratively finds roots using the first derivative (tangent lines). Exhibits rapid quadratic convergence for well-behaved functions.
Secant Method
Finite-Difference Roots
A root-finding algorithm that uses a succession of roots of secant lines to better approximate a root of a function f.
Bisection Method
Root Finding & Interval Halving
A fundamental numerical method for finding the roots of continuous functions. The conceptual ancestor of binary search and gradient descent.
Fixed Point Iteration
Cobweb Plot Convergence
A method that rewrites f(x)=0 as x=g(x). By applying g repeatedly, a sequence is generated that can converge to a fixed point, visualized through cobweb plots.
Volume II — Optimization & Learning
Gradient Descent
First-Order Optimization
The workhorse of machine learning. A first-order optimization algorithm that takes steps proportional to the negative of the gradient of a function at the current point.
Stochastic Gradient Descent
Noisy Optimization
An iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable).
AdaGrad
Adaptive Gradients
An optimization algorithm with parameter-specific learning rates, adapting to the geometry of the data. Ideal for sparse features, but suffers from premature freezing.
RMSProp
Root Mean Square Propagation
An optimization algorithm that resolves AdaGrad's radically diminishing learning rates by using a moving average of the squared gradients.
Adam Optimizer
Adaptive Moment Estimation
An optimization algorithm that combines the best properties of the AdaGrad and RMSProp algorithms to provide an optimization heuristic for noisy, sparse gradients.