Gradient Descent Math Lab

step 0
θ = (0.00, 0.00)
L(θ) = 0.000
‖∇L‖ = 0.000

converging

The Update Rule

v ← β·v − η·∇L(θ) θ ← θ + v η=0.020 β=0.00 → Δθ=(0,0)

learning rate η 0.020 momentum β 0.00

Surface: Convex Bowl

Regimes to Recognize

Converge	η < 2/λ_max (largest curvature): steps shrink, L → minimum smoothly.
Oscillate	η near the stability limit: the ball ping-pongs across steep walls while creeping along shallow ones — the classic ill-conditioning picture.
Diverge	η > 2/λ_max: each step overshoots by more than it corrects; L blows up. On a quadratic this bound is exact — provable, not folklore.

Why the math matters

Ball = parameters θ; the arrow is −∇L, always the locally steepest descent direction (first-order Taylor: L(θ+d) ≈ L(θ)+∇L·d).

Trail = optimization trajectory. Momentum (β) accumulates past gradients — a leaky integrator that damps oscillation and accelerates along consistent directions.

Saddle points, not local minima, dominate high-dimensional losses; gradient noise (SGD) helps escape them. That result comes from random-matrix theory — more evidence the theory pays rent.

A 3-line convergence proof sketch

For L(θ) = ½θᵀAθ (A symmetric positive-definite), one GD step gives θ⁺ = (I − ηA)θ. Errors shrink iff every eigenvalue of (I − ηA) has magnitude < 1, i.e. |1 − ηλᵢ| < 1 for all i, which rearranges to 0 < η < 2/λ_max. The slowest mode contracts by factor (1 − ηλ_min) — so the iteration count scales with the condition number κ = λ_max/λ_min. Try the Bowl surface (κ = 4): the shallow x-axis is always the last to settle. This one derivation explains learning-rate warmup, why normalization layers help (they improve conditioning), and why Adam rescales per-coordinate.

Batch GD vs SGD vs Mini-batch

Batch GD (shown here) uses the exact gradient — deterministic, but each step costs a full pass over the data. SGD estimates ∇L from one sample: unbiased (E[ĝ] = ∇L) but noisy; the noise acts like temperature, helping escape saddles and sharp minima. Mini-batch (the practical default, 32–512 samples) trades variance ∝ 1/batch for hardware parallelism. Classic Robbins–Monro conditions for SGD convergence: Ση_t = ∞ and Ση_t² < ∞ — the reason learning-rate schedules decay over time rather than staying constant.