Gradient Descent Playground & Math-for-Deep-Learning Roadmap

The Playground

Gradient descent repeats the update x ← x − η·f′(x). Pick a loss surface, choose a starting point and learning rate η, then step or run. Watch convergence, oscillation, escape from local minima, and outright divergence.

Loss function f(x)Learning rate η: 0.10Start x₀

Iteration0

x4.200

f(x)17.64

|gradient|8.400

Ready. Press Run to optimize.

Try η = 1.05 on the convex bowl to see oscillating divergence, or start the double well at x₀ = 0.1 to see which basin captures the iterate. On the wavy surface, small η gets trapped in local minima while a well-chosen larger η can hop over ridges.

Why It Works, and When It Does Not

Descent lemma

If f has L-Lipschitz gradients, then f(x−η∇f) ≤ f(x) − η(1−Lη/2)‖∇f‖². Choosing η ≤ 1/L guarantees every step decreases the loss. That inequality is the whole reason the algorithm works.

Convergence rates

Convex + smooth gives O(1/k) error after k steps; strong convexity upgrades this to a geometric (linear) rate. These are provable limits, not folklore, and they come straight from the analysis in convex optimization texts below.

Failure modes

Too large η on curvature L makes updates overshoot and diverge; nonconvex surfaces admit local minima and saddle points. You can reproduce every one of these behaviors in the playground above.

Math-for-Deep-Learning Roadmap

A hand-curated path through the classics. Check items off as you finish them; progress is saved in your browser.

Calculus & Linear Algebra

Linear Algebra Done RightSheldon Axler. Proof-first vector spaces, eigenvalues, spectral theorem.
CalculusMichael Spivak. Rigorous single-variable calculus and real proofs.
Introduction to Linear AlgebraGilbert Strang. Intuition, four fundamental subspaces, SVD.

Rigor & Uncertainty

Principles of Mathematical AnalysisWalter Rudin. Limits, continuity, and the machinery behind convergence proofs.
Introduction to ProbabilityBlitzstein & Hwang. Random variables, expectation, concentration.
All of StatisticsLarry Wasserman. Fast, honest tour of statistical inference for ML people.

Why Training Converges

Convex OptimizationBoyd & Vandenberghe. Duality, KKT, and the analysis behind gradient methods.
Lectures on Convex OptimizationYurii Nesterov. Where convergence rates and lower bounds are actually proved.
Matrix ComputationsGolub & Van Loan. Numerical linear algebra that makes big models feasible.

Machine Learning Proper

Pattern Recognition and Machine LearningChristopher Bishop. Probabilistic view of learning, from regression to graphical models.
Understanding Machine Learning: From Theory to AlgorithmsShalev-Shwartz & Ben-David. PAC learning, VC dimension, provable algorithm limits.
Deep LearningGoodfellow, Bengio & Courville. The canonical modern text tying it all together.