Learn the math, not just the API

Gradient Descent Playground & Math-for-Deep-Learning Roadmap

Understand why gradient descent works, watch it fail when the learning rate is wrong, and follow a hand-curated reading path through the classic mathematics that deep learning is built on.

The Playground

Gradient descent repeats the update x ← x − η·f′(x). Pick a loss surface, choose a starting point and learning rate η, then step or run. Watch convergence, oscillation, escape from local minima, and outright divergence.

Iteration0
x4.200
f(x)17.64
|gradient|8.400

Ready. Press Run to optimize.

Try η = 1.05 on the convex bowl to see oscillating divergence, or start the double well at x₀ = 0.1 to see which basin captures the iterate. On the wavy surface, small η gets trapped in local minima while a well-chosen larger η can hop over ridges.

Why It Works, and When It Does Not

Descent lemma

If f has L-Lipschitz gradients, then f(x−η∇f) ≤ f(x) − η(1−Lη/2)‖∇f‖². Choosing η ≤ 1/L guarantees every step decreases the loss. That inequality is the whole reason the algorithm works.

Convergence rates

Convex + smooth gives O(1/k) error after k steps; strong convexity upgrades this to a geometric (linear) rate. These are provable limits, not folklore, and they come straight from the analysis in convex optimization texts below.

Failure modes

Too large η on curvature L makes updates overshoot and diverge; nonconvex surfaces admit local minima and saddle points. You can reproduce every one of these behaviors in the playground above.

Math-for-Deep-Learning Roadmap

A hand-curated path through the classics. Check items off as you finish them; progress is saved in your browser.

Calculus & Linear Algebra

  • Sheldon Axler. Proof-first vector spaces, eigenvalues, spectral theorem.
  • Michael Spivak. Rigorous single-variable calculus and real proofs.
  • Gilbert Strang. Intuition, four fundamental subspaces, SVD.

Rigor & Uncertainty

  • Walter Rudin. Limits, continuity, and the machinery behind convergence proofs.
  • Blitzstein & Hwang. Random variables, expectation, concentration.
  • Larry Wasserman. Fast, honest tour of statistical inference for ML people.

Why Training Converges

  • Boyd & Vandenberghe. Duality, KKT, and the analysis behind gradient methods.
  • Yurii Nesterov. Where convergence rates and lower bounds are actually proved.
  • Golub & Van Loan. Numerical linear algebra that makes big models feasible.

Machine Learning Proper

  • Christopher Bishop. Probabilistic view of learning, from regression to graphical models.
  • Shalev-Shwartz & Ben-David. PAC learning, VC dimension, provable algorithm limits.
  • Goodfellow, Bengio & Courville. The canonical modern text tying it all together.

Gradient descent step size controls contraction or overshoot

Read the explanation

For the source convex bowl x squared, gradient is two x. At starting x four point two and learning rate point one, subtracting rate times gradient gives three point three six. Bars share eighty pixels per x unit. This educational browser computes the update directly, not through a training API. Those x values have loss seventeen point six four and eleven point two eight nine six. Bars share twenty pixels per loss unit. The update moves opposite the positive gradient and lowers loss for this example. Lower loss after one step is evidence for this specific surface and rate, not a universal convergence guarantee. For the x squared bowl, each update multiplies x by one minus twice the learning rate. Rates point one and point three give multipliers point eight and point four. Bars share four hundred pixels per multiplier magnitude. Larger rates may overshoot or diverge; other source surfaces have different curvature. Genuine native Step and Reset, full actual metric states, all canvas buffers, native resize and first-footer checks preserve the educational tool. Clipboard trace and automatic-run timing remain separately unverified; no native download exists.

Super generates helpful tools and automates fact-checking across the internet proactively. If you enjoyed this tool, build your own with Super and share it with a friend.