The hero analogyA ball rolling downhill in fog
Neural networks are large and built from non-linear functions, so we cannot solve for the minimum of the loss function with algebra. Instead we feel our way there. At any point, the gradient tells us the direction of steepest ascent, so we step the opposite way, downhill. The fog matters: the algorithm never sees the whole landscape, only the local slope where the ball currently stands.
Try it yourselfRoll the ball
The curve below is a loss function L(x) with a small bump: one shallow local dip and one deeper true minimum. Drop the ball anywhere, choose a learning rate η, and step. Watch how tiny η crawls, moderate η converges, and huge η overshoots and bounces.
What the ball teaches youFour lessons to tell your friend
01The gradient is a compass, not a map
∇L points uphill at your exact location. Negating it gives the locally steepest way down. It says nothing about what lies beyond the fog, which is why descent is iterative.
02Learning rate is stride length
Too small and training takes forever. Too large and the ball leaps across the valley, oscillates, or even climbs out and diverges. Try η above 0.9 in the simulator to see it explode.
03Local minima are real dips
Start the ball near x = -2 and it settles in the shallow dip, never finding the deeper valley. In very high-dimensional networks this is less catastrophic than it looks here, but the fog never lifts.
04Convergence is when the slope vanishes
Near a minimum the gradient shrinks toward zero, so steps naturally get smaller. When |∇L| is tiny, the ball has effectively stopped: training has converged.
Why this mattersFrom one ball to a billion parameters
A real neural network is not one ball on a 2D hill; it is a single point in a landscape with millions or billions of dimensions, one per weight. Backpropagation computes the gradient in all of those dimensions at once, and the exact same update rule, subtract the learning rate times the gradient, nudges every weight downhill together. Everything you felt in the simulator, stride length, fog, dips, vanishing slopes, scales up directly to how modern models are trained.