This page uses an authored quadratic loss, one half x squared plus ten y squared. Its gradient is x, twenty y. Start at x minus one point eight and y one point five. The initial adaptive step is fixed at zero point zero five from the initial curvature guess ten. Blue x and yellow y share coordinate units on the vertical axis. We will show only the first exact source update. Apply that initial step. Subtract zero point zero five times each gradient component. The x coordinate moves from minus one point eight to minus one point seven one. The y coordinate moves from one point five exactly to zero. The two vertices are actual optimizer states; moving between them is teaching interpolation, not extra optimizer steps. The first update removes the y contribution to this quadratic loss. Loss falls from twenty four point one two to one point four six two zero five. The two directions respond differently because the source gradient multiplies y by twenty, while it multiplies x by one. This is this specific source function, not training performance evidence. The source then estimates local curvature from the norm of gradient change divided by displacement. It uses theta zero point seven zero seven one divided by that estimate for the next learning rate. Here blue is the curvature estimate on the left scale, zero to twenty one; yellow is the learning rate on the right scale, zero to one. These are separate explicit axes, not the same numerical units. Advance the exact first eight updates. After the first update, the estimated curvature is about nineteen point nine six four and the next rate about zero point zero three five four. Once displacement lies along x with y already zero, the next estimate becomes one and the next rate becomes zero point seven zero seven one. Watch the estimate fall while the rate rises. The source retains that unit curvature estimate along the quadratic x direction. This local ratio controls the next update, while the initial step was zero point zero five. The page names an adaptive method, but the video teaches its actual theta over local estimate rule. This authored rule and one example do not establish a convergence theorem. Now compare two actual source trajectories on one shared loss axis from zero to twenty five. Blue uses the adaptive rule and yellow uses fixed learning rate zero point one. Both start at the same coordinates and loss twenty four point one two. The quadratic y update for the fixed method is y next equals minus y: its sign flips without reducing its magnitude. Advance twelve exact source updates. The adaptive first step takes y to zero and subsequent steps shrink x. Its loss approaches zero. The fixed method keeps y at plus or minus one point five, so the y loss contribution stays twenty two point five. Its x coordinate contracts by zero point nine each update, but that cannot remove the unchanged y loss. After twelve updates, the adaptive loss is tiny, while the fixed loss remains above twenty two point five. The shared scale makes the difference visible. These are deterministic trajectories for the displayed quadratic and chosen rates. Other surfaces can produce nonfinite values; the page fixed convergence label is not a diagnostic, and its original independently scaled charts do not directly compare numeric loss heights.
Distinguish a local curvature estimate from a convergence guarantee
What does the adaptive learning rate measure along a trajectory?
Compare three optimizer paths on authored two-dimensional functions. The adaptive path estimates gradient change divided by displacement, then uses theta divided by that estimate for its next step. These small examples illustrate sensitivity to curvature; they do not establish optimizer performance on a training dataset.
The convergence status is a fixed label, not a test of gradients, loss reduction or finite values. Each plotted loss series is independently rescaled, so vertical positions across colors do not directly compare numeric losses. Saddle and wave examples are nonconvex; no general convergence guarantee follows from their pictures.
Try a worked example
On the default quadratic f(x,y) = 0.5(x² + 20y²), the seed (-1.8, 1.5) has gradient (-1.8, 30) and loss 24.12. The adaptive first step is 0.05 from its initial estimate of 10, giving (-1.71, 0) and loss 1.46205. The fixed step 0.10 instead gives (-1.62, -1.5) and loss 23.8122. Its y-coordinate reverses sign without shrinking. Compare these values in the exported trajectories rather than judging overlapping chart heights.
Compare this demonstration with the original algorithm
Malitsky and Mishchenko describe two principles: controlling how quickly a step size increases and respecting local curvature. Their paper gives convergence results under stated assumptions, including convexity. This page implements a simplified theta-over-estimate update without an explicit step-growth safeguard, so citing the paper does not transfer its theorem to this implementation. Malitsky and Mishchenko: Adaptive Gradient Descent without Descent
Interpret controls and telemetry
The base learning-rate slider changes fixed gradient descent only. Adam uses its own fixed 0.1 learning rate, while adaptive descent begins at 0.05 and subsequently uses theta over the local estimate. Theta changes the adaptive update. The current loss and local estimate follow the playback step, but Final Point and average estimate summarize the complete 50-step run even before playback finishes. A smaller displayed loss is not always meaningful on the unbounded saddle function.
Use exported data for numerical comparisons
Export Run preserves the surface key, initial point, theta, fixed learning rate and all three trajectories, including losses. The adaptive trajectory also records local estimates and learning rates. Replay restarts the current configuration; it does not restore the original surface or seed. Negative losses are clamped when drawing the logarithmic chart, so use exported numeric values when interpreting the saddle or wave.
Original paper on adaptive local geometry and step-size control; the simplified page does not reproduce all algorithm safeguards or theorem assumptions.