Interpret optimizer warnings as simulation diagnostics
Does leaving this canvas prove that Adam failed to converge?
Compare optimizer trajectories on four authored two-dimensional surfaces. The page estimates gradients numerically and applies different moment updates. Its warnings describe local heuristics in this demonstration, not a benchmark ranking or a diagnosis of a real neural-network training run.
Diverged means a coordinate exceeded magnitude 10 or became NaN. A warning can also mean more than 15 gradient-direction reversals before step 100. The saddle objective is unbounded below, so movement away from the origin can decrease loss without approaching a minimum. No training data, held-out evaluation or shared stochastic gradient sequence is used.
Try a worked example
For the default saddle f(x,y) = x² - y² at (-1.8, 1.5), forward differences with h = 0.001 give approximately (-3.599, -3.001). Fixed SGD at learning rate 0.1 moves to about (-1.4401, 1.8001). Bias-corrected Adam’s first coordinate steps are about +0.1 each, giving (-1.7, 1.6). This illustrates coordinate scaling; it does not show that one algorithm solves an objective with no lower bound.
Relate moments to the original Adam method
Kingma and Ba describe Adam using estimates of first and second gradient moments with bias correction. The second moment is an average of squared gradients, not a centered variance estimate. This page uses those ideas on simple functions; the original paper does not validate its warning thresholds, moving-minimum scenario or chosen hyperparameters. Kingma and Ba: Adam—A Method for Stochastic Optimization
Compare runs with the necessary context
The moving minimum depends on simulation time, which advances with animation frames rather than a measured wall-clock training process. The noisy-cliff example draws separate random perturbations for each optimizer, so its paths do not share identical noise. Sliders change ongoing updates without clearing accumulated moments; use a relaunch when comparing a fresh configuration. Reset restores parameter defaults and the seed but preserves the chosen surface and pause state.
Read exports and loss labels carefully
Pause before exporting a snapshot. JSON contains hyperparameters, stored losses, reversal counts, divergence flags and recent coordinates, capped at 150 path points per optimizer. It omits the simulation clock, step count and moment accumulators, so it is not a complete replay checkpoint. Stored loss is computed before each coordinate update; the displayed infinity after divergence is a warning label rather than proof the stored loss is infinite.
Sources and further reading