PyTorch, Demystified: From Tensor to Training Loop

Six questions stall almost every PyTorch beginner: what is a tensor, why matrix multiplication, what loss.backward() actually does, how gradient descent "learns", what nn.Module is for, and how a training loop fits together. The 3D model below is gradient descent — the rest of the page maps each line of a real loop onto it.

Loss surface L(w₁, w₂) — drag to rotate
—
loss
—
‖gradient‖
0
steps

Try lr = 0.10 (smooth descent), lr = 0.95 (oscillation), lr = 1.20 (divergence — loss explodes). This is exactly why learning rate is the most important hyperparameter.

The five-line training loop
for xb, yb in dataloader:
    pred = model(xb)          # forward pass
    loss = loss_fn(pred, yb)  # scalar error
    loss.backward()           # compute ∂loss/∂w
    optimizer.step()          # w -= lr * grad
    optimizer.zero_grad()     # reset for next batch

The 3D ball is optimizer.step(). The surface height is the loss; the ball's (x, z) position is the two weights (w₁, w₂). Each click of step() runs one update: w ← w − lr·∇L(w). loss.backward() is what computed that arrow — the red gradient vector — via the chain rule, walking backward through the computation graph autograd recorded during the forward pass.

The six answers

What even is a tensor?
A tensor is just an n-dimensional array with superpowers: it can live on a GPU and it can remember how it was computed. A scalar is a 0-D tensor, a vector 1-D, a matrix 2-D. A batch of 32 RGB images at 224×224 is a 4-D tensor of shape [32, 3, 224, 224]. Shapes are the #1 source of beginner bugs — print x.shape constantly.
Why matrix multiplication?
A linear layer computes y = xWⅤ + b. One matrix multiply applies every neuron's weighted sum to every sample in the batch simultaneously — a [32, 784] @ [784, 128] multiply does 32 × 128 = 4,096 dot products of length 784 (~3.2M multiply-adds) in one GPU-optimized call. Deep learning is fast because GPUs are matrix-multiply machines.
What does loss.backward() do?
During the forward pass, autograd records every operation into a directed acyclic graph. loss.backward() traverses that graph in reverse, applying the chain rule at each node, and deposits ∂loss/∂w into each parameter's .grad attribute. It does not change any weights — that's the optimizer's job. Gradients accumulate by default, which is why you must call zero_grad() each iteration.
How does gradient descent actually learn?
The gradient points in the direction of steepest increase of loss, so stepping in the negative gradient direction reduces loss. With loss L(w₁, w₂) = w₁² + 1.6·w₂² (the bowl above), the gradient at (2, 1) is (4, 3.2). With lr = 0.1 the update is (2, 1) → (1.6, 0.68). Repeat a few hundred times and you're at the minimum. Real networks do this in millions of dimensions, but the picture is the same.
What is nn.Module?
A container class that (1) registers every nn.Parameter you assign so model.parameters() can hand them all to the optimizer, (2) defines forward() so calling model(x) runs your computation, and (3) nests — a ResNet is Modules inside Modules inside Modules. It's bookkeeping, not magic.
How does the loop fit together?
Forward pass builds the graph and produces a loss. Backward pass fills in gradients. optimizer.step() nudges every weight downhill by lr × its gradient. zero_grad() wipes the slate. One pass over all batches = one epoch. Training is just this loop until the loss stops improving on data the model has never seen.

The actual quadratic toy: loss, gradient and effective step rate

Read the explanation

Read the actual toy on this page: it runs JavaScript, not a PyTorch training job. Start with weights x equals three point six and z equals three point two. Its implemented loss is one quarter x squared plus zero point four z squared. Substitute those source weights. The blue x term contributes three point two four; the purple z term contributes four point zero nine six. Adding the two gives seven point three three six. Each fixed bar uses one hundred pixels per loss unit. These are exact initial quantities, not an animation of intermediate training states. Now perform a real source action: choose displayed learning rate zero point one and click Step. The implemented gradient at the initial point is one point eight, two point five six. The handler subtracts twice the displayed rate times that gradient. This factor of two matters: the effective multiplier is zero point two. The result is x three point two four and z two point six eight eight, with loss five point five one four five three seven six. Compare the two exact loss bars at sixty pixels per unit. The lower resulting loss follows the actual JavaScript update. The page does not execute a PyTorch optimizer. Return to the same initial point before comparing rates. The actual recurrence multiplies x by one minus the displayed rate, and z by one minus one point six times that rate. At zero point one, the factors are zero point nine and zero point eight four: both coordinates approach zero without changing sign. At one point two, the factors are negative zero point two and negative zero point nine two. The signs alternate, while magnitudes still shrink. The four exact loss states in each column remain visible at fifty five pixels per unit. A faster displayed rate does not mean divergence here. This comparison applies to this implemented quadratic toy, not a trained neural network.

Super generates helpful tools and automates fact-checking across the internet proactively. If you enjoyed this tool, build your own with Super and share it with a friend.