PyTorch Demystified: Tensors to Training Loop

Six questions stall almost every PyTorch beginner: what is a tensor, why matrix multiplication, what loss.backward() actually does, how gradient descent "learns", what nn.Module is for, and how a training loop fits together. The 3D model below is gradient descent — the rest of the page maps each line of a real loop onto it.

Loss surface L(w₁, w₂) — drag to rotate

lr = 0.10

—

loss

—

‖gradient‖

steps

Try lr = 0.10 (smooth descent), lr = 0.95 (oscillation), lr = 1.20 (divergence — loss explodes). This is exactly why learning rate is the most important hyperparameter.

The five-line training loop

for xb, yb in dataloader:
    pred = model(xb)          # forward pass
    loss = loss_fn(pred, yb)  # scalar error
    loss.backward()           # compute ∂loss/∂w
    optimizer.step()          # w -= lr * grad
    optimizer.zero_grad()     # reset for next batch

The 3D ball is optimizer.step(). The surface height is the loss; the ball's (x, z) position is the two weights (w₁, w₂). Each click of step() runs one update: w ← w − lr·∇L(w). loss.backward() is what computed that arrow — the red gradient vector — via the chain rule, walking backward through the computation graph autograd recorded during the forward pass.

The six answers

What even is a tensor?

A tensor is just an n-dimensional array with superpowers: it can live on a GPU and it can remember how it was computed. A scalar is a 0-D tensor, a vector 1-D, a matrix 2-D. A batch of 32 RGB images at 224×224 is a 4-D tensor of shape [32, 3, 224, 224]. Shapes are the #1 source of beginner bugs — print x.shape constantly.

Why matrix multiplication?

A linear layer computes y = xWⅤ + b. One matrix multiply applies every neuron's weighted sum to every sample in the batch simultaneously — a [32, 784] @ [784, 128] multiply does 32 × 128 = 4,096 dot products of length 784 (~3.2M multiply-adds) in one GPU-optimized call. Deep learning is fast because GPUs are matrix-multiply machines.

What does loss.backward() do?

During the forward pass, autograd records every operation into a directed acyclic graph. loss.backward() traverses that graph in reverse, applying the chain rule at each node, and deposits ∂loss/∂w into each parameter's .grad attribute. It does not change any weights — that's the optimizer's job. Gradients accumulate by default, which is why you must call zero_grad() each iteration.

How does gradient descent actually learn?

The gradient points in the direction of steepest increase of loss, so stepping in the negative gradient direction reduces loss. With loss L(w₁, w₂) = w₁² + 1.6·w₂² (the bowl above), the gradient at (2, 1) is (4, 3.2). With lr = 0.1 the update is (2, 1) → (1.6, 0.68). Repeat a few hundred times and you're at the minimum. Real networks do this in millions of dimensions, but the picture is the same.

What is nn.Module?

A container class that (1) registers every nn.Parameter you assign so model.parameters() can hand them all to the optimizer, (2) defines forward() so calling model(x) runs your computation, and (3) nests — a ResNet is Modules inside Modules inside Modules. It's bookkeeping, not magic.

How does the loop fit together?

Forward pass builds the graph and produces a loss. Backward pass fills in gradients. optimizer.step() nudges every weight downhill by lr × its gradient. zero_grad() wipes the slate. One pass over all batches = one epoch. Training is just this loop until the loss stops improving on data the model has never seen.