Backpropagation from first principles

Even before you learn backpropagation, it helps to sit with the maths of why it exists at all, and why it matters.

Almost every machine learning algorithm runs the same loop. You do a forward pass: inputs go through the model, you get an output. You compare that output to what you wanted and get a loss. Then you move backwards from that loss, figure out which way each weight should change, and take a gradient descent step. Repeat until the loss is small enough to live with.

Inside that loop, the output does not come from nowhere. It depends on a pile of hidden states and internal weights. Change one weight a little and the output nudges. Change another and it nudges differently. Training is the problem of finding, for every weight, how much it is responsible for the loss you just saw.

That is what backpropagation is for. You take the loss from the forward pass and ask: how did this particular weight influence that number? Once you know the direction and the size of that influence, you can update the weight. Without a cheap way to answer that for every weight at once, the training loop above does not scale past toy models.

So the rest of this post is the maths behind that question. Slope first, because that is what "how much does this weight nudge the loss" actually means. Then how a layer is just $y = mx + c$. Then every gradient in a tiny network walked by hand, checked against finite differences, and matched in code.

Why a derivative shows up

For each weight the training question is concrete. If I make this bigger, does the loss go up or down, and by how much?

You can answer that by measuring. Take a weight at $W_2[0] = 1.20$ with loss 0.2330. Push it to 1.21, loss goes to 0.2350. Pull it to 1.19, loss goes to 0.2310. Up is worse, so the weight needs to come down.

Divide change-in-loss by change-in-weight and you get about 0.199. Shrink the nudge and both sides land on 0.198877. That number is $dL/dW_2[0]$. Measuring works.

It also costs about two forward passes per weight. On a model with 175 billion parameters that is hundreds of billions of forwards for one training step. Nobody trains that way. Backpropagation gives you every $dL/dw$ in one backward pass that costs roughly the same as a single forward pass. That is why the algorithm matters in practice.

Before you can see how it does that, you need the thing it is computing: slope.

Slope, without the textbook voice

Slope is a comparison between two places. Two points $(x_1, y_1)$ and $(x_2, y_2)$, rise over run:

On a straight line $y = mx + c$, that ratio is always $m$. Pick any two points on $y = 2x + 1$ and you get 2.00. Move them closer and you still get 2.00. One number describes the whole line.

Curves are different. Write the second point as $x + h$:

For $f(x) = x^2$ at $x = 0.30$ with $h = 1$, that is 1.60. That is the slope of the line through the two points, not the slope of the curve at 0.30. It depends on how wide you made the gap.

Drag $h$ down and watch it settle:

At $h = 1.5$ you get 2.100000. At $h = 0.01you get0.610000. The numbers are clearly heading for 0.600000`. That limit is the derivative at the point. Training wants exactly this for every weight: the slope of the loss right where you are, not an average across a wide interval.

The bit that feels like a trick

You want the slope at one point, so the two points have to become the same point, so $h = 0$. Plug that in and you get $0/0$. Undefined.

What you do instead is rewrite the expression while $h$ is still allowed to be nonzero. For $f(x) = x^2$:

Now $h = 0$ is fine. You get $2x$. You cancelled first, then asked what happens as $h$ gets small. That is all

is doing. Read the limit as "what this closes in on," not "substitute zero."

Left and right sides have to agree. For $x^2$ they do. For $|x|$ at zero they do not, which is why ReLU's derivative at exactly 0 is a framework convention (PyTorch returns 0) rather than something the maths hands you. If you want the visual version of this argument, 3Blue1Brown's Paradox of the Derivative is still the best one I have seen.

The one idea

A neural network layer is $y = mx + c$.

$W_1$ is $m$. $b_1$ is $c$. More than one output means several of these lines side by side. The weights are the slopes.

So the local derivatives are free:

• move an input by 1 → output moves by $W_1$ → $dz_1/dx = W_1$
• move a weight by 1 → output moves by the input it was multiplying → $dz_1/dW_1 = x$

Each operation only needs its own local slope. Multiply gate: $d(xy)/dx = y$. Add gate: $d(x+y)/dx = 1$. Backprop is those local rules composed from the loss backwards. That is the whole algorithm. Everything else is bookkeeping.

Why you cannot just stack lines

Because stacking lines gives you a line:

A hundred linear layers collapse into one. Depth buys nothing. Put $\tanh$ or ReLU between them and the composition stops being linear. Depth starts to matter, and you are back on a curve, where the slope depends on where you are sitting.

For $\tanh$, the derivative is $1 - \tanh(x)^2$. At 0.30 that is 0.9151. At -1.00 it is 0.4200. Same function, different slope. That is also why gradient descent takes small steps. The gradient is only trustworthy near the weights you measured it at.

A network small enough to finish on paper

Two inputs, two hidden units with $\tanh$, one linear output, squared error:

Forward pass: $z_1 = [0.30, -1.00]$, $a_1 = [0.2913, -0.7616]$, $\hat{y} = 1.1827$. Target was 0.50, so $L = \tfrac{1}{2}(0.6827)^2 = 0.2330$.

Why you cannot read $dL/dW_1$ off the loss directly

$W_2$ is one operation from the loss. Easy. $W_1[0][0]$ is four operations away. It changes $z_1$, which changes $a_1$, which changes $\hat{y}$, which changes $L$.

If moving $x$ by 1 moves $u$ by 3, and moving $u$ by 1 moves $L$ by 2, then moving $x$ by 1 moves $L$ by 6:

$du/dw$ is local to one gate. $dL/du$ is whatever the step on the right already figured out. Start at the loss. Walk left. Multiply as you go.

The CS231n backprop notes do this as circuit diagrams if you want a second pass with different pictures.

Walking it backwards

Solid boxes held values on the way forward. Dashed boxes are parameters. Step through and the gradients fill in:

First number out is the error, 0.6827. For squared loss that is just $\hat{y} - y$. The rest of the pass is that one number being split up and handed left.

Steps 2 and 4 are the $y = mx + c$ derivatives again. Step 5 is where $\tanh$ can kill signal. Unit 2 sat at -0.7616, so its gradient got multiplied by 0.42 on the way past. One layer of that is fine. Twelve is a problem. Step 8 is what makes this recursive: $dL/dx$ is the same kind of thing as $dL/da_1$, so a deeper network just keeps walking.

Why it has to run backwards

You can push derivatives either direction.

Training has millions of inputs (the parameters) and one output (the scalar loss). Reverse mode matches that shape. Flip it and forward mode would be cheaper. Baydin et al's survey is the reference if you want the general AD framing.

The cost is memory. Backward needs values from the forward pass ($a_1$ shows up in $dL/dW_2$), so activations stay alive until their step. Gradient checkpointing drops some and recomputes them later. You buy memory back with extra compute.

Same thing when the batch gets bigger

Write $\delta_\ell = dL/dz_\ell$. Per layer it is three lines:

Four examples give four rows of $\delta_1$. One matmul sums them:

First row of $\Delta_1$ is the [0.7497, -0.2007] from the hand pass.

Taking a step

Gradients point uphill. Training subtracts:

With $\eta = 0.1$ on these numbers, loss goes 0.2330 → 0.0672. Learning rate matters more than anything else in this loop:

1.0 steps far enough that the local slope no longer applies. Same idea as the curve earlier, now inside the training loop.

What breaks when you go deep

At step 5 the gradient got multiplied by $\tanh'(z_1) < 1$. Do that $n$ times across $n$ layers, plus a multiply by each weight matrix, and early layers see a product of $2n$ factors. Products of many numbers do not stay near 1.

Twelve dense layers, width 48:

Sigmoid's derivative tops out at 0.25. Twelve layers is enough to stop early layers learning. Most of the defaults you take for granted are aimed at this:

• ReLU: derivative exactly 1 wherever it is active
• Init (Xavier, He): keep the weight factor near 1
• Residuals: $d(x + f(x))/dx = 1 + f'(x)$, so a path of 1 always exists
• Norm layers: keep units out of the flat regions

Exploding gradients are the same product leaning the other way. Usual fix: clip the global gradient norm before the update.

Code that matches the hand numbers

import numpy as np
 
x = np.array([[0.5, -1.0]])
y = np.array([[0.5]])
 
W1 = np.array([[0.8, -0.4], [0.2, 0.6]])
b1 = np.array([[0.1, -0.2]])
W2 = np.array([[1.2], [-0.7]])
b2 = np.array([[0.3]])
 
z1 = x @ W1 + b1
a1 = np.tanh(z1)
z2 = a1 @ W2 + b2
loss = 0.5 * np.sum((z2 - y) ** 2)
 
d2 = z2 - y
dW2 = a1.T @ d2
db2 = d2.sum(axis=0, keepdims=True)
 
da1 = d2 @ W2.T
d1 = da1 * (1 - a1**2)
dW1 = x.T @ d1
db1 = d1.sum(axis=0, keepdims=True)
dx = d1 @ W1.T

Shapes are the check. dW1 comes out 2×2 like W1. Wrong shape usually means a transpose in the wrong place, and you catch it before you look at values.

Autograd is the same graph, built as the forward runs:

import torch
 
x = torch.tensor([[0.5, -1.0]])
y = torch.tensor([[0.5]])
 
W1 = torch.tensor([[0.8, -0.4], [0.2, 0.6]], requires_grad=True)
b1 = torch.tensor([[0.1, -0.2]], requires_grad=True)
W2 = torch.tensor([[1.2], [-0.7]], requires_grad=True)
b2 = torch.tensor([[0.3]], requires_grad=True)
 
loss = 0.5 * ((torch.tanh(x @ W1 + b1) @ W2 + b2 - y) ** 2).sum()
loss.backward()
 
print(W1.grad)

backward() walks the recorded ops in reverse and does the same eight multiplies. PyTorch's autograd notes cover what gets saved and when the graph is freed.

Check it the slow way

The poke-and-measure method from the top is too slow to train with. It is the right way to verify a backward pass you wrote yourself. It is also the only tool that catches a gradient that is wrong in a way that still "trains," just badly.

def grad_check(loss_fn, param, analytic, eps=1e-6):
    for index in np.ndindex(param.shape):
        original = param[index]
 
        param[index] = original + eps
        up = loss_fn()
        param[index] = original - eps
        down = loss_fn()
        param[index] = original
 
        numeric = (up - down) / (2 * eps)
        assert abs(numeric - analytic[index]) < 1e-7, index

On this network every parameter matches to within $10^{-10}$:

Day to day you call loss.backward() and move on. Worth writing by hand when you ship a custom autograd.Function, or when a model trains but badly and you need to know which factor in that product of $2n$ terms went to zero.

If you want one level further down, Karpathy's micrograd is about 100 lines of scalar autograd walking the same kind of graph.