# Backpropagation from first principles

> Learn backpropagation with interactive gradient diagrams, the chain rule and worked neural network examples. Check the calculations using finite differences.

Source: https://krishparekhdev.com/blog/backpropagation
Published: 2026-08-09 (updated 2026-10-12)
Tags: ai, ml, backpropagation, autograd

---

Even before you learn backpropagation, it helps to sit with the maths of why it
exists at all, and why it matters.

Almost every machine learning algorithm runs the same loop. You do a forward
pass: inputs go through the model, you get an output. You compare that output to
what you wanted and get a loss. Then you move backwards from that loss, figure
out which way each weight should change, and take a gradient descent step. Repeat
until the loss is small enough to live with.

Inside that loop, the output does not come from nowhere. It depends on a pile of
hidden states and internal weights. Change one weight a little and the output
nudges. Change another and it nudges differently. Training is the problem of
finding, for every weight, how much it is responsible for the loss you just saw.

That is what backpropagation is for. You take the loss from the forward pass and
ask: how did this particular weight influence that number? Once you know the
direction and the size of that influence, you can update the weight. Without a
cheap way to answer that for every weight at once, the training loop above does
not scale past toy models.

So the rest of this post is the maths behind that question. Slope first, because
that is what "how much does this weight nudge the loss" actually means. Then how
a layer is just $y = mx + c$. Then every gradient in a tiny network walked by
hand, checked against finite differences, and matched in code.

## Why a derivative shows up

For each weight the training question is concrete. If I make this bigger, does
the loss go up or down, and by how much?

You can answer that by measuring. Take a weight at $W_2[0] = 1.20$ with loss
`0.2330`. Push it to `1.21`, loss goes to `0.2350`. Pull it to `1.19`, loss goes
to `0.2310`. Up is worse, so the weight needs to come down.

Divide change-in-loss by change-in-weight and you get about `0.199`. Shrink the
nudge and both sides land on `0.198877`. That number is $dL/dW_2[0]$. Measuring
works.

It also costs about two forward passes per weight. On a model with 175 billion
parameters that is hundreds of billions of forwards for one training step. Nobody
trains that way. Backpropagation gives you every $dL/dw$ in one backward pass
that costs roughly the same as a single forward pass. That is why the algorithm
matters in practice.

Before you can see how it does that, you need the thing it is computing: slope.

## Slope, without the textbook voice

Slope is a comparison between two places. Two points $(x_1, y_1)$ and
$(x_2, y_2)$, rise over run:

$$
\frac{y_2 - y_1}{x_2 - x_1}
$$

On a straight line $y = mx + c$, that ratio is always $m$. Pick any two points
on $y = 2x + 1$ and you get `2.00`. Move them closer and you still get `2.00`.
One number describes the whole line.

Curves are different. Write the second point as $x + h$:

$$
\frac{f(x + h) - f(x)}{h}
$$

For $f(x) = x^2$ at $x = 0.30$ with $h = 1$, that is `1.60`. That is the slope
of the line through the two points, not the slope of the curve at `0.30`. It
depends on how wide you made the gap.

Drag $h$ down and watch it settle:

[Interactive example: LimitSlider. See the source page for controls. The surrounding text explains the example.]

At $h = 1.5$ you get `2.100000`. At $h = 0.01` you get `0.610000`. The numbers
are clearly heading for `0.600000`. That limit is the derivative at the point.
Training wants exactly this for every weight: the slope of the loss right where
you are, not an average across a wide interval.

## The bit that feels like a trick

You want the slope at one point, so the two points have to become the same
point, so $h = 0$. Plug that in and you get $0/0$. Undefined.

What you do instead is rewrite the expression while $h$ is still allowed to be
nonzero. For $f(x) = x^2$:

$$
\frac{(x + h)^2 - x^2}{h}
= 2x + h
$$

Now $h = 0$ is fine. You get $2x$. You cancelled first, then asked what happens
as $h$ gets small. That is all

$$
\frac{dy}{dx} = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}
$$

is doing. Read the limit as "what this closes in on," not "substitute zero."

Left and right sides have to agree. For $x^2$ they do. For $|x|$ at zero they
do not, which is why ReLU's derivative at exactly 0 is a framework convention
(PyTorch returns 0) rather than something the maths hands you. If you want the
visual version of this argument, 3Blue1Brown's
[Paradox of the Derivative](https://www.3blue1brown.com/lessons/derivatives) is
still the best one I have seen.

## The one idea

A neural network layer is $y = mx + c$.

$$
z_1 = x W_1 + b_1
$$

$W_1$ is $m$. $b_1$ is $c$. More than one output means several of these lines
side by side. The weights are the slopes.

So the local derivatives are free:

- move an input by 1 → output moves by $W_1$ → $dz_1/dx = W_1$
- move a weight by 1 → output moves by the input it was multiplying →
  $dz_1/dW_1 = x$

Each operation only needs its own local slope. Multiply gate: $d(xy)/dx = y$.
Add gate: $d(x+y)/dx = 1$. Backprop is those local rules composed from the loss
backwards. That is the whole algorithm. Everything else is bookkeeping.

### Why you cannot just stack lines

Because stacking lines gives you a line:

$$
z = 3(2x + 1) - 4 = 6x - 1
$$

A hundred linear layers collapse into one. Depth buys nothing. Put $\tanh$ or
ReLU between them and the composition stops being linear. Depth starts to matter,
and you are back on a curve, where the slope depends on where you are sitting.

For $\tanh$, the derivative is $1 - \tanh(x)^2$. At `0.30` that is `0.9151`. At
`-1.00` it is `0.4200`. Same function, different slope. That is also why gradient
descent takes small steps. The gradient is only trustworthy near the weights you
measured it at.

## A network small enough to finish on paper

Two inputs, two hidden units with $\tanh$, one linear output, squared error:

$$
z_1 = x W_1 + b_1
\qquad
a_1 = \tanh(z_1)
\qquad
\hat{y} = a_1 W_2 + b_2
\qquad
L = \tfrac{1}{2}(\hat{y} - y)^2
$$

| | value |
| --- | --- |
| $x$ | `[0.50, -1.00]` |
| $y$ | `0.50` |
| $W_1$ | `[[0.80, -0.40], [0.20, 0.60]]` |
| $b_1$ | `[0.10, -0.20]` |
| $W_2$ | `[1.20, -0.70]` |
| $b_2$ | `0.30` |

Forward pass: $z_1 = [0.30, -1.00]$, $a_1 = [0.2913, -0.7616]$,
$\hat{y} = 1.1827$. Target was `0.50`, so
$L = \tfrac{1}{2}(0.6827)^2 = 0.2330$.

## Why you cannot read $dL/dW_1$ off the loss directly

$W_2$ is one operation from the loss. Easy. $W_1[0][0]$ is four operations away.
It changes $z_1$, which changes $a_1$, which changes $\hat{y}$, which changes
$L$.

If moving $x$ by 1 moves $u$ by 3, and moving $u$ by 1 moves $L$ by 2, then
moving $x$ by 1 moves $L$ by 6:

$$
\frac{dL}{dw} = \frac{dL}{du} \cdot \frac{du}{dw}
$$

$du/dw$ is local to one gate. $dL/du$ is whatever the step on the right already
figured out. Start at the loss. Walk left. Multiply as you go.

The [CS231n backprop notes](https://cs231n.github.io/optimization-2/) do this as
circuit diagrams if you want a second pass with different pictures.

## Walking it backwards

Solid boxes held values on the way forward. Dashed boxes are parameters. Step
through and the gradients fill in:

[Interactive example: BackpropGraph. See the source page for controls. The surrounding text explains the example.]

First number out is the error, `0.6827`. For squared loss that is just
$\hat{y} - y$. The rest of the pass is that one number being split up and handed
left.

Steps 2 and 4 are the $y = mx + c$ derivatives again. Step 5 is where $\tanh$
can kill signal. Unit 2 sat at `-0.7616`, so its gradient got multiplied by
`0.42` on the way past. One layer of that is fine. Twelve is a problem. Step 8
is what makes this recursive: $dL/dx$ is the same kind of thing as $dL/da_1$, so
a deeper network just keeps walking.


If a value is used in two places (residual, shared embedding, same weight every
timestep), it gets a gradient from each consumer and those add. That is why
autograd does `+=` into `.grad`, and why you call `zero_grad()` between steps.


## Why it has to run backwards

You can push derivatives either direction.

| | one pass gives you | passes needed |
| --- | --- | --- |
| forward mode | one input's effect on everything | one per parameter |
| reverse mode | everything's effect on one output | one |

Training has millions of inputs (the parameters) and one output (the scalar
loss). Reverse mode matches that shape. Flip it and forward mode would be
cheaper. Baydin et al's
[survey](https://arxiv.org/abs/1502.05767) is the reference if you want the
general AD framing.

The cost is memory. Backward needs values from the forward pass ($a_1$ shows up
in $dL/dW_2$), so activations stay alive until their step.
[Gradient checkpointing](https://arxiv.org/abs/1604.06174) drops some and
recomputes them later. You buy memory back with extra compute.

## Same thing when the batch gets bigger

Write $\delta_\ell = dL/dz_\ell$. Per layer it is three lines:

$$
\delta_{L} = \hat{y} - y
\qquad
\delta_{\ell} = \left(\delta_{\ell+1} W_{\ell+1}^{\top}\right) \odot f'(z_{\ell})
$$

$$
\frac{dL}{dW_{\ell}} = a_{\ell-1}^{\top} \delta_{\ell}
\qquad
\frac{dL}{db_{\ell}} = \sum_{\text{batch}} \delta_{\ell}
$$

Four examples give four rows of $\delta_1$. One matmul sums them:

[Interactive example: MatMul. See the source page for controls. The surrounding text explains the example.]

First row of $\Delta_1$ is the `[0.7497, -0.2007]` from the hand pass.


Softmax alone has an ugly Jacobian. Softmax followed by cross-entropy collapses
to $\delta = p - y$. That is why the API is `cross_entropy(logits, labels)` and
not `cross_entropy(softmax(logits), labels)`. Softmax twice flattens gradients.


## Taking a step

Gradients point uphill. Training subtracts:

$$
w \leftarrow w - \eta \, \frac{dL}{dw}
$$

With $\eta = 0.1$ on these numbers, loss goes `0.2330` → `0.0672`. Learning rate
matters more than anything else in this loop:

[Interactive example: LineChart. See the source page for controls. The surrounding text explains the example.]

`1.0` steps far enough that the local slope no longer applies. Same idea as the
curve earlier, now inside the training loop.

## What breaks when you go deep

At step 5 the gradient got multiplied by $\tanh'(z_1) < 1$. Do that $n$ times
across $n$ layers, plus a multiply by each weight matrix, and early layers see a
product of $2n$ factors. Products of many numbers do not stay near 1.

Twelve dense layers, width 48:

[Interactive example: LineChart. See the source page for controls. The surrounding text explains the example.]

Sigmoid's derivative tops out at `0.25`. Twelve layers is enough to stop early
layers learning. Most of the defaults you take for granted are aimed at this:

- **ReLU**: derivative exactly 1 wherever it is active
- **Init** ([Xavier](https://proceedings.mlr.press/v9/glorot10a.html), He): keep
  the weight factor near 1
- **Residuals**: $d(x + f(x))/dx = 1 + f'(x)$, so a path of 1 always exists
- **Norm layers**: keep units out of the flat regions

Exploding gradients are the same product leaning the other way. Usual fix: clip
the global gradient norm before the update.

## Code that matches the hand numbers

```python title="backprop.py" showLineNumbers
import numpy as np

x = np.array([[0.5, -1.0]])
y = np.array([[0.5]])

W1 = np.array([[0.8, -0.4], [0.2, 0.6]])
b1 = np.array([[0.1, -0.2]])
W2 = np.array([[1.2], [-0.7]])
b2 = np.array([[0.3]])

z1 = x @ W1 + b1
a1 = np.tanh(z1)
z2 = a1 @ W2 + b2
loss = 0.5 * np.sum((z2 - y) ** 2)

d2 = z2 - y
dW2 = a1.T @ d2
db2 = d2.sum(axis=0, keepdims=True)

da1 = d2 @ W2.T
d1 = da1 * (1 - a1**2)
dW1 = x.T @ d1
db1 = d1.sum(axis=0, keepdims=True)
dx = d1 @ W1.T
```

Shapes are the check. `dW1` comes out `2×2` like `W1`. Wrong shape usually means
a transpose in the wrong place, and you catch it before you look at values.

Autograd is the same graph, built as the forward runs:

```python title="torch_version.py"
import torch

x = torch.tensor([[0.5, -1.0]])
y = torch.tensor([[0.5]])

W1 = torch.tensor([[0.8, -0.4], [0.2, 0.6]], requires_grad=True)
b1 = torch.tensor([[0.1, -0.2]], requires_grad=True)
W2 = torch.tensor([[1.2], [-0.7]], requires_grad=True)
b2 = torch.tensor([[0.3]], requires_grad=True)

loss = 0.5 * ((torch.tanh(x @ W1 + b1) @ W2 + b2 - y) ** 2).sum()
loss.backward()

print(W1.grad)
```

`backward()` walks the recorded ops in reverse and does the same eight
multiplies. PyTorch's
[autograd notes](https://docs.pytorch.org/docs/stable/notes/autograd.html) cover
what gets saved and when the graph is freed.

## Check it the slow way

The poke-and-measure method from the top is too slow to train with. It is the
right way to verify a backward pass you wrote yourself. It is also the only tool
that catches a gradient that is wrong in a way that still "trains," just badly.

```python title="grad_check.py"
def grad_check(loss_fn, param, analytic, eps=1e-6):
    for index in np.ndindex(param.shape):
        original = param[index]

        param[index] = original + eps
        up = loss_fn()
        param[index] = original - eps
        down = loss_fn()
        param[index] = original

        numeric = (up - down) / (2 * eps)
        assert abs(numeric - analytic[index]) < 1e-7, index
```

On this network every parameter matches to within $10^{-10}$:

| parameter | analytic | numeric |
| --- | --- | --- |
| `W1[0,0]` | 0.374853 | 0.374853 |
| `W1[0,1]` | −0.100349 | −0.100349 |
| `W1[1,0]` | −0.749707 | −0.749707 |
| `W1[1,1]` | 0.200699 | 0.200699 |
| `W2[0]` | 0.198877 | 0.198877 |
| `W2[1]` | −0.519934 | −0.519934 |
| `b2` | 0.682691 | 0.682691 |


Use central differences, not $(L(w+\varepsilon) - L(w)) / \varepsilon$. One-sided
error is $O(\varepsilon)$ instead of $O(\varepsilon^2)$. Also use float64: at
float32, $\varepsilon = 10^{-6}$ is small enough that the difference is mostly
rounding noise.


Day to day you call `loss.backward()` and move on. Worth writing by hand when
you ship a custom `autograd.Function`, or when a model trains but badly and you
need to know which factor in that product of $2n$ terms went to zero.

If you want one level further down, Karpathy's
[micrograd](https://github.com/karpathy/micrograd) is about 100 lines of scalar
autograd walking the same kind of graph.
