Akash Kumar Sinha

10 min read

Schedule a call

Building an Autograd Engine from Scratch: A Beginner's Guide to How Neural Networks Learn

When people first start learning deep learning, they write lines of code like:

loss.backward()

And somehow, miraculously, every single number inside the neural network knows exactly whether it should go up or down to make the model smarter.

It feels like magic. But underneath, there is no magic at all-just a clever little program called an Autograd engine (short for Automatic Differentiation).

To understand how this actually works, I built one from scratch in Python (the complete implementation is available on GitHub ). In this guide, we are going to break it down step by step. No complicated math symbols, no PhD required-just simple, plain English and clear Python code.

Step 1: The Tensor (A Number with Memory)

In standard Python, numbers have no memory. When you define a = 3.0, Python simply holds the value 3.0 in isolation-it has no awareness of where that number originated, what operations produced it, or how downstream calculations depend on it.

For a neural network to learn, every number needs to remember two important things:

  • What is its value right now? (We call this data).
  • How much does this number affect the final error? (We call this grad, short for gradient).

Think of the gradient as a score:

  • If grad is positive (+2.0), it means: “If you increase this number, the error goes up. Make it smaller!”
  • If grad is negative (-2.0), it means: “If you increase this number, the error goes down. Make it bigger!”

Here is the simple Python class that wraps a number:

class Tensor: def __init__(self, data, label='', _children=()): self.data = data # The actual number (e.g. 3.0) self.grad = 0.0 # How to change it (starts at 0) self.label = label # A friendly name for debugging self._prev = set(_children) # Who created this number? self._backward = lambda: None # Function to propagate gradients backward

That's the whole foundation! Just a number that remembers who its parents are and how to pass feedback back to them.

Step 2: Doing Math and Building the Family Tree

When we add or multiply two tensors, we don't just compute the answer. We also create a link connecting the parents to the child.

In Python, you can make objects support the + sign using the special __add__ method:

def __add__(self, other): # If adding a regular number, wrap it in a Tensor first if not isinstance(other, Tensor): other = Tensor(other) # 1. Forward: compute the new answer out = Tensor(self.data + other.data, _children=(self, other), label='+') # 2. Backward: how feedback flows back to parents def _backward(): self.grad += out.grad other.grad += out.grad out._backward = _backward return out

Now look at multiplication (*):

def __mul__(self, other): if not isinstance(other, Tensor): other = Tensor(other) # 1. Forward out = Tensor(self.data * other.data, _children=(self, other), label='*') # 2. Backward def _backward(): self.grad += other.data * out.grad other.grad += self.data * out.grad out._backward = _backward return out

Notice how intuitive multiplication is: each input's impact is scaled by the other input's value! If you have out = 3 * x, changing x by 1 changes the result by 3.

Step 3: The Chain Rule (Passing the Blame)

Calculus textbooks often make the Chain Rule sound intimidating, but the core intuition is surprisingly simple: it is just tracing responsibility backward through a chain of contributors:

  • Suppose Santa is working with Banta, and Banta is working with Jeeta.
  • If Jeeta makes a mistake, Banta needs to know how much of that mistake was Banta's fault.
  • Then Banta turns around to Santa and says: “Hey, Jeeta blamed me by this much, and you influenced me by that much, so your blame is those two multiplied together.”

That is literally the chain rule:

Impact of Santa on final error = (Impact of Banta on error) * (Impact of Santa on Banta)

Every time we run an operation, the little _backward() function remembers this exact rule so it can pass the blame back when asked.

Step 4: The Sneaky Bug: Why += Saves the Day

Take a look at how we update the gradient in _backward:

self.grad += out.grad # Why '+=' and NOT '=' ?

Why must we write += instead of simple assignment =?

Imagine you use the same variable twice:

x = Tensor(3.0) y = x + x # y is 2 * x

Because y = 2 * x, if you increase x by 1, y increases by 2. So the gradient of x must be 2.0.

If we used self.grad = out.grad, the first branch would set x.grad = 1.0, and then the second branch would overwrite it with 1.0, throwing away the first half!

By using +=, the gradients from both branches add together: 1.0 + 1.0 = 2.0. It works flawlessly every time.

Step 5: Activations (Teaching Computers to Curve)

If all a neural network could do was add and multiply, it would only ever be able to draw straight lines. But real-world problems-like detecting a face, recognizing a voice, or separating intertwined patterns-are curved and twisty.

To bend lines into curves, we add simple non-linear functions called activations .

ReLU: The Simple Light Switch

ReLU (Rectified Linear Unit) sounds fancy, but it's the simplest function on earth: if the number is positive, keep it. If it's negative, throw it away and return 0.

def relu(self): out = Tensor(max(0, self.data), _children=(self,), label='relu') def _backward(): # If output was > 0, pass the gradient through. Otherwise, stop. self.grad += (out.data > 0) * out.grad out._backward = _backward return out

Sigmoid: Squeezing Numbers Between 0 and 1

When our neural network needs to make a final prediction (like: “Is this 80% likely to be Class A?”), we need the output to be between 0.0 and 1.0.

Sigmoid squeezes any number-whether it's -500 or +1000-into a smooth S-curve between 0 and 1.

Here is a neat trick we implemented to keep the computer from crashing:

def sigmoid(self): x = self.data # Safe computation to prevent math overflow if x >= 0: s = 1 / (1 + math.exp(-x)) else: exp_x = math.exp(x) s = exp_x / (1 + exp_x) out = Tensor(s, _children=(self,), label='sigmoid') def _backward(): # The beautiful derivative of sigmoid: s * (1 - s) self.grad += s * (1 - s) * out.grad out._backward = _backward return out

Checking if x >= 0 ensures we never ask Python to calculate something huge like e^500, which would crash the program.

Step 6: Running Backward in the Right Order

Now for the main event: when you call loss.backward(), how do we visit every node in the graph?

Imagine dominoes falling forward. When we want to send feedback backward, we have to start from the very last domino (the loss) and work backward to the start.

If a node provides input to multiple operations down the line, we must wait until all those operations have finished delivering their feedback before that node passes feedback to its own parents.

We solve this with a simple graph algorithm called a Topological Sort :

def backward(self): # 1. The error of the loss with itself is always 1.0 self.grad = 1.0 # 2. Build the order of nodes from end to beginning topo = [] visited = set() def build_topo(v): if v not in visited: visited.add(v) for child in v._prev: build_topo(child) topo.append(v) build_topo(self) # 3. Walk through them in reverse and trigger each _backward()! for v in reversed(topo): v._backward()

In less than 20 lines of Python, the entire backpropagation algorithm is complete.

Step 7: Building a Neural Network like Lego Blocks

Now that our Tensor can track math and calculate gradients, assembling a neural network is like putting Lego pieces together:

  • A Neuron: Takes a few inputs, multiplies each by a weight (w), adds a bias (b), and applies ReLU.
  • A Layer: A team of neurons sitting side-by-side looking at the same inputs.
  • An MLP (Multi-Layer Perceptron): Layers stacked in a line, where the output of Layer 1 feeds into Layer 2, and so on.

class Neuron: def __init__(self, nin): self.w = [Tensor(random.uniform(-1, 1)) for _ in range(nin)] self.b = Tensor(0.0) def __call__(self, x): # sum(w * x) + b act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b) return act.relu() def parameters(self): return self.w + [self.b]

And to stack multiple layers together into a full network:

class MLP: def __init__(self, nin, nouts): # e.g. 2 inputs -> 16 hidden -> 16 hidden -> 1 output sz = [nin] + nouts self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))] def __call__(self, x): for layer in self.layers: x = layer(x) return x def parameters(self): return [p for layer in self.layers for p in layer.parameters()]

Step 8: Training the Model on the 'Two Moons' Problem

To see this engine in action, we test it on a classic benchmark: the make_moons dataset. It looks like two interlocking crescent moons that a straight line cannot possibly separate.

Here is the entire training loop in 4 simple steps:

model = MLP(2, [16, 16, 1]) learning_rate = 0.001 for epoch in range(100): # 1. Forward Pass: Make predictions on all data points predictions = [model([Tensor(x[0]), Tensor(x[1])]) for x in X] # 2. Compute Loss (How far off were we?) loss = compute_cross_entropy(predictions, y) # 3. Backward Pass: Figure out who to blame loss.backward() # 4. Update: Nudge all weights in the right direction for p in model.parameters(): p.data -= learning_rate * p.grad # Take a small step downhill p.grad = 0.0 # Reset gradient for next round

As the loop runs, the error drops steadily with every epoch:

  • Epoch 1: Loss is high (~0.69). The network is just guessing randomly.
  • Epoch 50: Loss drops below 0.40. The network starts bending its boundary to wrap around the curves.
  • Epoch 100: Loss drops down to ~0.28. The crescents are clearly separated, and the model achieves high accuracy!

Wrapping Up

Building this system removes the mystique around modern AI:

  • Neural networks aren't thinking brains; they are networks of adjustable mathematical dials.
  • Backpropagation isn't magic; it's just applying the chain rule in reverse order across a family tree of operations.
  • Tools like PyTorch are essentially scaled-up, GPU-accelerated versions of this exact design.

Once you build an engine like this with your own hands, you stop treating AI like a black box-and start treating it like the fascinating engineering discipline it truly is.

The complete code implementation is available on GitHub at Akash-Kumar-Sinha/autograd . Feel free to star the repo, explore the source, and experiment with it!