Gradient Descent — The Deep Dive

Q12: How Does a Machine Walk Downhill? — The Derivative Becomes a Compass

Manish: Last chapter ended on a promise: that one quiet fact — the derivative is zero at the bottom of a valley — is how machines learn. And in the attention chapters we kept saying "training nudges the weights" without ever opening that box. I want to open it now. What is gradient descent? Not as a buzzword — I want to walk through it slowly enough that it feels inevitable, like something I would have come up with myself if you handed me the problem.

Response: Then let's hand you the problem, because gradient descent is genuinely something you would have invented. It is one of those ideas that sounds advanced and turns out to be almost embarrassingly simple — which is exactly why it works at the scale of trillion-parameter models without breaking.

Here is the whole chapter in advance:

---

1. First, the Problem That Demands It

No tool makes sense without the problem it answers. So forget machines for a moment. Here is the problem in its purest form:

Notice the shape of this situation, because it is the shape of all machine learning:

1. There is a setting you control (the dial).
2. There is a wrongness score you can measure (how far from 21°C).
3. You want the setting that makes the wrongness as small as possible.
4. And crucially — you cannot see the whole picture. You can only probe where you are.

That wrongness score has a name in machine learning: the loss (or error). And the problem — find the setting that minimizes the loss — is called optimization. Gradient descent is the simplest honest answer to it.

 wrongness
 (loss)  │ ╲                            ╱
         │  ╲                          ╱
         │   ╲                        ╱
         │    ╲                      ╱
         │     ╲_                  _╱
         │       ╲_              _╱
         │         ╲____    ____╱
         │              ╲__╱   ← the bottom: least wrong
         └───────────────┬───────────── dial setting
                    we want THIS

If you could see this whole curve, you'd just look at it and point at the bottom. The entire difficulty — the fog — is that you can't. The curve might depend on millions of dials, and computing the loss even at one setting takes real work. You only ever get to stand at one point and probe locally.

So the question becomes: standing at one point, in fog, what's the smartest thing you can learn about which way to go?

You already own the answer. You built it last chapter.

---

2. The Derivative Was a Compass All Along

Recall what the derivative tells you at a point — the table from §5 of last chapter:

derivative here is...	the curve here is...
positive	climbing (as you increase the setting, loss goes up)
negative	falling (as you increase the setting, loss goes down)
zero	flat — a bottom, a top, or a plateau
large in size	steep
small in size	gentle

Now read that table again, but as a lost hiker. It's a compass:

• Derivative positive? Uphill is to the right. So step left.
• Derivative negative? Uphill is to the left. So step right.
• Derivative zero? The ground is flat. Stop — you may have arrived.

One sentence captures all three rows:

And notice the bonus hiding in the last two rows of the table. The derivative doesn't just give a direction — it gives a size. Steep slope, big derivative: you're far from the bottom, stride confidently. Gentle slope, small derivative: the ground is flattening, you're close — tiptoe. The compass automatically tells you to slow down as you arrive.

---

3. Watch It Run, by Hand

Trust, then verify — house rules. Take a loss curve simple enough to do with mental arithmetic:

$L(w) = w^2$

(Read: the wrongness is the square of the setting $w$. The best setting is obviously $w = 0$, where the loss is $0$. But pretend we can't see that — we're in fog. We only know how to compute the loss and its slope at wherever we stand.)

From last chapter, we derived — strips and corner, no spells — that the derivative of $w^2$ is $2w$.

The recipe: new setting = old setting − step factor × derivative. Choose a step factor of $0.1$, start standing at $w = 4$, and just... follow the compass:

stand at $w$	slope $2w$	step: $-0.1 \times$ slope	new $w$
$4$	$8$	$-0.8$	$3.2$
$3.2$	$6.4$	$-0.64$	$2.56$
$2.56$	$5.12$	$-0.512$	$2.048$
$2.048$	$4.096$	$-0.4096$	$1.638$
$1.638$	$3.277$	$-0.328$	$1.311$
...	...	...	...
after 20 steps			$w \approx 0.046$

Look at what happened — really look, because this table is the entire field of deep learning in miniature:

• Every step moved toward $0$, the true bottom — and nobody told it where the bottom was. The slope alone steered it.
• The steps got smaller on their own: $0.8$, then $0.64$, then $0.51$... As the valley flattened, the derivative shrank, so the strides shrank. It glides in for a landing.
• It never quite reaches $0$ — it approaches it. By now that should feel familiar: it's another sneak-up, the same converging walk you met in limits, now happening in the world instead of on paper.

  L = w²  │╲
          │ ╲
          │  ╲        ● start (w = 4)
          │   ╲      ╱
          │    ╲    ●
          │     ╲  ●          each hop: shorter than the last,
          │      ╲●●          because the slope itself is shrinking
          │       ●●●
          └────────●●●──────────── w
                   ↑
              bottom (w = 0)

---

4. The Step Factor: The One Knob That Can Ruin Everything

That $0.1$ we chose has a famous name: the learning rate. It answers the question the compass doesn't: the slope says which way and roughly how urgently — but how far should one stride actually be?

This is the one place where gradient descent can genuinely go wrong, so let's see both failure modes honestly.

Too small — say $0.0001$. Every step is a millimeter. You will reach the bottom... in a few million years. Nothing breaks; you just crawl.

Too large — say $1.1$. Now watch the same table explode. Standing at $w=4$, slope $8$, you step $-8.8$ — clear across the valley to $w = -4.8$, which is further from the bottom than where you started. The slope there is even bigger, so the next leap is even longer. Each bound overshoots worse than the last:

   too small:                 too large:
   │╲                         │╲              ╱│
   │ ╲                        │ ╲            ╱ │
   │  ╲                       │  ●──────────╱──┼──→ ●
   │   ╲                      │   ╲        ╱   │   leaping the valley,
   │    ●●●●●●●... (crawling) │    ●←─────╱────┘   landing higher
   │     ╲___                 │     ╲___╱          every time
   └──────────── w            └──────────── w
   safe but slow              diverges — loss grows!

There is a subtle beauty in why a sane learning rate works at all: because the derivative shrinks as you near the bottom, a fixed trust factor still produces shrinking steps. You don't need a schedule that says "slow down now" — the valley itself whispers it through the slope.

---

5. Now Add Dials: From Derivative to Gradient

Everything so far had one dial. A real model has millions — the attention chapter counted about 3 million knobs behind a single word. Does the idea survive?

Completely — and this is the only genuinely new step left in the whole subject, so let's take it slowly with two dials.

With two settings, the loss is no longer a curve over a line. It's a surface over a plane — a landscape. Wrongness is now literally altitude:

        loss (altitude)
          │      ╱╲ ╱╲
          │  ╱╲ ╱  ╳   ╲        a real landscape:
          │ ╱  ╳  ╱ ╲   ╲      ridges, bowls, saddles
          │╱  ╱ ╲╱   ╲  ╱╲
          └──────────────────  dial 1
         ╱   the bowl's bottom
        dial 2    = best settings

Standing on a hillside, "which way is downhill?" is no longer a left-or-right question — it's a full direction question, 360 degrees of options. Here's the resolution, and it's friendlier than it looks:

Ask two separate one-dial questions, each of which you already know how to answer:

1. Freeze dial 2. Nudge only dial 1. How does the loss respond? → that's an ordinary derivative — the slope in the east-west direction.
2. Freeze dial 1. Nudge only dial 2. How does the loss respond? → another ordinary derivative — the slope in the north-south direction.

(Each of these is called a partial derivative — "partial" simply because it tells only its own dial's part of the story.)

Now staple the answers into a list:

$\text{gradient} = \big[\ \text{slope along dial 1},\ \ \text{slope along dial 2}\ \big]$

That list — that's the whole mystery — is the gradient. And it has one lovely property, which you can feel without proof: a list of "how much does uphill-ness lie in each direction" is itself an arrow, and that arrow points exactly toward steepest uphill. So the rule from §2 doesn't change by a single word:

---

6. So What Is "Training," Actually?

Now we can open the box the attention chapters left taped shut, and there's almost nothing inside — in the best way.

A neural network is a function with dials: feed in an input, out comes a prediction, and the prediction depends on the millions of weights — every entry of $W_Q$, $W_K$, $W_V$ from the attention chapters is one dial. Then:

1. Predict. Show the model a sentence; it guesses the next word.
2. Score. Compare the guess to the truth → one number, the loss. How wrong, right now?
3. Feel the slope. Compute the gradient of the loss — for each of the millions of weights, "if this one weight nudged up a hair, would the wrongness rise or fall?" (The technique that computes all these slopes efficiently is backpropagation — a chapter of its own. For today: it's how the slopes get computed, not what they mean.)
4. Step. Nudge every weight slightly against its slope. The model just got a tiny bit less wrong.
5. Repeat. Millions of sentences, millions of steps, gliding down a loss landscape of unimaginably many dimensions.

flowchart LR
    P["predict"] --> S["score the wrongness<br/>(loss)"]
    S --> G["feel the slope<br/>(gradient)"]
    G --> U["step downhill<br/>(nudge every weight)"]
    U --> P

That should land with a small shock. The mechanism inside the most capable artifacts humans have built is the blindfolded hiker — the derivative from last chapter, in a loop. The depth of the resulting behavior comes from the landscape (shaped by data and architecture), not from any cleverness in the walk.

---

7. The Honest Fine Print

The house rule is to never oversell. Three truths that keep the picture honest:

---

8. The Family of Walkers: Variations Used in Practice

Truth 3 opened a door worth stepping through, because in the real world nobody runs the textbook version of gradient descent. What runs instead is a small family of variations — and each one is just a different answer to two practical questions:

1. How many examples do you check before each step? (How carefully do you read the compass?)
2. Do you trust only the current reading, or also remember the previous ones?

That's it. Every "optimizer" you'll ever hear named is some combination of answers to these two questions. Let's meet them as walkers.

Walker 1: The Perfectionist — (full-batch) Gradient Descent

Before every single step, check every example in the dataset and average all their opinions about the slope. The compass reading is exact — and ruinously expensive. With billions of examples, one step could take hours. Each step is perfect; there are almost no steps.

Walker 2: The Impulsive — Stochastic Gradient Descent (SGD)

The opposite extreme: read the compass from one random example and step immediately. Each reading is wildly noisy — one example's opinion of "downhill" can be badly unrepresentative — so the path staggers and zigzags. But the steps come fast, and on average the stagger drifts downhill. And recall from Truth 3: the wobble is partly a feature — it shakes the walker out of shallow dips.

Walker 3: The Sensible Middle — Mini-Batch Gradient Descent

Read the compass from a small random handful — 8, 16, 32 examples — average them, step. A handful of opinions cancels most of the noise (their individual errors point in different directions and wash out), yet costs almost nothing compared to the full dataset. This is the everyday workhorse: nearly all real training is mini-batch, even when people casually call it "SGD."

 the three walkers, same valley:

 full batch:        SGD:                mini-batch:
 │╲                 │╲                  │╲
 │ ●                │ ●╲ ╱╲             │ ●
 │  ╲               │  ╳  ╳ ╲           │  ╲●
 │   ● (one slow,   │ ╱ ●╲╱ ●╲          │   ●╲
 │   ╲  perfect     │     ╳   ●         │    ●╲      mostly smooth,
 │    ● step at     │    ╱ ╲ ╱╲●        │     ●●     occasionally wobbly,
 │    ╲ a time)     │   (drunk but      │      ●●●   always cheap
 │     ●             │    downhill)      │
 └──────── w        └──────── w         └──────── w

Walker 4: The Walker With Memory — Adam

The first three walkers share a blind spot: each step trusts only the current reading and treats every dial identically. Adam (Adaptive Moment Estimation) fixes both with two ideas, each of which is pure common sense:

• Momentum — remember your recent direction. If the last many readings all said "roughly east," don't let one noisy reading yank you north; keep a running average of the slope and lean on it. Like a ball rolling downhill: it builds speed along directions that keep agreeing, and barrels through small bumps and potholes instead of getting trapped by them.
• A per-dial learning rate — not all dials deserve the same trust. Some weights sit on consistently steep ground (take careful small steps there, or you'll overshoot); some sit on flat, quiet ground (take bolder steps, or you'll never get anywhere). Adam tracks how large each dial's slopes have typically been, and scales each dial's step size accordingly — bold where the ground is calm, careful where it's violent.

So Adam is mini-batch walking, plus a memory of where it's been heading, plus a separately tuned stride for every one of the millions of dials — all updated automatically as it walks. That's why it became the default optimizer for training large models: it removes most of the hand-tuning agony of §4's single global learning rate.

The family at a glance

walker	slope measured on	character of the walk	you choose
Gradient Descent	the entire dataset	exact, smooth, painfully slow	learning rate
SGD	one random example	fast, noisy, staggering	learning rate
Mini-Batch GD	a small random batch	fast and mostly steady — the workhorse	learning rate, batch size
Adam	mini-batches, usually	remembers direction, adapts stride per dial	learning rate, two memory-decay settings ($\beta_1$, $\beta_2$)

---

9. The Shallow Valley Problem — Escaping a False Bottom

Manish: Hold on — Truth 1 admitted the trap but never resolved it. The territory can have many valleys. Some are shallow dips, some are deep. The walker reaches the bottom of a shallow one, the ground reads flat, the compass says zero — and it stops, convinced it has arrived. But that's a false answer: if it just climbed up and over the ridge, it would find a much deeper valley on the other side. The walker, by its own rule, can never climb. So how does this ever work?

Response: Your diagnosis is exactly right — and let's state it at full strength before softening anything. The textbook walker of §3 is permanently trapped by the first shallow dip it falls into. The compass reads zero, the rule says "step proportional to the slope," zero slope means zero step, and it stands there forever, wrong and certain. No fix exists inside the pure rule.

 loss │╲
      │ ╲      ridge
      │  ╲    ╱╲
      │   ╲  ╱  ╲
      │    ╲╱    ╲              the textbook walker stops HERE,
      │     ●      ╲            a ridge away from the truth
      │  shallow    ╲    ╱
      │  dip         ╲  ╱
      │               ╲╱
      │                ●  ← the deep valley it never finds
      └──────────────────────── w

So how does real training survive? Not by one trick — by four, and you have already met the pieces. They just haven't been pointed at this problem yet.

Escape 1: The noisy compass can't sit still

Recall Truth 3 and the mini-batch walker: the slope is estimated from a random handful of examples, so every reading is slightly wrong — each batch gives the walker a small random shove. Now picture that walker standing in a shallow dip. The dip's walls are low — and the random shoves are, on average, bigger than the walls. The walker physically cannot rest there; the very next noisy reading kicks it over the rim.

But a deep valley has high walls — the same small shoves can't throw the walker out of it.

Escape 2: Momentum doesn't stop at the first flat ground

The Adam walker from §8 carries memory — a running average of recent slopes, which behaves like the speed of a rolling ball. A ball that has been rolling downhill for a while doesn't halt the instant the ground flattens; it coasts, and a shallow dip's far wall is too low to absorb its motion. It rolls in, rolls through, rolls out. Only a wall tall enough to spend all its accumulated speed can hold it — and tall walls belong to deep valleys.

Escape 3: Bold first, careful later

The learning rate need not stay fixed for the whole walk, and in real training it almost never does. The standard play: start large, shrink over time. Early on, strides are so long the walker steps clean over small dips without even registering them as valleys — at that stride length, they're just texture in the ground. As training proceeds the rate is gradually lowered, and the walker becomes capable of settling — but by then it has already coasted past the potholes and is far more likely to be inside something deep. Like panning for gold: shake hard first so the gravel can't settle, then gently, so only the heavy stuff stays down.

Escape 4: Don't send one hiker — send many

Nothing says the walk must start from one place. Start several walkers from different random starting weights, let each settle, keep whichever found the deepest resting point. The simplest, bluntest fix — and a real one for small models, though at the scale of large language models (where one walk costs millions of dollars) you mostly rely on Escapes 1 through 3.

And now the plot twist: the deepest valley was never the goal

Here is the part of your question that contains its own answer. You said we need the deep valley "in order to generalize" — and that instinct is right, but it points somewhere surprising: depth is not actually what generalization wants. What it wants is width.

Remember what the landscape is: wrongness measured on the training data. Tomorrow the model meets data it has never seen — which means it will be standing in a landscape that is almost this one, but shifted slightly, everywhere. Now compare two resting places:

      a narrow crack                a wide basin
      │    ││                       │
      │    ││                       │   ╲          ╱
      │    ││  ← deepest point      │    ╲___●___╱   ← slightly less deep
      │    ●│     on the map        │
      └──────── w                   └──────────── w
      landscape shifts a little:    landscape shifts a little:
      the crack moves — you're      the basin moves — you're
      suddenly HIGH on a wall       still on the floor

A narrow crack is depth without tolerance — the model is spectacular on exactly the training data and brittle the moment anything shifts. That is overfitting, drawn as terrain. A wide basin is forgiving: the test landscape differs a little from the training landscape, and a walker resting in a broad valley stays low even after the shift.

And now notice the quiet coincidence that makes deep learning work: the noisy walker can't rest in a crack anyway. Its wobble bounces it off narrow walls; only a basin wide enough to contain the wobble can hold it. The same noise that kicks the walker out of shallow dips also steers it away from sharp cracks and into wide basins — which are exactly the resting places that generalize.

---

10. The Dials Nobody Walks: Hyperparameters

One more piece of vocabulary, and it's worth a section because the name confuses people far more than the idea.

Notice that two kinds of dials have appeared in this chapter, and they live very different lives:

• The weights — the millions of dials the walk itself turns. Nobody sets these by hand; gradient descent finds them. They are what the model learns.
• The settings of the walk — learning rate, batch size... dials that shape how the walking happens. The walk cannot turn these, because they aren't part of the landscape — they're part of the walker. A human (or an automated search) must choose them before the walk begins.

That second kind has a name: hyperparameters. Hyper- as in "above" — parameters that sit above the training loop, steering it from outside.

You've already met two, in their natural habitats. Two more complete the everyday set:

hyperparameter	the question it answers	where the intuition lives
learning rate	how far do you trust one compass reading?	§4 — too small crawls, too large leaps the valley
batch size	how many examples per compass reading?	§8 — one is noisy, all is slow, a handful is the workhorse
epoch	how many times do you walk through the whole dataset?	below
dropout	how do you stop the model from memorizing instead of learning?	below

Epoch: laps around the dataset

Each mini-batch step uses only a handful of examples. Work through the entire dataset once, batch by batch — that's one epoch. One pass is rarely enough: the model was nearly random when it saw the early examples, so revisiting them later, with wiser weights, extracts more from the same data. Like rereading a difficult book — the second pass teaches you things the first couldn't, because of the first.

But there's a limit, and it's one of the deepest tensions in all of learning. Too many laps and the model stops learning the patterns in the data and starts memorizing the examples themselves — every quirk, every coincidence, every noise wrinkle. It becomes flawless on data it has seen and brittle on data it hasn't. That failure has a name — overfitting — and it's the difference between understanding the material and memorizing the answer key the night before the exam.

Dropout: training with a hand tied behind the back

Dropout is the strangest entry in the table, and the most charming once it clicks: during training, randomly silence a fraction of the network — say 10% of the neurons, a different random set each step — by setting their outputs to zero.

Why would deliberately breaking the network help? Because of what it forbids. Without dropout, a network can develop fragile internal arrangements — a few neurons quietly memorizing specific training examples, with the rest leaning on them. With dropout, no neuron can be relied upon — it might be silenced at any moment. So the network is forced to spread knowledge redundantly, encoding patterns in many places at once. Like a team where anyone might miss tomorrow's meeting: no critical knowledge can live in one head, so it ends up shared — and shared knowledge is robust knowledge. (At test time, nothing is dropped; the full network plays, healthier for having practiced shorthanded.)

---

11. Gradient Descent at Five Depths

In the house tradition — the same truth, descending:

At the school level:

An algorithm that minimizes a function by repeatedly stepping in the direction opposite to its gradient.

At the deeper mathematical level:

An iterative method that needs only local, first-order information — the slope where you stand — to make global progress on a problem too large to see whole. It trades omniscience for a loop.

At the philosophical level:

A proof that you don't need to see the destination to reach it. A purely local rule — of all the ways I could change right now, take the one that reduces wrongness fastest — compounds, step by step, into something that looks from the outside like purpose.

At the poetic level:

Gradient descent is humility, iterated: admit where you stand is wrong, feel which way is less wrong, take one small step, and have the patience to ask again — a million times.

At the first-principle level:

It is the marriage of the two great ideas already in hand. Differentiation supplies the compass: the best local direction, extracted from a vanishing nudge. Iteration supplies the legs. Neither alone moves a model an inch; looped together, they are how mathematics — which only ever described change — finally learned to cause it.

---

12. What to Carry Out of This Chapter

---

13. The Bridge Forward

One door in this chapter stayed deliberately shut. Step 3 of the loop said "compute the slope for each of millions of weights" — and inside a deep network, each weight affects the loss only through a long chain: weight → attention score → context vector → prediction → loss. How do you trace blame backward through all those layers, for millions of weights, efficiently?

That trick has a name we've now bumped into twice: backpropagation. It is the chain rule of differentiation — how nudges pass through chained functions — turned into one of the most consequential algorithms ever written. It deserves its own chapter, built the same way: slowly, by hand, until it feels inevitable.

Differentiation reads the instant; integration gathers the whole. Gradient descent points the reading downhill and walks. Backpropagation is how the walk affords a million legs.

The hiker has a compass and knows how to walk. Next: how the compass gets built, layer by layer, inside the machine.

---

← Back: Differentiation and Integration | Back to Index | Related: Self-Attention in Transformers