Blog 1: Foundations of Gradient Descent

How neural networks learn — and why the obvious approach breaks immediately Every optimizer you'll ever use — Adam, AdamW, Lion, LAMB — is an answer to a problem that gradient descent creates. To understand why those answers exist, you need to feel the problem first. The loss surface is a landscape you can't see Imagine you're blindfolded, standing somewhere on a hilly terrain. Your only tool is a stick: you can poke the ground around you and measure the slope. Your goal is to reach the lowest v