Blog 3: Adaptive Learning Rate Methods (Part 1)

When one learning rate isn't enough — per-parameter scaling and the decay problem Momentum gave the optimizer a memory across time. Adaptive methods give it a memory across parameters. Both are necessary. Neither is sufficient alone. The problem with a single learning rate Every optimizer we've covered so far shares one architectural assumption: a single scalar η governs every parameter in the model. This seems reasonable until you think about what parameters actually experience during training.