Chapter 8: RMS Normalisation and Residual Connections

What You'll Build Two architectural patterns that make deep networks trainable: RMSNorm (keeps activations from exploding or vanishing) and residual connections (gives gradients a highway to flow through). Depends On Chapters 1-2 (Value), Chapter 5 (Helpers). The Problem They Solve As data flows through many Linear operations and activation functions like ReLU (both of which you've already seen), the magnitude of the numbers can drift. They grow huge, or shrink to near-zero. Both are catastrophi