Teaching Small Language Models to Remember: Giving LLMs a Notebook with Differentiable Neural Computers

"Large models memorize the world in their weights. Small models need a notepad." The Problem: Small Models Forget Facts Large Language Models (LLMs) like GPT-4 are remarkably good at recalling facts — "Delhi is the capital of India," "Einstein developed the theory of relativity" — because they have billions of parameters acting as a massive, compressed knowledge store. The model bakes facts into weights during pre-training, and retrieval is implicit in the forward pass. But what happens when you