Reproducing Chinchilla Scaling on a Budget

Training a 70B parameter model costs millions of dollars. Scaling laws exist so you don't have to guess how to spend that budget. Here's what I learned reproducing them on a free GPU. Introduction Scaling laws are basically rules that tell us how model performance improves as you increase quantities such as model size, dataset size, and compute. Instead of guessing "bigger models = better", scaling laws gives a mathematical relationship between: model size (N, number of parameters) dataset size