Chapter 3: The Tokenizer - Text to Numbers and Back

What You'll Build A Tokenizer class that converts between characters and integer IDs, plus a special BOS (Beginning of Sequence) token. Depends On Nothing. This chapter is independent of Chapters 1-2. It sits here because the next chapter (Bigram) needs it. Why Characters? Production LLMs use subword tokenizers (like BPE) that merge frequent character sequences into single tokens, giving them a vocabulary of ~100K tokens. MicroGPT uses the simplest possible tokenizer: one token per unique charac