Rate Limiting in LLM Applications: Why You Need It and How to Build It

TL;DR: Rate limiting for LLM APIs requires counting tokens, not requests. A single 200K-token context window costs as much as 50 normal API calls. This post covers the gap between request-count limits and token-aware limits, and walks through implementation at both the application layer and the gateway layer. This post assumes familiarity with LLM APIs (OpenAI, Anthropic), basic Redis or caching concepts, and running AI applications in production. Why Standard Rate Limiting Falls Short Most deve