My LLM Keeps Failing in Production. Here's What I Built to Fix It Automatically.

A story about debugging, frustration, and why I stopped doing it manually. I want to tell you about a Tuesday afternoon I had about a year ago. I was staring at a Langfuse dashboard. Fifty-three failed traces. All from the same RAG pipeline I'd spent three weeks building. The failures were spread across different users, different inputs, different times of day — but they all shared one thing: the model was returning JSON that didn't match the schema my downstream code expected. I knew what the f