Every LLM Eval Library Has the Same Bug: Stochastic Judges Used as Deterministic Oracles

Book: LLM Observability Pocket Guide: Picking the Right Tracing & Evals Tools for Your Team Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You ran the eval. Pass rate is 87%. You ship. Your colleague reruns the same suite, same model, same prompts, same dataset, ten minutes later. Pass rate is 81%.