Your LLM Judge Has Opinions. They're Not About Quality.

When your eval score goes up, the natural conclusion is that your model got better. But there's another explanation: your LLM judge has systematic biases, and your latest change happened to produce outputs that those biases favor. This is the core problem with LLM-as-a-judge evaluation. The problem isn't that LLM judges are bad. It's that their scores are a mixture of quality signal and bias signal, and most eval setups make no attempt to separate the two. When you optimize for a score without k