LLM-as-Judge: Automated Quality Gate for LLM Outputs in Production

LLM-as-Judge is a pattern where one language model evaluates another model's outputs against defined criteria. An automatic quality gate: every response gets checked before reaching the user, or after, for monitoring. Standard production monitoring metrics (200 OK, latency 340ms, rate limits within bounds) are useless for assessing quality — the model can hallucinate in 15% of responses while HTTP status codes tell you nothing about it. Manual review doesn't scale. One person can handle 100 requ