Why I spun my benchmark into its own repo (and why every dev tool with a benchmark should)

This week I shipped a benchmark for code-intelligence MCP servers and posted the results — including the cases where my own tool lost. Within 36 hours, the maintainer of one of the competing tools (jcodemunch-mcp) had shipped three back-to-back releases addressing specific findings the benchmark exposed. Adding new tests for those fixes then exposed a symmetric blind spot in my own parser. I shipped a fix. That whole loop — competing maintainers iterating on the same eval, in opposite directions