Why Your Non-Significant Benchmark Result Might Be a Power Problem (Not a Model Problem)

In Week 11, Tenacious-Bench reported: Delta A = -2.34 pts, 95% CI [-11.09, +6.20], p = 0.71 (not significant) Delta B = +22.18 pts, 95% CI [+14.43, +29.82], p = 0.0 (reported significant) At first glance, this looks straightforward: one result is meaningful, one is not. But this interpretation can be wrong if the benchmark is underpowered for the effect sizes we actually care about. This post answers two practical questions: With 216 binary pass/fail tasks, what size improvement can this benchma