When Your Training Loss Is Lying to You Building a Tenacious-Specific Sales Outreach Benchmark Eyoel Nebiyu · May 2026

This post documents a real negative result: my trained model worked… but a well-written prompt worked better. TL;DR I built a 266-task evaluation benchmark for B2B sales-outreach agents — something existing benchmarks don’t measure well. Then I trained a small preference-learning judge model using SimPO. What happened surprised me: Training accuracy → 100% Held-out accuracy → 25% Classic overfitting. But the real lesson wasn’t about the model. It was about the data. After fixing dataset construc