When Retries Turn Hostile — How Control Logic Kills Production Systems

"Your retries are killing us." A service team received this message from a downstream dependency during an outage. The upstream API was timing out, so naturally, the client retried. 3 times, 5 times, 10 times. The client thought it was doing the right thing. From the dependency's perspective, they were at half capacity due to the outage — and receiving several times the normal traffic. Retries were making the outage worse and preventing recovery. This isn't a fable. In August 2012, Knight Capita