The Workflow Problem That Made Me Stop Trusting Glue Code

A few years ago I watched a webhook handler charge a customer's card twice in the same minute. The success path committed. The retry path committed. They'd been written six months apart by different engineers, neither aware the other existed. The fix took an afternoon. The conversation about why the system allowed it took two weeks. The fix was an idempotency key. The two weeks were about everything else. I've been writing this kind of code for 20 years, and that incident wasn't the first or the