Building a Payment Processor: The Final One Percent

I had idempotency, a message queue, and retries. I thought I was finally building something production-ready. Then I started thinking about the "in-between" failures. What if the worker starts a charge, the provider succeeds, but the worker crashes before it can update the database? The payment stays stuck in processing forever. The user was charged, but my system thinks it's still "happening." This is an orphaned payment, and it’s a nightmare to debug because, from the outside, it looks like no