How I Stopped Retry Storms from Destroying My Scraping Budget

Last month I watched a scraping job quietly burn $240 overnight. The target site started returning 403s around 2 AM. The agent retried. Got another 403. Retried again. By morning it had made 600+ identical requests to a URL that was never going to respond — and I paid for every single one of them in compute units. The frustrating part: I had already set maxRetries: 3 . It didn't matter. The URL kept getting requeued across multiple runs, and each new run reset the counter. Three retries × 200 ac