War Story: We Survived a 2-Hour Outage with Redis 8.0 Cluster and Sentinel

At 14:17 UTC on October 12, 2024, our Redis 8.0.2 cluster serving 142,000 writes/sec and 89,000 reads/sec across 12 shards dropped 100% of traffic for 11 minutes, then entered a 111-minute partial outage that cost $47,000 in SLA penalties and churned 3 enterprise customers. We didn’t just fix it—we reverse-engineered Redis 8.0’s new Sentinel gossip protocol to find the root cause, and benchmarked every fix to ensure it never happened again. 📡 Hacker News Top Stories Right Now LLMs consistently