Postmortem: HashiCorp Vault 1.16 Secret Rotation Failure Caused 4-Hour Outage for Payment Services

At 09:15 UTC on August 14, 2024, 47 payment service instances across three AWS regions simultaneously began returning 401 Unauthorized errors for every transaction initiation request. The root cause? A race condition in HashiCorp Vault 1.16’s secret rotation logic that corrupted KV v2 secret metadata during high-frequency rotations, leading to a 4-hour total outage that cost the affected payment processor $1.2M in lost transaction volume. This postmortem breaks down the exact failure mode, bench