The GKE Upgrade That Took Down Our Production Pods for 45 Minutes

I want to start this blog with the story that made me realise I had been running Kubernetes on GKE without actually understanding how GKE runs Kubernetes. This is about a node pool upgrade that should have been routine and wasn't. The Setup We run three GKE Standard clusters on GCP. One for production, one for staging and one for our internal tooling. The production cluster runs about 40 pods across 8 namespaces handling customer-facing workloads. Nothing exotic. Deployments, services, a few sta