Into every cloud's life, sometimes a little rain will fall. Or, maybe it was water from a water hose? Either way, Google Cloud saw its Paris datacenter, a.k.a europe-west9, come to an abrupt stop early on the morning of Wednesday, April 26, thanks to "water intrusion." Yes, this was an entire cloud provider region, not "just" an availability zone.
First water, then fire!
According to a DataCenterDynamics report, the proximate cause was Global Switch's Parisian data center catching fire. It was a case of one thing after another. First, a cooling system water pump failed. This caused water to leak into the battery room, which then sparked a fire. However, the fire wasn't so much of a problem as the water it took the Paris fire brigade to put it out.
We don't know how much water flooded the servers, but it was way, way too much water. After reporting the other day its first profits, Google reported an emergency shutdown of some hardware in that zone. Worse still, "There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted."
What services were impacted? Pretty much all of them. This includes: Google Cloud SQL, Google Cloud Bigtable, Persistent Disk, Google Compute Engine and Google Kubernetes Engine. They, and everything else, are still DOA.
Looking ahead, there's no good news to be seen. The Google Cloud status page states: “There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted."
Maybe it's time to consider cloud outage insurance. A mere service-level agreement won't be enough to cover the damage from this particular failure.
Have an outage to report? Send a note to the editors here.