Executive Summary
On November 18, TagHub experienced service degradation due to a global outage within our upstream provider’s (Cloudflare) network. While initially observed as a regional issue in Oslo due to our user demographics, the incident was caused by widespread internal service degradation at the provider level. Crucially, our ability to mitigate the issue was severely hampered because the provider's management dashboard and DNS administration were also inaccessible during the incident.
Impact
-
User Experience: Users experienced intermittent 5xx errors, connection timeouts, and high latency.
-
Mitigation Delays: The engineering team was unable to execute standard "break-glass" procedures (bypassing the proxy/load-balancer) because the DNS control plane was unavailable, extending the duration of the impact.
Root Cause
-
Primary: Cloudflare Global Network internal service degradation (Incident 8gmgl950y3h7).
-
Secondary (Mitigation Failure): Simultaneous failure of the Cloudflare Dashboard prevented administrative access to DNS records, making it impossible to reroute traffic away from the failing infrastructure.
Timeline (CET)
-
12:40: Engineering identifies intermittent connectivity issues.
-
12:44: Issues initially correlated with the Oslo datacenter (primary traffic source). Attempting to isolate.
-
13:00: Investigation reveals global provider instability. Team attempts to bypass Cloudflare proxy to restore direct connectivity.
-
13:10: Blocker Identified: Access to Cloudflare DNS administration is denied due to the provider's dashboard outage. Unable to update A-records to bypass the load balancer.
-
14:13: Partial mitigation implemented where possible; monitoring continues while awaiting provider recovery. Production environment operational.
-
20:41: Cloudflare confirms full resolution of both the network edge and the management dashboard. Incident closed.
Lessons Learned & Next Steps
The incident highlighted a critical dependency: relying on the same provider for both Edge Traffic (Load Balancing/CDN) and DNS Management creates a single point of failure during control-plane outages.
-
Action Item: We’ll continue improving our system according to the lessons learnt during this incident. All action items and goals are documented in our internal QA system.