Service provider outage

Incident Report for Taghub

Postmortem

Executive Summary

On November 18, TagHub experienced service degradation due to a global outage within our upstream provider’s (Cloudflare) network. While initially observed as a regional issue in Oslo due to our user demographics, the incident was caused by widespread internal service degradation at the provider level. Crucially, our ability to mitigate the issue was severely hampered because the provider's management dashboard and DNS administration were also inaccessible during the incident.

Impact

  • User Experience: Users experienced intermittent 5xx errors, connection timeouts, and high latency.
  • Mitigation Delays: The engineering team was unable to execute standard "break-glass" procedures (bypassing the proxy/load-balancer) because the DNS control plane was unavailable, extending the duration of the impact.

Root Cause

  • Primary: Cloudflare Global Network internal service degradation (Incident 8gmgl950y3h7).
  • Secondary (Mitigation Failure): Simultaneous failure of the Cloudflare Dashboard prevented administrative access to DNS records, making it impossible to reroute traffic away from the failing infrastructure.

Timeline (CET)

  • 12:40: Engineering identifies intermittent connectivity issues.
  • 12:44: Issues initially correlated with the Oslo datacenter (primary traffic source). Attempting to isolate.
  • 13:00: Investigation reveals global provider instability. Team attempts to bypass Cloudflare proxy to restore direct connectivity.
  • 13:10: Blocker Identified: Access to Cloudflare DNS administration is denied due to the provider's dashboard outage. Unable to update A-records to bypass the load balancer.
  • 14:13: Partial mitigation implemented where possible; monitoring continues while awaiting provider recovery. Production environment operational.
  • 20:41: Cloudflare confirms full resolution of both the network edge and the management dashboard. Incident closed.

Lessons Learned & Next Steps

The incident highlighted a critical dependency: relying on the same provider for both Edge Traffic (Load Balancing/CDN) and DNS Management creates a single point of failure during control-plane outages.

  • Action Item: We’ll continue improving our system according to the lessons learnt during this incident. All action items and goals are documented in our internal QA system.
Posted Nov 19, 2025 - 10:14 CET

Resolved

Cloudflare has resolved their issues, and all systems should be operational once again. Post-mortem report is estimated released by end of Thursday. This incident has been registered in our quality system with the id 844.
Posted Nov 18, 2025 - 20:41 CET

Update

Production system bypasses affected service providers. We'll keep monitoring before re-enabling indexing.
Certain images might load slowly until the service provider has restored it's image scaling capabilities.
Posted Nov 18, 2025 - 14:13 CET

Update

We're seeing improving successful requests passing through. Still continuing our mitigation work.
Posted Nov 18, 2025 - 13:13 CET

Update

Impact has increased, reducing availability accordingly. We're still working on mitigation.
Posted Nov 18, 2025 - 12:58 CET

Monitoring

Our service provider has provided the following status update:
https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
We're monitoring for anything we can do to improve our stability until the service provider fixes the issue on their end.
Posted Nov 18, 2025 - 12:51 CET

Update

Issue is localized in the datacenter in Oslo, seeing sporadic failures for requests. Working on rerouting traffic to other datacenters.
Posted Nov 18, 2025 - 12:44 CET

Identified

We're experiencing intermittent issues with a service providers datacenter. Mitigation work is in progress.
Posted Nov 18, 2025 - 12:40 CET
This incident affected: Production (Taghub), Staging (Taghub), and Demo (Taghub).