🚨 Incident report: Baserow API disruption (2am–7am UTC)

Announcements
,
1

Between 2am and 7am UTC, Baserow.io experienced a disruption caused by a loop involving user-created webhooks. The issue occurred when a webhook triggered an automation, which updated a row. That row update, in turn, re-triggered the webhook, creating an infinite loop that overwhelmed our system—effectively acting as a DDoS attack. As the task queue system filled with these webhook requests, some systems ran out of memory. While the systems automatically restarted, they quickly became overwhelmed again.

Actions taken:

  1. Manual intervention: We temporarily disabled webhooks to restore normal operations.
  2. Preventive fix: A code update to prevent the task queue from being overwhelmed by webhooks will be deployed today on baserow.io. We will closely monitor the situation and intervene manually if needed.
  3. Improved alerts: We’ve set up phone call notifications to ensure the team is immediately alerted should a similar issue occur during the night.

We are committed to maintaining high uptime and are proud to have achieved nearly 100% uptime over the past 90 days. We sincerely apologize for this inconvenience and appreciate your understanding as we work to prevent such incidents in the future. :raised_hands:

2

Thanks @olgatrykush
So was this webhook created by another user using baserow.io services?
To prevent this from happening, could I host a Baserow instance from my own cloud server?

3

Hey @dog87, you’re correct. While we’re taking steps to prevent similar situations in future, you can self-host Baserow to ensure you’re not affected by any potential downtime. This gives you complete control over your infrastructure.

4

Hi @dog87, the webhook was indeed created by another user using baserow.io. In fact, 8 webhooks were created, all ending up in a loop, quickly causing an exponential effect. There is documentation how to self-host Baserow in various ways here Install with Docker.