We’re experiencing degraded performance on Baserow.io due to a high volume of API requests. Our team is investigating the issue.
We apologize for any disruption and will provide updates as we resolve this situation.
We’ve identified a significant number of requests to https://baserow.io/api/user/token-auth/. We’ve temporarily disabled this endpoint so that Baserow will work as expected. A side effect is that it’s currently impossible to login.
Thanks. Why can’t you just rate limit that endpoint base on IP or something?
It seemed that a request was coming in every millisecond or so. We can’t rate limit that on application level because the load is too high. We’re looking into solutions where we can do that on webserver level, but this is proving to be difficult with the Caddy Ingress Controller, which we need for automatic SSL certificates in the application builder.
Is the traffic originated from a single source (like the downtime that occurred recently due to a webhook infinite calls) or it’s coming from many multiple sources ( some kind of DDOS) ?
Checkout if your cloud provider (I think you are using Digital Ocean) provides Web Application Firewall. It’s useful for such scenarios and can be activated with a click.
@bram Do we have any updates on the expected resolution? I need to decide whether to wait or explore alternative options. Thanks for keeping us informed!
@Ivan Baserow is working as expected at the moment, you just can’t log in. If you’re already logged in, then it will work as expected. Are you not logged in at the moment?
@bram I need to use the API.
@Ivan Do you have a database token, or a valid refresh token? Because those still work. It’s just not possible to obtain a new refresh token using the https://baserow.io/api/user/token-auth/ endpoint.
Apart from that, I think we should have the endpoint back online within an hour or so.
@bram It’s not that it seems to have happened. All my automations in N8N stopped working.
Ah yeah, n8n indeed depends on generating a new refresh token using the mentioned endpoint. We’re working very hard on a fix. If we enable the endpoint right now, we’re receiving so many requests that the servers go down. Blocking this IP on the level where it’s needed requires some infra changes. Once that’s implemented, we’ll make the endpoint available again.
@bram Is there a new estimate? Unfortunately, this downtime is compromising my operation and causing dissatisfaction among my customers.
Hey @Ivan, logging in is already possible. We had to make some DNS changes, which might not have propagated for you yet. Would you mind trying again?