Baserow opening 60 to 90+ database connections

Technical Help
1

Are you using our SaaS platform (Baserow.io) or self-hosting Baserow?

Self-hosted

If you are self-hosting, what version of Baserow are you running?

1.35.2 - All in One Image

If you are self-hosting, which installation method do you use to run Baserow?

Docker / Podman - All in One Image with external Postgres

What are the exact steps to reproduce this issue?

Baserow is opening a lot of connections to postgres. When I look into postgres I usually see between 60 to 90+ open connections.

I tried reducing the the workers with setting:
BASEROW_AMOUNT_OF_GUNICORN_WORKERS=4
BASEROW_AMOUNT_OF_WORKERS=2

This does not seem to affect the overall connection amount.Is there any way to limit the amount of connections?

The instance has a very low usage with max 5 concurrent users.

2

Hey @m3tam3re, how many API requests are roughly being made to the instance? If you’re using the all-in-one image, then HTTP is served using the ASGI workers, but they automatically spin up more threads, each with their own database connection, if the number of API request increases.

3

The instance is running a test database only being used through the frontend. There are no external API calls.

Is there a way to limit the max connections?

4

There is, but it will also limit the number of concurrent API requests that it can handle. What happens if you set `BASEROW_ASGI_HTTP_MAX_CONCURRENCY=1`? This will ensure that each gunicorn worker only allows one concurrent HTTP request.

5

Thanks for the info :slight_smile: Could you please share more info on the following points?

  1. Is it possible to set CONN_MAX_AGE from the underlying django framework? / How is this set by default in Baserow?
  2. How many worker processes are created per CPU?
  3. How many DB connections per worker?

Thank you :slight_smile:

6

Is it possible to set CONN_MAX_AGE from the underlying django framework? / How is this set by default in Baserow?

I just checked the source code, but it does not seem to be possible to set the CONN_MAX_AGE. It seems to be 0 by default, so if you’re using the DATABASE_HOST env vars, then that’s the value. However, if you set the DATABASE_URL it’s set at 600 by default.

if "DATABASE_URL" in os.environ:
    DATABASES = {
        "default": dj_database_url.parse(os.getenv("DATABASE_URL"), conn_max_age=600)
    }
else:
    DATABASES = {
        "default": {
            "ENGINE": "django.db.backends.postgresql",
            "NAME": os.getenv("DATABASE_NAME", "baserow"),
            "USER": os.getenv("DATABASE_USER", "baserow"),
            "PASSWORD": os.getenv("DATABASE_PASSWORD", "baserow"),
            "HOST": os.getenv("DATABASE_HOST", "db"),
            "PORT": os.getenv("DATABASE_PORT", "5432"),
        }
    }
    if "DATABASE_OPTIONS" in os.environ:
        DATABASES["default"]["OPTIONS"] = json.loads(
            os.getenv("DATABASE_OPTIONS", "{}")
        )

How many worker processes are created per CPU?

If the BASEROW_AMOUNT_OF_GUNICORN_WORKERS is set, then that’s the number of workers is starts. If the value is empty, then I think it starts num of CPUs + 1 workers.

How many DB connections per worker?

The ASGI workers start a new connection for every async request they’re handling. But if the BASEROW_ASGI_HTTP_MAX_CONCURRENCY is set to 1, then only one connection should be opened per worker.

I hope that helps!

7

Thank you that helps :slight_smile: