Connecting to external database

I’m using docker-compose to self-host Baserow. It works fine with its own database but I’d like to use an external one (specifically one I’m using with Laravel - also running on Docker).

Whenever I try this I get the following error:

Error: Failed to connect to the postgresql database at 127.0.0.1
Please see the error below for more details:
could not connect to server: Connection refused
 	Is the server running on host "127.0.0.1" and accepting
 	TCP/IP connections on port 5432?

My .env file contains the following variables:

DATABASE_USER=sail
DATABASE_NAME=filament
DATABASE_HOST=127.0.0.1
DATABASE_PORT=5432
DATABASE_PASSWORD=password

I am able to open the database, using the same credentials, using TablePlus.

My docker-compose.yml file (with database section commented out) is as follows:

version: "3.4"
########################################################################################
#
# READ ME FIRST!
#
# Use the much simpler `docker-compose.all-in-one.yml` instead of this file
# unless you are an advanced user!
#
# This compose file runs every service separately using a Caddy reverse proxy to route
# requests to the backend or web-frontend, handle websockets and to serve uploaded files
# . Even if you have your own http proxy we recommend to simply forward requests to it
# as it is already properly configured for Baserow.
#
# If you wish to continue with this more advanced compose file, it is recommended that
# you set environment variables using the .env.example file by:
# 1. `cp .env.example .env`
# 2. Edit .env and fill in the first 3 variables.
# 3. Set further environment variables as you wish.
#
# More documentation can be found in:
# https://baserow.io/docs/installation%2Finstall-with-docker-compose
#
########################################################################################

# See https://baserow.io/docs/installation%2Fconfiguration for more details on these
# backend environment variables, their defaults if left blank etc.
x-backend-variables: &backend-variables
  # Most users should only need to set these first four variables.
  SECRET_KEY: ${SECRET_KEY:?}
  BASEROW_JWT_SIGNING_KEY: ${BASEROW_JWT_SIGNING_KEY:-}
  DATABASE_PASSWORD: ${DATABASE_PASSWORD:?}
  REDIS_PASSWORD: ${REDIS_PASSWORD:?}
  # If you manually change this line make sure you also change the duplicate line in
  # the web-frontend service.
  BASEROW_PUBLIC_URL: ${BASEROW_PUBLIC_URL-http://localhost}

  # Set these if you want to use an external postgres instead of the db service below.
  DATABASE_USER: ${DATABASE_USER:-baserow}
  DATABASE_NAME: ${DATABASE_NAME:-baserow}
  DATABASE_HOST:
  DATABASE_PORT:
  DATABASE_URL:

  # Set these if you want to use an external redis instead of the redis service below.
  REDIS_HOST:
  REDIS_PORT:
  REDIS_PROTOCOL:
  REDIS_URL:
  REDIS_USER:

  # Set these to enable Baserow to send emails.
  EMAIL_SMTP:
  EMAIL_SMTP_HOST:
  EMAIL_SMTP_PORT:
  EMAIL_SMTP_USE_TLS:
  EMAIL_SMTP_USER:
  EMAIL_SMTP_PASSWORD:
  FROM_EMAIL:

  # Set these to use AWS S3 bucket to store user files.
  AWS_ACCESS_KEY_ID:
  AWS_SECRET_ACCESS_KEY:
  AWS_STORAGE_BUCKET_NAME:
  AWS_S3_REGION_NAME:
  AWS_S3_ENDPOINT_URL:
  AWS_S3_CUSTOM_DOMAIN:

  # Misc settings see https://baserow.io/docs/installation%2Fconfiguration for info
  BASEROW_AMOUNT_OF_WORKERS:
  BASEROW_ROW_PAGE_SIZE_LIMIT:
  BATCH_ROWS_SIZE_LIMIT:
  INITIAL_TABLE_DATA_LIMIT:
  BASEROW_FILE_UPLOAD_SIZE_LIMIT_MB:

  BASEROW_EXTRA_ALLOWED_HOSTS:
  ADDITIONAL_APPS:
  BASEROW_PLUGIN_GIT_REPOS:
  BASEROW_PLUGIN_URLS:

  BASEROW_ENABLE_SECURE_PROXY_SSL_HEADER:
  MIGRATE_ON_STARTUP: ${MIGRATE_ON_STARTUP:-true}
  SYNC_TEMPLATES_ON_STARTUP: ${SYNC_TEMPLATES_ON_STARTUP:-true}
  DONT_UPDATE_FORMULAS_AFTER_MIGRATION:
  BASEROW_TRIGGER_SYNC_TEMPLATES_AFTER_MIGRATION:
  BASEROW_SYNC_TEMPLATES_TIME_LIMIT:

  BASEROW_BACKEND_DEBUG:
  BASEROW_BACKEND_LOG_LEVEL:
  FEATURE_FLAGS:

  PRIVATE_BACKEND_URL: http://backend:8000
  PUBLIC_BACKEND_URL:
  PUBLIC_WEB_FRONTEND_URL:
  MEDIA_URL:
  MEDIA_ROOT:

  BASEROW_AIRTABLE_IMPORT_SOFT_TIME_LIMIT:
  HOURS_UNTIL_TRASH_PERMANENTLY_DELETED:
  OLD_ACTION_CLEANUP_INTERVAL_MINUTES:
  MINUTES_UNTIL_ACTION_CLEANED_UP:
  BASEROW_GROUP_STORAGE_USAGE_ENABLED:
  BASEROW_GROUP_STORAGE_USAGE_QUEUE:
  BASEROW_COUNT_ROWS_ENABLED:
  DISABLE_ANONYMOUS_PUBLIC_VIEW_WS_CONNECTIONS:
  BASEROW_WAIT_INSTEAD_OF_409_CONFLICT_ERROR:
  BASEROW_FULL_HEALTHCHECKS:
  BASEROW_DISABLE_MODEL_CACHE:
  BASEROW_PLUGIN_DIR:
  BASEROW_JOB_EXPIRATION_TIME_LIMIT:
  BASEROW_JOB_CLEANUP_INTERVAL_MINUTES:
  BASEROW_MAX_ROW_REPORT_ERROR_COUNT:
  BASEROW_JOB_SOFT_TIME_LIMIT:
  BASEROW_FRONTEND_JOBS_POLLING_TIMEOUT_MS:
  BASEROW_INITIAL_CREATE_SYNC_TABLE_DATA_LIMIT:
  BASEROW_MAX_SNAPSHOTS_PER_GROUP:
  BASEROW_SNAPSHOT_EXPIRATION_TIME_DAYS:
  BASEROW_WEBHOOKS_ALLOW_PRIVATE_ADDRESS:
  BASEROW_WEBHOOKS_IP_BLACKLIST:
  BASEROW_WEBHOOKS_IP_WHITELIST:
  BASEROW_WEBHOOKS_URL_REGEX_BLACKLIST:
  BASEROW_WEBHOOKS_URL_CHECK_TIMEOUT_SECS:
  BASEROW_WEBHOOKS_MAX_CONSECUTIVE_TRIGGER_FAILURES:
  BASEROW_WEBHOOKS_MAX_RETRIES_PER_CALL:
  BASEROW_WEBHOOKS_MAX_PER_TABLE:
  BASEROW_WEBHOOKS_MAX_CALL_LOG_ENTRIES:
  BASEROW_WEBHOOKS_REQUEST_TIMEOUT_SECONDS:
  BASEROW_PERMISSION_MANAGERS:

services:
  # A caddy reverse proxy sitting in-front of all the services. Responsible for routing
  # requests to either the backend or web-frontend and also serving user uploaded files
  # from the media volume.
  caddy:
    image: caddy:2
    restart: unless-stopped
    environment:
      # Controls what port the Caddy server binds to inside its container.
      BASEROW_CADDY_ADDRESSES: ${BASEROW_CADDY_ADDRESSES:-:80}
      PRIVATE_WEB_FRONTEND_URL: ${PRIVATE_WEB_FRONTEND_URL:-http://web-frontend:3000}
      PRIVATE_BACKEND_URL: ${PRIVATE_BACKEND_URL:-http://backend:8000}
    ports:
      - "${HOST_PUBLISH_IP:-0.0.0.0}:${WEB_FRONTEND_PORT:-80}:80"
      - "${HOST_PUBLISH_IP:-0.0.0.0}:${WEB_FRONTEND_SSL_PORT:-443}:443"
    volumes:
      - $PWD/Caddyfile:/etc/caddy/Caddyfile
      - media:/baserow/media
      - caddy_config:/config
      - caddy_data:/data
    healthcheck:
      test: [ "CMD", "wget", "--spider", "http://localhost/caddy-health-check" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      local:

  backend:
    image: baserow/backend:1.14.0
    restart: unless-stopped
    environment:
      <<: *backend-variables
    depends_on:
      # - db
      - redis
    volumes:
      - media:/baserow/media
    networks:
      local:

  web-frontend:
    image: baserow/web-frontend:1.14.0
    restart: unless-stopped
    environment:
      BASEROW_PUBLIC_URL: ${BASEROW_PUBLIC_URL-http://localhost}
      PRIVATE_BACKEND_URL: ${PRIVATE_BACKEND_URL:-http://backend:8000}
      PUBLIC_BACKEND_URL:
      PUBLIC_WEB_FRONTEND_URL:
      BASEROW_DISABLE_PUBLIC_URL_CHECK:
      INITIAL_TABLE_DATA_LIMIT:
      DOWNLOAD_FILE_VIA_XHR:
      BASEROW_DISABLE_GOOGLE_DOCS_FILE_PREVIEW:
      HOURS_UNTIL_TRASH_PERMANENTLY_DELETED:
      DISABLE_ANONYMOUS_PUBLIC_VIEW_WS_CONNECTIONS:
      FEATURE_FLAGS:
      ADDITIONAL_MODULES:
      BASEROW_MAX_IMPORT_FILE_SIZE_MB:
      BASEROW_MAX_SNAPSHOTS_PER_GROUP:
    depends_on:
      - backend
    networks:
      local:

  celery:
    image: baserow/backend:1.14.0
    restart: unless-stopped
    environment:
      <<: *backend-variables
    command: celery-worker
    # The backend image's baked in healthcheck defaults to the django healthcheck
    # override it to the celery one here.
    healthcheck:
      test: [ "CMD-SHELL", "/baserow/backend/docker/docker-entrypoint.sh celery-worker-healthcheck" ]
    depends_on:
      - backend
    volumes:
      - media:/baserow/media
    networks:
      local:

  celery-export-worker:
    image: baserow/backend:1.14.0
    restart: unless-stopped
    command: celery-exportworker
    environment:
      <<: *backend-variables
    # The backend image's baked in healthcheck defaults to the django healthcheck
    # override it to the celery one here.
    healthcheck:
      test: [ "CMD-SHELL", "/baserow/backend/docker/docker-entrypoint.sh celery-exportworker-healthcheck" ]
    depends_on:
      - backend
    volumes:
      - media:/baserow/media
    networks:
      local:

  celery-beat-worker:
    image: baserow/backend:1.14.0
    restart: unless-stopped
    command: celery-beat
    environment:
      <<: *backend-variables
    # See https://github.com/sibson/redbeat/issues/129#issuecomment-1057478237
    stop_signal: SIGQUIT
    # We don't yet have a healthcheck for the beat worker, just assume it is healthy.
    healthcheck:
      test: [ "CMD-SHELL", "exit 0" ]
    depends_on:
      - backend
    volumes:
      - media:/baserow/media
    networks:
      local:

  # db:
  #   # ports:
  #   #   - "${HOST_PUBLISH_IP:-127.0.0.1}:${POSTGRES_PORT:-5432}:5432"
  #   image: postgres:11
  #   restart: unless-stopped
  #   environment:
  #     - POSTGRES_USER=${DATABASE_USER:-baserow}
  #     - POSTGRES_PASSWORD=${DATABASE_PASSWORD:?}
  #     - POSTGRES_DB=${DATABASE_NAME:-baserow}
  #   healthcheck:
  #     test: [ "CMD-SHELL", "su postgres -c \"pg_isready -U ${DATABASE_USER:-baserow}\"" ]
  #     interval: 10s
  #     timeout: 5s
  #     retries: 5
  #   networks:
  #     local:
  #   volumes:
  #     - pgdata:/var/lib/postgresql/data

  redis:
    image: redis:6
    command: redis-server --requirepass ${REDIS_PASSWORD:?}
    healthcheck:
      test: [ "CMD", "redis-cli", "ping" ]
    networks:
      local:

  # By default, the media volume will be owned by root on startup. Ensure it is owned by
  # the same user that django is running as, so it can write user files.
  volume-permissions-fixer:
    image: bash:4.4
    command: chown 9999:9999 -R /baserow/media
    volumes:
      - media:/baserow/media
    networks:
      local:

volumes:
  pgdata:
  media:
  caddy_data:
  caddy_config:

networks:
  local:
    driver: bridge

Any suggestions?

Hey @Android63

There are some important things in order for this to work

  • the database server needs to be set to accept connections over the network (if it is a regular postgres docker container this should work out of the box)
  • the postgres container needs to be connected to the same docker network than your baserow container

Your .env life refers to 127.0.0.1 as the DATABASE_HOST. This should be the hostname or the ip address of the docker container running postgres.

Please also make sure the database container has access to the same docker network as the baserow container.

You can check which containers are connected to your docker network with:

docker network inspect <network_name>

Hope this helps

Thanks @m3tam3re.

I’ve tried a few things following your advice but nothing so far works.

  • the database server needs to be set to accept connections over the network (if it is a regular postgres docker container this should work out of the box)

How can I know this? Does the fact that I’m able to open the database in TablePlus confirm this? I’m using Laravel Sail, so I imagine it would be able to accept connections for code to function.

  • the postgres container needs to be connected to the same docker network than your baserow container

How can I check this and make it so?

Your .env life refers to 127.0.0.1 as the DATABASE_HOST . This should be the hostname or the ip address of the docker container running postgres.

I’m using the same hostname that I use without any problem in TablePlus (127.0.0.1). If it’s wrong, where should I get the correct hostname or ip address?

Some more information:

➜  baserow git:(master) ✗ docker network list
NETWORK ID     NAME                DRIVER    SCOPE
b8cfc41ce56e   baserow_local       bridge    local
3f4d8099f842   bridge              bridge    local
81f0a3660622   certificates_sail   bridge    local
9ae6ee2a8212   devkinsta_network   bridge    local
9b830c40093b   filament_sail       bridge    local
6b449e76ac8c   host                host      local
39346725e38d   none                null      local
➜  baserow git:(master) ✗ docker network inspect baserow_local
[
    {
        "Name": "baserow_local",
        "Id": "b8cfc41ce56e19c8b05f0bb5730c3b0ccd7e67770589ef381cdbb3c19fba00c8",
        "Created": "2023-01-24T15:58:04.210628185Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.96.0/20",
                    "Gateway": "192.168.96.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "39bbe22b5ff0b4658da7b476a6c939d2cd160c76f3578d829bdb3cbbdd21404e": {
                "Name": "baserow-caddy-1",
                "EndpointID": "2013df5d29328fbf74113a26bc41c760917d6ce697b12032c95f137a8d887b30",
                "MacAddress": "02:42:c0:a8:60:04",
                "IPv4Address": "192.168.96.4/20",
                "IPv6Address": ""
            },
            "6684d6936a06812fed8b438316ae9657592863836f5523d13939335fbd4a660f": {
                "Name": "baserow-redis-1",
                "EndpointID": "733dfb8b78300ca3f3f2a020c0a531f5a2656736eddcca5df1c9b2a71e494cae",
                "MacAddress": "02:42:c0:a8:60:03",
                "IPv4Address": "192.168.96.3/20",
                "IPv6Address": ""
            },
            "9a0e5bc9e50831180e9bee12ee38e787915e9abb7f043bc4c7b4427051b07ea5": {
                "Name": "baserow-celery-beat-worker-1",
                "EndpointID": "4d373df2f10e731dabaee22e3cdd8c6703b8e4b857b6c3e60cf68101060ea35e",
                "MacAddress": "02:42:c0:a8:60:02",
                "IPv4Address": "192.168.96.2/20",
                "IPv6Address": ""
            },
            "a81e9ae66ab025ecbb5a803dc89be08c0474b46c9a575f5032cc8296852cc243": {
                "Name": "baserow-celery-export-worker-1",
                "EndpointID": "c093a803df6907a6236319b81f1f8fc6f25ffa0fb8c510493cbbcba88ff6f60a",
                "MacAddress": "02:42:c0:a8:60:07",
                "IPv4Address": "192.168.96.7/20",
                "IPv6Address": ""
            },
            "c99eabad3e50e317a6d73c2d9c7b22d9a4cfabd6b20de4416eeaba1dc5d30b7c": {
                "Name": "baserow-backend-1",
                "EndpointID": "648bfa05a34c03ecdcdc80f3dc9183e8e844e213bcbbaa3d4d32bad2eab3f166",
                "MacAddress": "02:42:c0:a8:60:05",
                "IPv4Address": "192.168.96.5/20",
                "IPv6Address": ""
            },
            "fc610be0a549a86ae4040c14cb37d83cd1923a90fd4a135ac6c2656b86afdca5": {
                "Name": "baserow-celery-1",
                "EndpointID": "c9eb3ea2f187b520e432fd46246c491c75d6be207b578a6920b9f9b37e25b596",
                "MacAddress": "02:42:c0:a8:60:06",
                "IPv4Address": "192.168.96.6/20",
                "IPv6Address": ""
            },
            "fcb876538bf416ad36aa8b6409ee7c88bd086da5491d6b58426698064ed3c294": {
                "Name": "baserow-web-frontend-1",
                "EndpointID": "07c635373b4a6c0b107d303d8fac4e85a05932df16d37158a126f6e08f824cd0",
                "MacAddress": "02:42:c0:a8:60:08",
                "IPv4Address": "192.168.96.8/20",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "local",
            "com.docker.compose.project": "baserow",
            "com.docker.compose.version": "2.13.0"
        }
    }
]
➜  baserow git:(master) ✗ docker network inspect filament_sail
[
    {
        "Name": "filament_sail",
        "Id": "9b830c40093bcfdf4c8a58a01e96b625ee17550b75586e8542d2631e7679e7e8",
        "Created": "2023-01-20T16:44:16.819049984Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.160.0/20",
                    "Gateway": "192.168.160.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "b38b959cce1aa70b8b125d74d5278be198d43e31746fdb9816d0481353b722b9": {
                "Name": "filament-laravel.test-1",
                "EndpointID": "16c9bdc6abec55d48ce15c88a9f2536a58c8af19ba950e9d50e1f154a47a9581",
                "MacAddress": "02:42:c0:a8:a0:03",
                "IPv4Address": "192.168.160.3/20",
                "IPv6Address": ""
            },
            "d37c5c123e07c3414ecd6a8a99f4a482979e78b48cc9e30d6053bf1320fcc046": {
                "Name": "filament-pgsql-1",
                "EndpointID": "69cf1cbf19bcf92314e2e5b791cfb04622c8f46324d49de1b69387d935d272a0",
                "MacAddress": "02:42:c0:a8:a0:02",
                "IPv4Address": "192.168.160.2/20",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "sail",
            "com.docker.compose.project": "filament",
            "com.docker.compose.version": "2.13.0"
        }
    }
]
➜  baserow git:(master) ✗ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "3f4d8099f842f24cd21d4d484cc679ffcfdf52546baf223f45e41a9cf8d2e266",
        "Created": "2023-01-17T16:34:51.207719996Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

@Android63 One key thing to understand is that being inside docker container is like being inside another computer or virtual machine. When you tell Baserow to connect to the database at 127.0.0.1 on port 5432, Baserow is running inside of a container, and so looks for a database listening on that part inside of the container only. There is no such database and so you see the error.

So as a result, if you want to have your Baserow container connect to and use your filament-pgsql-1 postgresql, they need to:

  1. Be on a shared docker network
  2. Then you can connect to the filament psql by using it’s container name as the hostname

You can do this by in the Baserow compose file adding to the network section at the bottom:

networks:
  filament_sail:
    external: true
  local:
    driver: bridge

and then changing the backend, celery, celery-export-worker and celery-beat-worker services in the Baserow docker-compose to put them on the same network as the database like so:

    networks:
      local:
      filament_sail:

Finally then you should be able to put DATABASE_HOST=filament-pgsql-1 or DATABASE_HOST=the service name in the docker compose file of the pgsql service in the Baserow .env, down all of the Baserow containers to delete them all and then up them all again now running on the shared network.

I would also suggest checking out the docker and docker-compose documentation on networking: Networking in Compose | Docker Documentation

Thank you for explaining that @nigel. It makes perfect sense to consider each docker container as a separate virtual machine. The settings you suggested worked straight away and now I’m using the external database.