How to import Big base from Airtable

"I attempted to import a large Airtable database, but encountered an error message that read:

Something went wrong

The Airtable server could not be reached.

What can I do to resolve this issue?"

The following is a Docker error log.

[BASEROW-WATCHER][2023-02-18 03:24:22] WARNING: Baserow has become unhealthy.
[BACKEND][2023-02-18 03:24:22] [2023-02-18 03:24:15 +0000] [166] [CRITICAL] WORKER TIMEOUT (pid:402)
[BACKEND][2023-02-18 03:24:22] [2023-02-18 03:24:22 +0000] [166] [WARNING] Worker with pid 402 was terminated due to signal 9
[EXPORT_WORKER][2023-02-18 03:24:22] [2023-02-18 03:23:38,130: INFO/MainProcess] missed heartbeat from default-worker@aa931e3aec5d
[CELERY_WORKER][2023-02-18 03:24:22] [2023-02-18 03:23:38,130: INFO/MainProcess] missed heartbeat from export-worker@aa931e3aec5d
[CELERY_WORKER][2023-02-18 03:24:22] [2023-02-18 03:24:22,326: INFO/MainProcess] missed heartbeat from export-worker@aa931e3aec5d
[EXPORT_WORKER][2023-02-18 03:24:22] [2023-02-18 03:24:22,324: INFO/MainProcess] missed heartbeat from default-worker@aa931e3aec5d
[EXPORT_WORKER][2023-02-18 03:24:22] [2023-02-18 03:24:22,522: ERROR/ForkPoolWorker-1] Task baserow.core.jobs.tasks.run_async_job[e5fc2713-bf0b-471e-ab5c-2b3852ebabae] raised unexpected: ConnectionError(ProtocolError(‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’)))
[EXPORT_WORKER][2023-02-18 03:24:22] Traceback (most recent call last):
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 703, in urlopen
[EXPORT_WORKER][2023-02-18 03:24:22] httplib_response = self._make_request(
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 449, in _make_request
[EXPORT_WORKER][2023-02-18 03:24:22] six.raise_from(e, None)
[EXPORT_WORKER][2023-02-18 03:24:22] File “”, line 3, in raise_from
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 444, in _make_request
[EXPORT_WORKER][2023-02-18 03:24:22] httplib_response = conn.getresponse()
[EXPORT_WORKER][2023-02-18 03:24:22] File “/usr/lib/python3.9/http/client.py”, line 1347, in getresponse
[EXPORT_WORKER][2023-02-18 03:24:22] response.begin()
[EXPORT_WORKER][2023-02-18 03:24:22] File “/usr/lib/python3.9/http/client.py”, line 307, in begin
[EXPORT_WORKER][2023-02-18 03:24:22] version, status, reason = self._read_status()
[EXPORT_WORKER][2023-02-18 03:24:22] File “/usr/lib/python3.9/http/client.py”, line 276, in _read_status
[EXPORT_WORKER][2023-02-18 03:24:22] raise RemoteDisconnected(“Remote end closed connection without”
[EXPORT_WORKER][2023-02-18 03:24:22] http.client.RemoteDisconnected: Remote end closed connection without response
[EXPORT_WORKER][2023-02-18 03:24:22]
[EXPORT_WORKER][2023-02-18 03:24:22] During handling of the above exception, another exception occurred:
[EXPORT_WORKER][2023-02-18 03:24:22]
[EXPORT_WORKER][2023-02-18 03:24:22] Traceback (most recent call last):
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/requests/adapters.py”, line 439, in send
[EXPORT_WORKER][2023-02-18 03:24:22] resp = conn.urlopen(
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 787, in urlopen
[EXPORT_WORKER][2023-02-18 03:24:22] retries = retries.increment(
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/util/retry.py”, line 550, in increment
[EXPORT_WORKER][2023-02-18 03:24:22] raise six.reraise(type(error), error, _stacktrace)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/packages/six.py”, line 769, in reraise
[EXPORT_WORKER][2023-02-18 03:24:22] raise value.with_traceback(tb)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 703, in urlopen
[EXPORT_WORKER][2023-02-18 03:24:22] httplib_response = self._make_request(
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 449, in _make_request
[EXPORT_WORKER][2023-02-18 03:24:22] six.raise_from(e, None)
[EXPORT_WORKER][2023-02-18 03:24:22] File “”, line 3, in raise_from
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 444, in _make_request
[EXPORT_WORKER][2023-02-18 03:24:22] httplib_response = conn.getresponse()
[EXPORT_WORKER][2023-02-18 03:24:22] File “/usr/lib/python3.9/http/client.py”, line 1347, in getresponse
[EXPORT_WORKER][2023-02-18 03:24:22] response.begin()
[EXPORT_WORKER][2023-02-18 03:24:22] File “/usr/lib/python3.9/http/client.py”, line 307, in begin
[EXPORT_WORKER][2023-02-18 03:24:22] version, status, reason = self._read_status()
[EXPORT_WORKER][2023-02-18 03:24:22] File “/usr/lib/python3.9/http/client.py”, line 276, in _read_status
[EXPORT_WORKER][2023-02-18 03:24:22] raise RemoteDisconnected(“Remote end closed connection without”
[EXPORT_WORKER][2023-02-18 03:24:22] urllib3.exceptions.ProtocolError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))
[EXPORT_WORKER][2023-02-18 03:24:22]
[EXPORT_WORKER][2023-02-18 03:24:22] During handling of the above exception, another exception occurred:
[EXPORT_WORKER][2023-02-18 03:24:22]
[EXPORT_WORKER][2023-02-18 03:24:22] Traceback (most recent call last):
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/celery/app/trace.py”, line 451, in trace_task
[EXPORT_WORKER][2023-02-18 03:24:22] R = retval = fun(*args, **kwargs)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/celery/app/trace.py”, line 734, in protected_call
[EXPORT_WORKER][2023-02-18 03:24:22] return self.run(*args, **kwargs)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/backend/src/baserow/core/jobs/tasks.py”, line 73, in run_async_job
[EXPORT_WORKER][2023-02-18 03:24:22] raise e
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/backend/src/baserow/core/jobs/tasks.py”, line 36, in run_async_job
[EXPORT_WORKER][2023-02-18 03:24:22] JobHandler().run(job)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/backend/src/baserow/core/jobs/handler.py”, line 62, in run
[EXPORT_WORKER][2023-02-18 03:24:22] return job_type.run(job, progress)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/backend/src/baserow/contrib/database/airtable/job_type.py”, line 118, in run
[EXPORT_WORKER][2023-02-18 03:24:22] databases, id_mapping = AirtableHandler.import_from_airtable_to_group(
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/backend/src/baserow/contrib/database/airtable/handler.py”, line 601, in import_from_airtable_to_group
[EXPORT_WORKER][2023-02-18 03:24:22] baserow_database_export, files_buffer = cls.to_baserow_database_export(
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/backend/src/baserow/contrib/database/airtable/handler.py”, line 529, in to_baserow_database_export
[EXPORT_WORKER][2023-02-18 03:24:22] user_files_zip = cls.download_files_as_zip(
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/backend/src/baserow/contrib/database/airtable/handler.py”, line 330, in download_files_as_zip
[EXPORT_WORKER][2023-02-18 03:24:22] response = requests.get(url, headers=BASE_HEADERS)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/requests/api.py”, line 75, in get
[EXPORT_WORKER][2023-02-18 03:24:22] return request(‘get’, url, params=params, **kwargs)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/requests/api.py”, line 61, in request
[EXPORT_WORKER][2023-02-18 03:24:22] return session.request(method=method, url=url, **kwargs)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/requests/sessions.py”, line 542, in request
[EXPORT_WORKER][2023-02-18 03:24:22] resp = self.send(prep, **send_kwargs)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/requests/sessions.py”, line 655, in send
[EXPORT_WORKER][2023-02-18 03:24:22] r = adapter.send(request, **kwargs)
[EXPORT_WORKER][2023-02-18 03:24:22] File “/baserow/venv/lib/python3.9/site-packages/requests/adapters.py”, line 498, in send
[EXPORT_WORKER][2023-02-18 03:24:22] raise ConnectionError(err, request=request)
[EXPORT_WORKER][2023-02-18 03:24:22] requests.exceptions.ConnectionError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))
[EXPORT_WORKER][2023-02-18 03:24:22] [2023-02-18 03:24:22,838: INFO/ForkPoolWorker-1] Cleaning up 0 old jobs

Regards.
Binh

If its a huge database you might be hitting some type of rate limit or timeout from Airtable? The import is still experimental, so issues like that are to be expected.

Yes, I think so. I believe that Baserow team should create a cron job for this. That way, the import process will not fail even if there are issues with rate limits or timeouts.

Thank you Alex

@nigel is this also linked to the issue we just recently solved on dev and won’t be happening again after the next release?

Nah this looks purely like Airtable timing out the API request to download all the files from the Base, the other issue was unrelated to this.