File import failure on hosted baserow using Openstack/Swift in S3 mode

Technical Help
, ,
1

I am using a hosted instance of baserow (based on baserow docker images) that is configured to use Openstack/Swift as S3 storage.

Started with 1.19, I had no problem in baserow usage.

Since upgrade in 1.20.x and now 1.21.x, I can no longer import any file.

  • The user interface respond with “File import failed for unknown reason”.
  • The log of baserow-wsgi shows a boto3 error:
ERROR 2023-11-05 13:37:32,468 django.request.log_response:224- Internal Server Error: /api/database/tables/database/193/async/ 
Traceback (most recent call last):
  File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/baserow/backend/src/baserow/api/decorators.py", line 107, in func_wrapper
    return func(*args, **kwargs)
  File "/baserow/backend/src/baserow/api/decorators.py", line 232, in func_wrapper
    return func(*args, **kwargs)
  File "/baserow/backend/src/baserow/contrib/database/api/tables/views.py", line 303, in post
    file_import_job = JobHandler().create_and_start_job(
  File "/baserow/backend/src/baserow/core/jobs/handler.py", line 169, in create_and_start_job
    job_type.after_job_creation(job, kwargs)
  File "/baserow/backend/src/baserow/contrib/database/file_import/job_types.py", line 103, in after_job_creation
    job.data_file.save(None, data_file)
  File "/baserow/venv/lib/python3.9/site-packages/django/db/models/fields/files.py", line 89, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/baserow/venv/lib/python3.9/site-packages/django/core/files/storage.py", line 53, in save
    name = self.get_available_name(name, max_length=max_length)
  File "/baserow/venv/lib/python3.9/site-packages/storages/backends/s3boto3.py", line 599, in get_available_name
    return super().get_available_name(name, max_length)
  File "/baserow/venv/lib/python3.9/site-packages/django/core/files/storage.py", line 91, in get_available_name
    while self.exists(name) or (max_length and len(name) > max_length):
  File "/baserow/venv/lib/python3.9/site-packages/storages/backends/s3boto3.py", line 463, in exists
    self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
  File "/baserow/venv/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

I am looking for ways to trace boto3 calls to understand what is happening.

The S3 access works for storing images, download tables and so on…

I have already created an issue in the baserow gitlab repo : File import failure when configured to use S3 (#1865) · Issues · Baserow / baserow · GitLab

Thanks in advance for any help…

2

Looks like the problem is worst than what I imagine. I can’t even create a new table.

If I choose “Add a table”, then “start with an empty table”, give it a name and click on “Add the table”, I reach again the “operation did not complete for unknown reasons”.

Looking at the log, I see that I find the same error message on boto3 call with headObject returning error 400.

I can’t figure the relation between my request to create a new empty table and a boto3 request at that time !

I did the upgrade the same way I do with many other products, just replacing the version numbers of the images and restarting containers. But the upgrade, if any happen, has probably failed somewhere.

Any chance to unlock the situation without loosing my data ?

Thanks

3

Hi @philippelt, I’m sorry to hear that you’re running into some problems here. The only thing we changed on our side regarding the S3 configuration was setting the AWS_S3_FILE_OVERWRITE to False. I don’t expect that to be causing your problems.

When looking into the error message, I see some people saying that providing a region solved their problem (amazon web services - A client error (400) occurred when calling the HeadObject operation: Bad Request Completed 1 part(s) with ... file(s) remaining - Stack Overflow). Have you set the AWS_S3_REGION_NAME environment variable? If so, can you double-check if it’s configured the right way. If not, can you set it based on the settings provided by Openstack/Swift?

4

Hello @bram ,

many thanks for looking at the problem.

As you suggested, I added the region name parameter but this did not change the behavior, I am still on the same error message in baserow-wsgi.

I didn’t investigate this parameter before because I am using aws cli tools without specifying the region name and it works flawlessly.

Looking at the error message, it seems that the failure occurs on the get_available_name function which purpose seems to find a free file name to store job data_file.

I suppose that the headobject call is a test to check whether the file exist or not. It should return a 404 error if the filename is free.

There seems to be an anomaly in the headobject (thus the 400 error instead of the 404) call and I would try to find a way to trace parameters of the call to understand what may happen. I understand that this a part of the Django framework and you may not have a debug method.

I have seen on boto3 issues that this error (precisely on headObject) had happen in the past due to an outdated version of the boto3 library. Do you think there would be a way to update this library, may be by building an updated container image adding a new release of boto3 ?

I have seen in the baserow bucket a lot of jobs json files. This is a big concern for me : it has been working for several weeks and probably started to fail after my upgrade to 1.20.x. I just restarted baserow with new container images versions. Did I forgot an upgrade step somewhere ? Is there a manual upgrade operation to perform ?

I would try to unlock my users, do you think I could try to revert the installation to use of a regular file volume and disable S3 entirely ?

Thanks again for your support

Edit: Just checked the date/time of last import job json file in S3 and the date is precisely the date I upgraded to 1.20.2. This confirm the failure started with the update. Job files are numbered, any risk that this sequence has been reseted and that create collisions ?

5

I’m doubting a bit whether the upgrade to version 1.21 of Baserow is causing these problems. We last updated the boto3 library to version 1.26.103 in March this year. That was around the period when Baserow was around version 1.15.

I’d love to update the boto3 library, but we’re not planning on doing a release anytime soon. So even if that fixes the problem, it won’t fix it right now. We should be able to find a workaround by either using the develop latest version, or maybe by executing a boto3 update command on start of your container. How are you currently self-hosting?

If you like, I can debug this issue from my local machine, but I would need to have access to an empty S3 bucket on Openstack/swift. I’ve never worked with that platform before, so it would help me a lot if you can create an empty bucket, and share the credentials with me. It can be deleted when I’m finished, of course.

6

Hello @bram ,

many thanks for your support.
I did set up a new S3 account at our cloud provider (Infomaniak).

It is a separate account so you can do anything required for your tests without risks.

What is your preferred method to receive the AWS credentials in a secure way ? I can encrypt them but I need to send you the secret and the key to use it on two different channels of course…

Let me know how you want to proceed. You have my email address on registration in this forum, feel free to use it for the purpose.

Thanks again

7

You are welcome to email the S3 account information to bram [at] baserow [dot] io, or share them via a private message here in the community. I just need access to the S3 bucket, I can replicate everything else on my local machine.

8

As mentioned in the email, here is the key : fM0FF0d2QPpBJebzN7o9m5RJ
Thanks to confirm that your access is working.
I created a “baserow” empty bucket but feel free to create other buckets if required for your tests.

9

Hey @philippelt, thank you for providing access. I’ve tried to upload a file from my local machine using Baserow, and encoutered the same problem. I updated boto3 to the latest version boto3==1.28.78, but still ran into the botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request.

Furthermore, I’ve even downloaded Commander One, which is an app for MacOS that allows you to connect to an S3 bucket. I was able to connect using your credentials, but when I tried to upload a file, I ran into this error:

image
image512×678 38 KB

Have you tried connecting and uploading to your bucket using other tools? If other tools fail as well, it might be a problem server side somehow. Normally, with other S3 providers we don’t have any problems.

10

Good to see that you are reproducing the issue.

I am using aws cli on a debian server to access the bucket. I just give it a try with the credentials I supplied to you:

aws --endpoint-url=https://s3.pub1.infomaniak.cloud s3api put-object --bucket baserow --key test.txt --body test.txt

I have no problem uploading or downloading objects with aws cli. The object is still in the bucket if you want to download it.

On Mac, I am using CyberDuck without problem and on IOS, I use a small app called “S3 Manager” that have a very basic user interface but that works without issues too.

IMPORTANT REMARK: The only tool that is requesting a “region” parameter is aws configure.
I never used this parameter as it is not mentioned as mandatory. Cyberduck as well as S3 Manager do not request this parameter.

I tried specifying dca-3 as region parameter in aws configure and this throws an error:

aws --endpoint-url=https://s3.pub1.infomaniak.cloud s3api put-object --bucket baserow --key test.txt --body test.txt

An error occurred () when calling the PutObject operation:

There may be an issue when specifying the region. This region name is included in the swift credentials but may not be used when using S3 compatibility mode.

I am going to remove the AWS_REGION_NAME parameter and try again on baserow.

11

Looks like this was the problem.
I removed the AWS_REGION_NAME parameter and it seems to work fine again now.

Conclusion : When using Openstack/Swift in S3 compatibility mode, the region name used in the swift native credentials may not be used with the EC2 (aws) credentials.

MANY THANKS for your support sorting out this configuration issue.
Tell me when I can close the Openstack test account just in case you would do some testings on your side.

Note: I am not sure this is a regular swift restriction. This may be a configuration issue in the swift setup of our cloud operator Infomaniak.

12

Hey @philippelt, glad to hear that the problem was solved. If everything is good on your side, then I don’t need access to the S3 bucket anymore. Feel free to close it.