Upload file - Bucket S3

Technical Help
1

Hello,

I have a probleme when trying to upload a picture. The probleme occurs from web as well as with API.

Here is the log

 [BACKEND][2023-11-13 12:45:47] 127.0.0.1:46652 - "GET /api/_health/ HTTP/1.1" 200  
 [BACKEND][2023-11-13 12:45:47] ERROR 2023-11-13 12:45:47,974 django.request.log_response:224- Internal Server Error: /api/user-files/upload-file/   
 [BACKEND][2023-11-13 12:45:47] Traceback (most recent call last):  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 486, in thread_handler  
 [BACKEND][2023-11-13 12:45:47]     raise exc_info[1]  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 38, in inner  
 [BACKEND][2023-11-13 12:45:47]     response = await get_response(request)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 233, in _get_response_async  
 [BACKEND][2023-11-13 12:45:47]     response = await wrapped_callback(request, *callback_args, **callback_kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 448, in __call__  
 [BACKEND][2023-11-13 12:45:47]     ret = await asyncio.wait_for(future, timeout=None)  
 [BACKEND][2023-11-13 12:45:47]   File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for  
 [BACKEND][2023-11-13 12:45:47]     return await fut  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/current_thread_executor.py", line 22, in run  
 [BACKEND][2023-11-13 12:45:47]     result = self.fn(*self.args, **self.kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 490, in thread_handler  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view  
 [BACKEND][2023-11-13 12:45:47]     return view_func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view  
 [BACKEND][2023-11-13 12:45:47]     return self.dispatch(request, *args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch  
 [BACKEND][2023-11-13 12:45:47]     response = self.handle_exception(exc)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception  
 [BACKEND][2023-11-13 12:45:47]     self.raise_uncaught_exception(exc)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception  
 [BACKEND][2023-11-13 12:45:47]     raise exc  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch  
 [BACKEND][2023-11-13 12:45:47]     response = handler(request, *args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/usr/lib/python3.9/contextlib.py", line 79, in inner  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwds)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/api/decorators.py", line 107, in func_wrapper  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/api/user_files/views.py", line 61, in post  
 [BACKEND][2023-11-13 12:45:47]     user_file = UserFileHandler().upload_user_file(request.user, file.name, file)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/user_files/handler.py", line 252, in upload_user_file  
 [BACKEND][2023-11-13 12:45:47]     self.generate_and_save_image_thumbnails(image, user_file, storage=storage)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/user_files/handler.py", line 168, in generate_and_save_image_thumbnails  
 [BACKEND][2023-11-13 12:45:47]     handler.save(thumbnail_path, thumbnail_stream)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/storage.py", line 9, in save  
 [BACKEND][2023-11-13 12:45:47]     if self.storage.exists(name):  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/storages/backends/s3boto3.py", line 463, in exists  
 [BACKEND][2023-11-13 12:45:47]     self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call  
 [BACKEND][2023-11-13 12:45:47]     return self._make_api_call(operation_name, kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/botocore/client.py", line 960, in _make_api_call  
 [BACKEND][2023-11-13 12:45:47]     raise error_class(parsed_response, operation_name)  
 [BACKEND][2023-11-13 12:45:47] botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request  
 [BACKEND][2023-11-13 12:45:47] ERROR 2023-11-13 12:45:47,974 django.request.log_response:224- Internal Server Error: /api/user-files/upload-file/   
 [BACKEND][2023-11-13 12:45:47] Traceback (most recent call last):  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 486, in thread_handler  
 [BACKEND][2023-11-13 12:45:47]     raise exc_info[1]  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 38, in inner  
 [BACKEND][2023-11-13 12:45:47]     response = await get_response(request)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 233, in _get_response_async  
 [BACKEND][2023-11-13 12:45:47]     response = await wrapped_callback(request, *callback_args, **callback_kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 448, in __call__  
 [BACKEND][2023-11-13 12:45:47]     ret = await asyncio.wait_for(future, timeout=None)  
 [BACKEND][2023-11-13 12:45:47]   File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for  
 [BACKEND][2023-11-13 12:45:47]     return await fut  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/current_thread_executor.py", line 22, in run  
 [BACKEND][2023-11-13 12:45:47]     result = self.fn(*self.args, **self.kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 490, in thread_handler  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view  
 [BACKEND][2023-11-13 12:45:47]     return view_func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view  
 [BACKEND][2023-11-13 12:45:47]     return self.dispatch(request, *args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch  
 [BACKEND][2023-11-13 12:45:47]     response = self.handle_exception(exc)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception  
 [BACKEND][2023-11-13 12:45:47]     self.raise_uncaught_exception(exc)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception  
 [BACKEND][2023-11-13 12:45:47]     raise exc  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch  
 [BACKEND][2023-11-13 12:45:47]     response = handler(request, *args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/usr/lib/python3.9/contextlib.py", line 79, in inner  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwds)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/api/decorators.py", line 107, in func_wrapper  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/api/user_files/views.py", line 61, in post  
 [BACKEND][2023-11-13 12:45:47]     user_file = UserFileHandler().upload_user_file(request.user, file.name, file)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/user_files/handler.py", line 252, in upload_user_file  
 [BACKEND][2023-11-13 12:45:47]     self.generate_and_save_image_thumbnails(image, user_file, storage=storage)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/user_files/handler.py", line 168, in generate_and_save_image_thumbnails  
 [BACKEND][2023-11-13 12:45:47]     handler.save(thumbnail_path, thumbnail_stream)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/storage.py", line 9, in save  
 [BACKEND][2023-11-13 12:45:47]     if self.storage.exists(name):  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/storages/backends/s3boto3.py", line 463, in exists  
 [BACKEND][2023-11-13 12:45:47]     self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call  
 [BACKEND][2023-11-13 12:45:47]     return self._make_api_call(operation_name, kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/botocore/client.py", line 960, in _make_api_call  
 [BACKEND][2023-11-13 12:45:47]     raise error_class(parsed_response, operation_name)  
 [BACKEND][2023-11-13 12:45:47] botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request  
 [BACKEND][2023-11-13 12:45:47] ERROR 2023-11-13 12:45:47,974 django.request.log_response:224- Internal Server Error: /api/user-files/upload-file/   
 [BACKEND][2023-11-13 12:45:47] Traceback (most recent call last):  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 486, in thread_handler  
 [BACKEND][2023-11-13 12:45:47]     raise exc_info[1]  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 38, in inner  
 [BACKEND][2023-11-13 12:45:47]     response = await get_response(request)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 233, in _get_response_async  
 [BACKEND][2023-11-13 12:45:47]     response = await wrapped_callback(request, *callback_args, **callback_kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 448, in __call__  
 [BACKEND][2023-11-13 12:45:47]     ret = await asyncio.wait_for(future, timeout=None)  
 [BACKEND][2023-11-13 12:45:47]   File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for  
 [BACKEND][2023-11-13 12:45:47]     return await fut  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/current_thread_executor.py", line 22, in run  
 [BACKEND][2023-11-13 12:45:47]     result = self.fn(*self.args, **self.kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py", line 490, in thread_handler  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view  
 [BACKEND][2023-11-13 12:45:47]     return view_func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view  
 [BACKEND][2023-11-13 12:45:47]     return self.dispatch(request, *args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch  
 [BACKEND][2023-11-13 12:45:47]     response = self.handle_exception(exc)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception  
 [BACKEND][2023-11-13 12:45:47]     self.raise_uncaught_exception(exc)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception  
 [BACKEND][2023-11-13 12:45:47]     raise exc  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch  
 [BACKEND][2023-11-13 12:45:47]     response = handler(request, *args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/usr/lib/python3.9/contextlib.py", line 79, in inner  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwds)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/api/decorators.py", line 107, in func_wrapper  
 [BACKEND][2023-11-13 12:45:47]     return func(*args, **kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/api/user_files/views.py", line 61, in post  
 [BACKEND][2023-11-13 12:45:47]     user_file = UserFileHandler().upload_user_file(request.user, file.name, file)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/user_files/handler.py", line 252, in upload_user_file  
 [BACKEND][2023-11-13 12:45:47]     self.generate_and_save_image_thumbnails(image, user_file, storage=storage)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/user_files/handler.py", line 168, in generate_and_save_image_thumbnails  
 [BACKEND][2023-11-13 12:45:47]     handler.save(thumbnail_path, thumbnail_stream)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/backend/src/baserow/core/storage.py", line 9, in save  
 [BACKEND][2023-11-13 12:45:47]     if self.storage.exists(name):  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/storages/backends/s3boto3.py", line 463, in exists  
 [BACKEND][2023-11-13 12:45:47]     self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call  
 [BACKEND][2023-11-13 12:45:47]     return self._make_api_call(operation_name, kwargs)  
 [BACKEND][2023-11-13 12:45:47]   File "/baserow/venv/lib/python3.9/site-packages/botocore/client.py", line 960, in _make_api_call  
 [BACKEND][2023-11-13 12:45:47]     raise error_class(parsed_response, operation_name)  
 [BACKEND][2023-11-13 12:45:47] botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

From the api, the call response:

{"error":"ERROR_FILE_URL_COULD_NOT_BE_REACHED","detail":"The provided URL could not be reached."}

I’ve updated from 1.18 to 1.21.2 few days ago, it can be linked.

Any ideas ?

2

Some updates that discard access/security problems from the server to the S3 bucket.

Indeed, I succeeded to upload some file, with commande lines, from the server where the docker is hosted.

Here are the steps:

Installing python, awscli and awscli-plugin-endpoint

sudo apt-get install python3-venv
python3 -m venv env
source env/bin/activate
pip3 install awscli awscli-plugin-endpoint

set up aws credentials with

aws configure

then accessing s3 and upload file with

aws s3 ls --endpoint-url 'https://s3.gra.io.cloud.ovh.net/'
aws s3 cp ~/my-file.txt s3://my-bucket --endpoint-url 'https://my-bucket.s3.gra.io.cloud.ovh.net'

To be sure, I re-lanched the docker with the below command:

docker run \
  -d \
  --name baserow \
  -e BASEROW_PUBLIC_URL=https://site.com \
  -e BASEROW_CADDY_ADDRESSES=https://site.com\
  -e DATABASE_HOST=postgresql-xxxxxxxxxxxxxxxxxxx.cloud.ovh.net \
  -e DATABASE_NAME=defaultdb \
  -e DATABASE_USER=admin \
  -e DATABASE_PASSWORD=xxxxxxxxxxxxxxxxxxx \
  -e DATABASE_PORT=20184 \
  -e EMAIL_SMTP='TRUE' \
  -e EMAIL_SMTP_HOST='smtp.gmail.com' \
  -e EMAIL_SMTP_PORT='587' \
  -e EMAIL_SMTP_USER='email@gmail.com' \
  -e EMAIL_SMTP_PASSWORD='xxxxxxxxxxxxxxxxxxx' \
  -e EMAIL_SMTP_USE_TLS='TRUE' \
  -e FROM_EMAIL='email@gmail.com' \
  -e HOURS_UNTIL_TRASH_PERMANENTLY_DELETED=720 \
  -e BASEROW_TRIGGER_SYNC_TEMPLATES_AFTER_MIGRATION=false \
  -e BASEROW_ROW_PAGE_SIZE_LIMIT=5000\
  -e AWS_ACCESS_KEY_ID="xxxxxxxxxxxxxxxxxxx"\
  -e AWS_SECRET_ACCESS_KEY="xxxxxxxxxxxxxxxxxxx"\
  -e AWS_STORAGE_BUCKET_NAME="my-bucket"\
  -e AWS_S3_ENDPOINT_URL="https://s3.gra.io.cloud.ovh.net/"\
  -v baserow_data:/baserow/data \
  -p 80:80 \
  -p 443:443 \
  --restart unless-stopped \
  baserow/baserow:1.21.2

But still the same error :

Capture d’écran 2023-11-14 à 17.45.27
Capture d’écran 2023-11-14 à 17.45.27780×375 21.7 KB

Any thoughts ?

3

I tried to play with the S3 relative environment variable and “AWS_S3_REGION_NAME” is mandatory in my case.

Here is what worked for me:

  -e AWS_ACCESS_KEY_ID="xxxxxxxxxxxxxxxxxxx"\
  -e AWS_SECRET_ACCESS_KEY="xxxxxxxxxxxxxxxxxxx"\
  -e AWS_STORAGE_BUCKET_NAME="my-bucket"\
  -e AWS_S3_ENDPOINT_URL="https://s3.gra.io.cloud.ovh.net"\
  -e AWS_S3_REGION_NAME="gra"\
4

Thanks for sharing the solution, @Clapp. If you encounter any further issues, I’ll ask the dev team to help you asap. :raised_hands:

1 Like