MalformedSecurityHeader error when saving attachments to Google Cloud Storage bucket

hugh · April 16, 2023, 2:06am

Thanks to @nigel 's help I have been able to get a self-hosted instance running as a Google Cloud Run service.

I’m now trying to get file storage in Google Cloud Storage working, but I’m running into an issue.

I first tested by setting up an AWS S3 bucket, and configuring the instance to use it for storage, which succeeded. I was able to save attachments and subsequently download them.

I then created a Google Cloud Storage bucket, and generated an HMAC key pair for the default service account for the project. I swapped in the bucket name, key, and secret key environment variables, and also set the URL to the Google Cloud Storage endpoint:

AWS_ACCESS_KEY_ID = EG9aer[...MY ACCESS KEY]

AWS_SECRET_ACCESS_KEY = GOOG1E[...MY SECRET KEY]

AWS_STORAGE_BUCKET_NAME = [MY BUCKET NAME]

AWS_S3_ENDPOINT_URL = https://storage.googleapis.com

But when I attempt to upload an attachment it fails, and in the logs I see

botocore.exceptions.ClientError: An error occurred (MalformedSecurityHeader) when calling the PutObject operation: Your request has a malformed header.

My understanding is Google Cloud Storage is meant to be S3 compliant.

But maybe there’s some variation in the API that the upload client is not handling?

Here’s more of the logs:

"e[34m [BACKEND][2023-04-16 01:56:30] 169.254.1.1:0 - ""POST /api/user-files/upload-file/ HTTP/1.1"" 500 e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:27] 127.0.0.1:44624 - ""GET /_health/ HTTP/1.1"" 200 e(Be[m "
"e[35m [CADDY][2023-04-16 01:56:11] {""level"":""debug"",""ts"":1681610142.5168178,""logger"":""http.handlers.reverse_proxy"",""msg"":""upstream roundtrip"",""upstream"":""localhost:8000"",""duration"":0.131112681,""request"":{""remote_addr"":""169.254.1.1:43904"",""proto"":""HTTP/1.1"",""method"":""PATCH"",""host"":""baserow-jcm5wjg5rq-nn.a.run.app"",""uri"":""/api/database/rows/table/1348/1/"",""headers"":{""Content-Length"":[""18""],""Dnt"":[""1""],""Websocketid"":[""855ebbdd-604d-4309-b8aa-905fc74cee0f""],""Forwarded"":[""for=\""38.141.208.122\"";proto=https""],""Sec-Ch-Ua"":[""\""Chromium\"";v=\""112\"", \""Google Chrome\"";v=\""112\"", \""Not:A-Brand\"";v=\""99\""""],""Authorization"":[""JWT eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjgxNjEwNjkzLCJpYXQiOjE2ODE2MTAwOTMsImp0aSI6IjBlYWNjM2M4NTNlODRiZDNiZmJiOWY4MDhhOGVlMWMxIiwidXNlcl9pZCI6MX0.f3smuuw283MYEKRUuVMvBMWXCPXBKIjp8AoEDDLJlTI""],""Content-Type"":[""application/json""],""Sec-Ch-Ua-Platform"":[""\""Windows\""""],""Referer"":[""https://baserow-jcm5wjg5rq-nn.a.run.app/database/260/table/1348""],""Sec-Gpc"":[""1""],""X-Cloud-Trace-Context"":[""f086613440d6f80bc8ebedc9afa722e0/15924109557000586493""],""Sec-Ch-Ua-Mobile"":[""?0""],""Clientsessionid"":[""f3a029af-8b14-48e9-b5d2-0af36fc5950e""],""Origin"":[""https://baserow-jcm5wjg5rq-nn.a.run.app""],""Sec-Fetch-Dest"":[""empty""],""X-Forwarded-For"":[""38.141.208.122, 169.254.1.1""],""Accept-Encoding"":[""gzip, deflate, br""],""X-Forwarded-Proto"":[""https""],""Accept"":[""application/json""],""User-Agent"":[""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36""],""Sec-Fetch-Site"":[""same-origin""],""Sec-Fetch-Mode"":[""cors""],""Accept-Language"":[""en-US,en;q=0.9""],""Cookie"":[""i18n-language=en; jwt_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0b2tlbl90eXBlIjoicmVmcmVzaCIsImV4cCI6MTY4MjIxNDU5MywiaWF0IjoxNjgxNjA5NzkzLCJqdGkiOiIyMWU0NWQ0NDhiMTU0NzJkODk1MDU2YTE2OTNiYTM3ZCIsInVzZXJfaWQiOjF9.dg6CWejrFsj1APKVkkPJ1VROOzweHpWA9lOKUKUvbc0; baserow_group_id=254""],""Traceparent"":[""00-f086613440d6f80bc8ebedc9afa722e0-dcfdcd45c56c44fd-00""]}},""headers"":{""Date"":[""Sun, 16 Apr 2023 01:55:42 GMT""],""Content-Type"":[""application/json""],""Allow"":[""GET, PATCH, DELETE, HEAD, OPTIONS""],""X-Frame-Options"":[""DENY""],""Access-Control-Allow-Origin"":[""*""],""Server"":[""uvicorn""],""Content-Length"":[""139""],""X-Content-Type-Options"":[""nosniff""],""Referrer-Policy"":[""same-origin""],""Vary"":[""Origin""]},""status"":200} e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11] botocore.exceptions.ClientError: An error occurred (MalformedSecurityHeader) when calling the PutObject operation: Your request has a malformed header. e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]     raise error_class(parsed_response, operation_name) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/botocore/client.py"", line 960, in _make_api_call e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     return self._make_api_call(operation_name, kwargs) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/botocore/client.py"", line 530, in _api_call e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/s3transfer/upload.py"", line 758, in _main e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     return_value = self._main(**kwargs) e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/s3transfer/tasks.py"", line 162, in _execute_main e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     return self._execute_main(kwargs) e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/s3transfer/tasks.py"", line 139, in __call__ e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     raise self._exception e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/s3transfer/futures.py"", line 266, in result e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     return self._coordinator.result() e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/s3transfer/futures.py"", line 103, in result e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     return future.result() e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/boto3/s3/inject.py"", line 636, in upload_fileobj e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     return self.meta.client.upload_fileobj( e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/boto3/s3/inject.py"", line 725, in object_upload_fileobj e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     obj.upload_fileobj(content, ExtraArgs=params) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/storages/backends/s3boto3.py"", line 459, in _save e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     name = self._save(name, content) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/django/core/files/storage.py"", line 54, in save e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     storage.save(full_path, stream) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/backend/src/baserow/core/user_files/handler.py"", line 257, in upload_user_file e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     user_file = UserFileHandler().upload_user_file(request.user, file.name, file) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/backend/src/baserow/api/user_files/views.py"", line 61, in post e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     return func(*args, **kwargs) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/backend/src/baserow/api/decorators.py"", line 107, in func_wrapper e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     return func(*args, **kwds) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/usr/lib/python3.9/contextlib.py"", line 79, in inner e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     response = handler(request, *args, **kwargs) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py"", line 506, in dispatch e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     raise exc e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py"", line 480, in raise_uncaught_exception e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     self.raise_uncaught_exception(exc) e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py"", line 469, in handle_exception e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     response = self.handle_exception(exc) e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py"", line 509, in dispatch e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     return self.dispatch(request, *args, **kwargs) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/django/views/generic/base.py"", line 70, in view e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     return view_func(*args, **kwargs) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/django/views/decorators/csrf.py"", line 54, in wrapped_view e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     return func(*args, **kwargs) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py"", line 490, in thread_handler e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     result = self.fn(*self.args, **self.kwargs) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/asgiref/current_thread_executor.py"", line 22, in run e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     return await fut e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/usr/lib/python3.9/asyncio/tasks.py"", line 442, in wait_for e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     ret = await asyncio.wait_for(future, timeout=None) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py"", line 448, in __call__ e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]     response = await wrapped_callback(request, *callback_args, **callback_kwargs) e(Be[m "
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/django/core/handlers/base.py"", line 233, in _get_response_async e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     response = await get_response(request) e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/django/core/handlers/exception.py"", line 38, in inner e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11]     raise exc_info[1] e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11]   File ""/baserow/venv/lib/python3.9/site-packages/asgiref/sync.py"", line 486, in thread_handler e(Be[m "
e[34m [BACKEND][2023-04-16 01:56:11] Traceback (most recent call last): e(Be[m 
"e[34m [BACKEND][2023-04-16 01:56:11] ERROR 2023-04-16 01:56:11,162 django.request.log_response:224- Internal Server Error: /api/user-files/upload-file/  e(Be[m "

hugh · April 16, 2023, 5:21am

Looks like there’s a Google Cloud Storage-specific storage backend that requires different configuration variables.

https://django-storages.readthedocs.io/en/latest/backends/gcloud.html

It looks like GC could be used for storage by handling some additional environment variables. Something like this might work:

# baserow.config.settings.base

# Optional Google Cloud Storage storage configuration
if os.getenv("GS_BUCKET_NAME", "") != "":

    DEFAULT_FILE_STORAGE = 'config.storage_backends.GoogleCloudMediaStorage'
    GS_BUCKET_NAME = 'GS_BUCKET_NAME'
    
    # "The storage must always overwrite existing files."
    GS_FILE_OVERWRITE = True

    if os.getenv("GS_PROJECT_ID", "") != "":
        GS_PROJECT_ID = os.getenv("GS_PROJECT_ID")

    if os.getenv("GS_CUSTOM_ENDPOINT", "") != "":
        GS_CUSTOM_ENDPOINT = os.getenv("GS_CUSTOM_ENDPOINT")

    if os.getenv("GS_DEFAULT_ACL", "") != "":
        GS_CUSTOM_ENDPOINT = os.getenv("GS_DEFAULT_ACL")

    if os.getenv("GS_QUERYSTRING_AUTH", "") != "":
        GS_CUSTOM_ENDPOINT = os.getenv("GS_QUERYSTRING_AUTH")

Authentication could be handled by granting the service account in use by the Cloud Run service or Compute Engine instance access to the bucket.

I haven’t tested this.

bram · April 20, 2023, 8:39am

Hey @hugh, thanks for sharing that. I’m checking in to see if you already managed to fix storing the files in Google Cloud Media Storage? If not, then I recommend creating an issue here https://gitlab.com/bramw/baserow/-/issues/new so that our developers can introduce those environment variables.

hugh · April 21, 2023, 12:43am

I haven’t tested modifying the configuration to acknowledge those environment variables myself (yet). I will add a new issue requesting that it be done. Thanks.

Edit: done - Enable Google Cloud Storage storage backend (#1702) · Issues · Bram Wiepjes / baserow · GitLab