Snapshot error : Something went wrong during the create_snapshot job execution (KeyError)

Technical Help
1

Please fill in the questionnaire below.

Technical Help Questionnaire

Have you read and followed the instructions at: *READ ME FIRST* Technical Help FAQs - #2 by nigel ?

Answer: Yes

Self-Hosted Installation and Setup Questions

How have you self-hosted Baserow.

Custom Kubernetes install

What are the specs of the service or server you are using to host Baserow.

Large

Which version of Baserow are you using.

1.27.2

How have you configured your self-hosted installation?

n/a

What commands if any did you use to start your Baserow server?

Multiples pods, multiple commands

Describe the problem

Describe, step by step, how to reproduce the error or problem you are encountering.

Until 3 days ago, We did snapshot without any problems of all important DB (~15 projects/DB).
On october 30, one database suddenly became impossible to backup, nor interactively, nor using API requests.
Using API, and requesting status of the created job ID, I get the following results (two job status call at 5 sec interval):

{'human_readable_error': '',
 'id': 1550,
 'progress_percentage': 0,
 'state': 'started',
 'type': 'create_snapshot'}

{'human_readable_error': 'Something went wrong during the create_snapshot job '
                         'execution.',
 'id': 1550,
 'progress_percentage': 0,
 'state': 'failed',
 'type': 'create_snapshot'}

Interactively, I get the same error message in a red box.

Provide screenshots or include share links showing:

We are running multiple projects, most with 1 db, some with multiple DBs and one with a single DB and an app.

How many rows in total do you have in your Baserow tables?

Less than 200, some tables a lot less

Please attach full logs from all of Baserow’s services

The log from baserow workers pod includes :

[2024-11-01 17:02:36,546: ERROR/ForkPoolWorker-8] Task baserow.core.jobs.tasks.run_async_job[92567f81-b988-4e10-96cb-d62b20a92bdc] raised unexpected: KeyError('DED')

The full name of the database that we can non longer backup is DED & DPR

Where should I try to find more information to help you diagnose the issue ?

Everythings worked fine up to october 29. I don’t know if users made significant changes in the DB structure on october 30.

Two more informations :

  • All others backups are done every day without troubles
  • This project is the only one which have 2 applications (their backup is done without problems too)
2

More extract from the log of baserow workers (container export) :

[2024-11-02 02:02:54,437: INFO/MainProcess] Task baserow.core.jobs.tasks.run_async_job[e87f2072-d44e-44f8-8d38-a97979416317] recei
ved
[2024-11-02 02:02:57,621: ERROR/ForkPoolWorker-8] Task baserow.core.jobs.tasks.run_async_job[e87f2072-d44e-44f8-8d38-a97979416317]
 raised unexpected: KeyError('DED')
Traceback (most recent call last):
  File "/baserow/venv/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/baserow/venv/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/baserow/backend/src/baserow/core/jobs/tasks.py", line 34, in run_async_job
    JobHandler().run(job)
  File "/baserow/backend/src/baserow/core/jobs/handler.py", line 59, in run
    return job_type.run(job, progress)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/baserow/backend/src/baserow/core/snapshots/job_types.py", line 46, in run
    action_type_registry.get(CreateSnapshotActionType.type).do(
  File "/baserow/backend/src/baserow/core/snapshots/actions.py", line 50, in do
    snapshot_created = SnapshotHandler().perform_create(snapshot, progress)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/baserow/backend/src/baserow/core/snapshots/handler.py", line 411, in perform_create
    application_type.import_serialized(
  File "/baserow/backend/src/baserow/contrib/database/application_types.py", line 894, in import_serialized
    self.import_tables_serialized(
  File "/baserow/backend/src/baserow/contrib/database/application_types.py", line 477, in import_tables_serialized
    imported_fields = self._import_table_fields(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/baserow/backend/src/baserow/contrib/database/application_types.py", line 349, in _import_table_fields
    fields_with_dependencies = deferred_field_importer.run_deferred_field_imports(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/baserow/backend/src/baserow/contrib/database/fields/utils/deferred_field_importer.py", line 101, in run_deferred_field_im
ports
    dependencies = self.get_all_fields_dependencies(field_name_fields_mapping)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/baserow/backend/src/baserow/contrib/database/fields/utils/deferred_field_importer.py", line 70, in get_all_fields_depende
ncies
    link_row_field = field_name_fields_mapping[table.id][via_field_name]
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'DED'
3

For information, the problem disappeared after the user has done some cleanup in tables of this databases that had invalid fields.

Fields where identified with a warning symbol (red triangle) in the column header. I did not obtain detailed information from the user on the list of changes done and as I was not able to backup the “corrupted” database, I can’t compare tables definition.

Before the user did the cleanup, I tried to export each of the 16 tables, trying to locate the problematic table, but it was possible to export all of them despite the backup was still impossible.

By the way, that means that it is possible to create incorrect data definition in tables (like referencing foreign tables/fields that no longer exist) and this situation is allowed by baserow but do no longer allow backup.

It may be a good practice to not allow invalid fields in tables when they are used for links. Cascading a deletion when a field used as a link is destroyed should be considered…

Thanks for your work

4

Hey @philippelt, I’m glad to hear that the issue is now resolved.

I’ll share your feedback with the development team. :slightly_smiling_face:

5

Hey @philippelt, I’ve shared your feedback with the development team, and they’ve confirmed this is actually a bug—we’ll fix it in the next release. :slightly_smiling_face:

1 Like
6

Many thanks for the feedback.
And best wishes of success for the dev team that is moving fast !

1 Like