Problems when using remote debug

Technical Help
1

I used the supervisor to run baserow on my server, but the function of importing files to create tables often failed. I used Pycharm to connect to my server and remotely debug to analyze the problem, but I couldn’t complete the debug. Does anyone know how to solve the problem?

WARNING: Baserow is configured to use a PUBLIC_BACKEND_URL of http://localhost:8000. If you attempt to access Baserow on any other hostname requests to the backend will fail as they will be from an unknown host.Please ensure you set PUBLIC_BACKEND_URL if you will be accessing Baserow from any other URL then http://localhost.
WARNING: Baserow is configured to use a default PUBLIC_WEB_FRONTEND_URL of http://localhost:3000. Emails sent by Baserow will use links pointing to http://localhost:3000 when telling users how to access your server. If this is incorrect please ensure you have set PUBLIC_WEB_FRONTEND_URL to the URL where users can access your Baserow server.
/home/baserow/env/lib/python3.8/site-packages/debugpy/_vendored/force_pydevd.py:20: UserWarning: incompatible copy of pydevd already imported:
 /root/.pycharm_helpers/pydev/_pydev_bundle/__init__.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/_pydev_calltip_util.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/_pydev_completer.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/_pydev_filesystem_encoding.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/_pydev_imports_tipper.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/_pydev_tipper_common.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/fix_getpass.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/pydev_console_types.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/pydev_imports.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/pydev_is_thread_alive.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/pydev_log.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/pydev_override.py
  /root/.pycharm_helpers/pydev/_pydev_bundle/pydev_stdin.py
  /root/.pycharm_helpers/pydev/_pydev_imps/__init__.py
  /root/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py
  /root/.pycharm_helpers/pydev/_pydev_imps/_pydev_saved_modules.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/__init__.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_additional_thread_info.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_additional_thread_info_regular.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_breakpointhook.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_breakpoints.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_bytecode_utils.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_collect_try_except_info.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm_constants.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_command_line_handling.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_console.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_console_integration.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_console_output.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_console_pytest.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_constants.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_custom_frames.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_dont_trace.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_dont_trace_files.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_exec2.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_extension_api.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_extension_utils.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_frame.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_frame_utils.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_import_class.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_io.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_kill_all_pydevd_threads.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_plugin_utils.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_process_net_command.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_resolver.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_save_locals.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_signature.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_tables.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_api.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_traceproperty.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_user_type_renderers.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_user_type_renderers_utils.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_utils.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_vars.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_vm_type.py
  /root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_xml.py
  /root/.pycharm_helpers/pydev/_pydevd_frame_eval/__init__.py
  /root/.pycharm_helpers/pydev/_pydevd_frame_eval/pydevd_frame_eval_main.py
  /root/.pycharm_helpers/pydev/pydev_ipython/__init__.py
  /root/.pycharm_helpers/pydev/pydevd.py
  /root/.pycharm_helpers/pydev/pydevd_concurrency_analyser/__init__.py
  /root/.pycharm_helpers/pydev/pydevd_concurrency_analyser/pydevd_concurrency_logger.py
  /root/.pycharm_helpers/pydev/pydevd_concurrency_analyser/pydevd_thread_wrappers.py
  /root/.pycharm_helpers/pydev/pydevd_file_utils.py
  /root/.pycharm_helpers/pydev/pydevd_plugins/__init__.py
  /root/.pycharm_helpers/pydev/pydevd_plugins/django_debug.py
  /root/.pycharm_helpers/pydev/pydevd_plugins/extensions/__init__.py
  /root/.pycharm_helpers/pydev/pydevd_plugins/extensions/types/__init__.py
  /root/.pycharm_helpers/pydev/pydevd_plugins/extensions/types/pydevd_helpers.py
  /root/.pycharm_helpers/pydev/pydevd_plugins/extensions/types/pydevd_plugin_numpy_types.py
  /root/.pycharm_helpers/pydev/pydevd_plugins/extensions/types/pydevd_plugins_django_form_str.py
  /root/.pycharm_helpers/pydev/pydevd_plugins/jinja2_debug.py
  /root/.pycharm_helpers/pydev/pydevd_tracing.py
  warnings.warn(msg + ':\n {}'.format('\n  '.join(_unvendored)))
python-BaseException
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/baserow/env/lib/python3.8/site-packages/debugpy/server/__init__.py", line 9, in <module>
    import debugpy._vendored.force_pydevd  # noqa
  File "/home/baserow/env/lib/python3.8/site-packages/debugpy/_vendored/force_pydevd.py", line 72, in <module>
    from _pydevd_bundle import pydevd_defaults
ImportError: cannot import name 'pydevd_defaults' from '_pydevd_bundle' (/root/.pycharm_helpers/pydev/_pydevd_bundle/__init__.py)

explain:
Baserow version: 1.12.1
Pycharm version：2022.1
Server: CentOS 7
Python env: Env used by baserow on the server
Deployment and access：Supervisor+Nginx

1 Like
2

In addition, I encountered import problems:
1、The page displays an error: Something went wrong during the file_import job execution，
2、I check the information of the import task in the database:

image
image1569×174 27.6 KB

3、
The “job__ 639. json” exists under “MEDIA_ROOT/user_id/file_import”：
image
Sometimes the import is successful, if it fails, it will succeed after several attempts.

@bram @nigel @Alex excuse me, I run Baserow to record various information of my class, so I need to use the function of importing files frequently, do you have any ideas on this question?

Update:
I seem to have found the reason! I now start two servers A and B, sharing the same Redis and Database. When I import files, server A’s backend receives interface requests:/api/database/tables/database/{database
_ Id}/async/, and “job__ {job_id}. json” is generated, But asynchronous tasks seem to be assigned to server B for execution, server B’s log will display:

==> exportworker.error <==
[2022-12-12 09:18:55,430: INFO/MainProcess] Task baserow.core.jobs.tasks.run_async_job[fabd16a6-ee86-432d-a383-5e899403d09f] received
[2022-12-12 09:18:55,459: FileNotFoundError: [Errno 2] No such file or directory: '/home/baserow/media/user_2/file_import/job__1076.json'

Is this problem caused by sharing Redis and database? Can you share your thoughts？My Supervisor Configuration:

[supervisord]
nodaemon = true
environment =
    DJANGO_SETTINGS_MODULE='baserow.config.settings.base',
    MEDIA_ROOT='/home/baserow/media',
    DATABASE_HOST='142.468.0.00',      
    DATABASE_PASSWORD='',
    SECRET_KEY='0ifhVlGHfxSVTY6mn0lzIwAuKsJNm9Xrdm1box10xnaaY4EklR8JFnVrEMp3CsWe',
    DATABASE_NAME="baserow",
    DATABASE_USER="baserow",
    PRIVATE_BACKEND_URL='http://backend:8000',  ---> (private IP of each server)
    PUBLIC_WEB_FRONTEND_URL='http://10.416.77.000:3000', ---> (public web-frontend nginx url)
    PUBLIC_BACKEND_URL='http://10.416.77.000:8000', ---> (public backend nginx url)
    MEDIA_URL='http://10.416.77.000:8080/media/',  ---> (media  nginx url)
    REDIS_HOST='142.328.0.00', 
    REDIS_PASSWORD='',
[program:gunicorn]
command = /home/baserow/python/virtual/env/bin/gunicorn -w 5 -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker baserow.config.asgi:application --log-level=debug --chdir=/home/baserow/
stdout_logfile=/home/baserow/logs/backend.log
stderr_logfile=/home/baserow/logs/backend.error
user=baserow

[program:worker]
command=/home/baserow/python/virtual/env/bin/celery -A baserow worker -l INFO -Q celery -n worker%%h
stdout_logfile=/home/baserow/logs/worker.log
stderr_logfile=/home/baserow/logs/worker.error
user=baserow

[program:exportworker]
command=/home/baserow/python/virtual/env/bin/celery -A baserow worker -l INFO -Q export -n exportworker%%h
stdout_logfile=/home/baserow/logs/exportworker.log
stderr_logfile=/home/baserow/logs/exportworker.error
user=baserow

[program:beatworker]
directory=/home/baserow/python
command=/home/baserow/python/virtual/env/bin/celery -A baserow beat -l INFO -S redbeat.RedBeatScheduler
stdout_logfile=/home/baserow/logs/exportworker.log
stderr_logfile=/home/baserow/logs/exportworker.error
user=baserow

[program:nuxt]
directory = /home/baserow/web-frontend
command = node ./node_modules/.bin/nuxt start --hostname 196.542.15.00 --port 3000 --config-file ./config/nuxt.config.dev.js
stdout_logfile = /home/baserow/logs/frontend.log
stderr_logfile = /home/baserow/logs/frontend.error
user=baserow
3

Hi @Chase, you must actually share the same Redis and PostgreSQL between servers and workers. Both servers need to be able to access the same uploaded media files. Would it be possible to share the same volume between servers or to use something S3 compatible?

4

Hi @bram I confirm that different servers have used the same Redis and PostgreSQL, but each server has set up its own “MEDIA_ROOT”, and the server cannot access the directory of other servers.
So I need to setup a common “MEDIA_ROOT” directory that all servers use for uploads file and reads file?

5

Yes, exactly. And you also need to have a web server that serves those static files on the MEDIA_URL.

6

OK, I got it, thank you for your guidance! But I don’t quite understand your words:

And you also need to have a web server that serves those static files on the MEDIA_URL .

I only know the “MEDIA_URL” to be used to download the exported file, What are the static files? Can you give me an example?

7

Sorry, I mean the media files instead of static files.

8

Ok, let me confirm my understanding again: I need to create a “MEDIA_ROOT” shared by all servers, and build a web server for all servers, and use “MEDIA_URL” to access this “MEDIA_ROOT”, Is that right?

9

That’s correct @Chase