AI field vs celery workers

Technical Help
1

I’ve found that for generating large numbers of AI fields concurrently in the self-host environment I seem to be limited by the number of celery workers.
Most of them only use up to 20-25% of the thread’s (logical core) processing power as reported by top.

I was wondering if there is any room for improvement here in cpu utilisation or maybe more concurrent workers. I’m not entirely sure what the actual bottleneck is in this case, so I’d like to brainstorm some options.

What would be the risks of going for more celery workers than available cores in this specific context?

2

Hi!

Could you remind how you’re running Baserow?

  • Is it all-in-one deployment or you’re running frontend/backend/celery in separate containers?
  • Did you adjust/change number of CPUs assigned to specific containers or for Docker on your host?
  • Could you post logs from Celery containers from app initialization? There’s an important hint there: Celery reports its concurrency configuration.
3

Also, take a look at Celery workers configuration in Baserow: Configuring Baserow