PostgreSQL synchronization guidance

Technical Help
1

Are you using our SaaS platform (Baserow.io) or self-hosting Baserow?

Self-hosted

If you are self-hosting, what version of Baserow are you running?

1.30

If you are self-hosting, which installation method do you use to run Baserow?

Docker

What are the exact steps to reproduce this issue?

Hello!, I need guidance here, I am looking to use Baserow to hold a large amount of records from our Postgres db, we currently have over 3m+ records in a table, and I am trying to use Baserow - Postgres table sync functionality, the thing is, it crashes the application, I have it running on a 16gb RAM machine, it says it ran out of memory, can you guys tell me if I need to tweak configurations? or am I doing this incorrectly

The main goal is to get all new records from that table and use them in baserow only, but data will keep growing and we will need to include these new records in Baserow too.

2

Hey @pedrolabrador, the data sync functionality loads all the data in memory, and then does a comparison on which rows already exist. Working with 3m is quite a lot for a no-code database, but not impossible. Is there a way you can temporarily increase the memory of the machine? I’m not sure how much is exactly needed because that depends on the number of columns and how much data is in the cells as well.

3

Hi, we created a machine with 16gb and then the progress bar got stuck in 10% (the progress bar shown during the sync), after we increased it to 128gb (which we thought would be enough), it only took us to 14%, but that machine costs a lot, and that is a lot of memory, so we thought it wouldn’t work

4

Do the output logs of your container give any error?