I Need A "Human-in-the-Loop" Data Sync Strategy

Automations Builder
1

Hi Baserow Team,

I am building a workflow to merge a large external dataset (+5,000 records from a collaborator’s Google Sheet) into a mission-critical main database. Because the external data structure is inconsistent and requires manual verification before going live, I want to use a “Buffer Table” strategy.

My planned workflow:

  1. Initial Import: Using Make.com to push raw data from the external source into a “Sandbox” table (which is a duplicate of my Main Table structure).

  2. Manual Linking: Using a Link to Table field in the Sandbox to manually associate the incoming records with the correct existing records in the Main Table. This is a maybe…I would rather not have to individually link 5,000 items.

  3. The Injection: Creating a native Baserow Automation that triggers when an Approved checkbox is checked in the Sandbox.

The Automation Logic:

  • Trigger: Row updated (where Approved = true).

  • Action: Update row in Main Table or Create Row if row not present.

  • Row ID: Using the ID from the Link to Table field. (Again….MAYBE….)

My questions for the team:

  • Performance: With a potential batch of 5,000 updates, are there any rate limits or performance considerations I should be aware of when using native automations to “inject” data from one table to another?

  • Mapping Efficiency: Is the “Update Row” action using a Link field ID the most stable way to handle this, or is there a more efficient way to perform a “bulk upsert” from a buffer table natively? (Again…I don’t want to link 5,000 items…)

  • Data Integrity: If the automation is triggered for a row where the Link field is empty, will it simply fail gracefully, or is there a way to add a “field is not empty” condition to the trigger to prevent errors?

I want to ensure this is as “leak-proof” as possible so that no unverified data accidentally hits the main database. Looking forward to your insights on the best way to handle this!

Just a quick FYI: I’m not a business or a corporation—just an individual working on a personal project to catalog items. I’m definitely not an automation wizard; honestly, I have no idea what I’m doing half the time! I rely heavily on AI to help me figure out how to set these workflows up, so I really appreciate your patience and any “plain English” guidance you can provide. :purple_heart:

I did consider trying to pay someone to set this all up for me, but I wouldn’t even know where to look, and my budget is very small. Not to mention, I don’t feel comfortable giving a complete stranger access to my database! That’s why I’m trying to learn the best way to do this myself.

2

Hi @Iokaar

Performance: With a potential batch of 5,000 updates, are there any rate limits or performance considerations I should be aware of when using native automations to “inject” data from one table to another?

If you are using Baserow SaaS, the execution rate limits are currently set to 10 executions/5s, 30/60s and 100/120s. These values can changed in the future since automations are still in beta. But since your automation is triggered by manually setting the Approved flag. I don’t see any risk here.

Mapping Efficiency: Is the “Update Row” action using a Link field ID the most stable way to handle this, or is there a more efficient way to perform a “bulk upsert” from a buffer table natively? (Again…I don’t want to link 5,000 items…)

Maybe you can rely on a unique field that is shared by the sandbox and the targeted table if that feasbile ? That way you can probably resolve the target row via the worflow logic and no need to handle this manually. Otherwise The Link field ID is a stable way to achieve that.

Data Integrity: If the automation is triggered for a row where the Link field is empty, will it simply fail gracefully, or is there a way to add a “field is not empty” condition to the trigger to prevent errors?

You can use a router type node right after the trigger so you can define 2 different behaviors according to your field value. If the Link field is empty you create a row, if not you update a row.

I hope this helps.