Webhook Infinite Loop

[Thorby]

Hi,

I have run into an unexpected error using Webhook.

I have a webhook:

When a row is created or updated I receive a webhook (both states are needed).

On my server I evaluate the data I receive from the webhook, and if necessary update row fields (on the same row) via the Baserow API.

However, this is causing an infinite loop.

Each time I update Baserow via the API , it appears the the webhook excutes again… and so we get stuck in an infinite loop!!!

Surely, webhooks are only be instgated when changes (create/update) are executed via the Baserow UI ???

Please advise.

[cwinhall]

Hi @Thorby,

You make a good point and I’ve brought it up as a discussion internally.

I can only think of a solution for a workaround by creating a new boolean field which you can use as a “flag” and you can set this on or off via the API update and then filter by that in the following webhook update which would essentially stop the loop.

[Thorby]

Thanks Colin,
Many thanks for coming back to me and suggesting a workaround, however I’m not sure the workaround is viable.

Lets say I have a field named FLAG, and FLAG = true.
And a field named DATA

I make changes to DATA via Baserow UI.
This triggers a webhook.

I receive the webhook on my server and can test the field FLAG:
if FLAG = true, then process data
else Stop

my process data function:

1. process the data on my application,
2. update the row on Baserow, via the API (field_DATA = {my new data}, field_FLAG = false)

Once the API upates the 2 fields, the webhook will trigger again. However, this time my test will see FLAG = fale and will Stop - no further API updates, and no more webhook triggers.

This results in field FLAG = false, on Baserow.

The next time I amend field DATA on this row via the Baserow UI, the webhook will trigger (as normal), however my will test will read FLAG=false, and will NOT process the required data (in my case it will not generate an invoice to a customer).

I have a week-10 days before this project is due to be rolled out into the business operation, so I would be grateful if you could let me know if your discussion come up with a viable work around, or even if there will be a fix ?

Many thanks,
Martin

[Thorby]

********** UPDATE **********
Hi Colin,

I have been looking at my logs and can see the webhook triggers each time the API (update row) is executed, regardless of changes field values!!!

Therefore, I believe I may have a short-term workaround.

Instead of creating a new boolean field on Baserow, and compounding the issue (see previous previous posts), I might be able to implement a routine that compares the items array with the old items array in the webhook.

Using PHP I should be able to utilise the array_diff() function.

for example, once I’ve extracted items array and old items array from the webhook (as arrays) and stored their values, I sould be able to do something like:

if array_diff($items, $old_items) == NULL, then Stop.

I’m not sure if this would help other using differnt codebase, but it may help PHP users.

I would like to add, that in some use-cases this can lead to a huge overhead for me, depending on the type/number of data fields that need processing/updating.

Regards,
Martin

[cwinhall]

Hi Martin,

Is it outside of the workflow use case to just set the flag to true again manually when using the UI? That way the automation via the webhook will get triggered and then set it to false again? You could potentially have 2 flag fields, one for UI update and one for API update. It would give more flexibility but would still require manual change in the UI.

Is it not also a plausible scenario where you can perform a check between old values and new values to determine there has not been any change and if no change then stop the loop? As old values and new values are provided in the webhook payload.

[davide]

Hey @Thorby,

based on my understanding, the main issue is that your code is attempting to update the same field that a user can modify in the UI. There are multiple ways to resolve this, but here are a couple of ideas:

• Save the processed data in a different field instead of the one the user can edit. This way, the second webhook will have the same input data, and assuming your code consistently produces the same output from the same input, the second execution can be skipped.

• If you are unable to store the output in a different field, store the output in both the DATA field and another field. Then, for each webhook, you can check whether the DATA is different from the server-processed data. If it is, then the user has modified the DATA field and the data needs to be processed again; otherwise, no action is necessary.

IMO the first one is cleaner, because you clearly separate user inputs from server generated output.

I hope this helps,
davide

[Thorby]

Many thanks Davide,

I have managed to create a working solution that both eliminates the need for additional fields in Baserow and manual intervention.

NOTE: only tested for [event_type] => rows.updated.

Lets assume:
my target field is DATA.
my local processing of DATA outputs a new DATA value.

When a user changes the value of DATA, Baserow generates a webhook.

On receipt of the webhook I run a compare function on my local server:

function compare()
if webhook payload [items] not equal to webhook payload [old_items]

1. Process DATA on my server,
2. update DATA via Baserow API.
   else
   return HTTP 200

Therefore:
(a) The first time I receive the webhook [items] and [old_items] are different (expected), therefore I process DATA locally, then update DATA via the API

(b) Baserow triggers webhook again.

(c) The second time I receive the webhook [items] and [old_items] are again different (expected as I have updated DATA via API update, as in (a)), therefore I process DATA locally, then update DATA via the API. However, this time I am updating DATA with the same values as in (a).

(d) Baserow triggers webhook again.

(e) The third time I receive the webhook [items] and [old_items] are identical (expected), therefore I return HTTP 200 which stops Baserow from triggering additional webhooks.

As I said before, this is an unnecessary overhead for both Baserow and my server, however, it can act as a short-term workaround that does not require additional database columns or manual intervention.

I am in the process of testing [event_type] => rows.created and will update accordingly.

Again, many thanks for your help.

Martin

[Thorby]

********** UPDATE **********
I have now successfully tested for [event_type]=>rows.created

I have amended my function compare()
if [event_type]=>rows.created

1. Process DATA on my server,
2. update DATA via Baserow API.
   else if [event_type]=>rows.updated
   if webhook payload [items] not equal to webhook payload [old_items]
3. Process DATA on my server,
4. update DATA via Baserow API.
   else
   return HTTP 200

This also works as expected
updating DATA via the API on receipt of [event_type]=> rows.created causes Baserow to generate a webhook with [event_type]=> rows.updated

I can then process [event_type]=> rows.updated as described in the post above and kill the process once the webhook payloads [items] and [old_items] are equal.

Yes, it’s a bit messy with a significant number of unnecessary process on both servers, but hopefully a permenant fix wont be too far around the corner (fingers crossed).