📰 Community digest: contributions, AI in Baserow?, ambassador program, and much more

olgatrykush · May 26, 2023, 8:21pm

Hello hello!

New month, new digest. Here’s everything you need to know to stay up to date on all things Baserow:

Community spotlight

First and foremost, we would like to thank our community members who have been actively sharing their feature ideas and feedback with us over the past month. @dynnammo, @frederikdc, @computersrmyfriends, @Harsh, @verifybudi, @deet, @LumenYang @ESMP, @bfranklin, @guiolmar, @Alexander, and @philipcristiano: your input and contributions mean the world to us. Continue with your activity.

Here are some cool new things our community created:

→ @frederikdc posted insights on how to use your browser inspector to detect API endpoints and payload in Baserow.
→ @lucw open sourced his plugin that adds complex custom field types to Baserow.
→ @nigel shared a tip on how to increase your Baserow’s performance in 1.17.2 using SQL caching.
→ Elestio did a video overview about Baserow.

News and announcements from the team

Release 1.17

We’ve released Baserow 1.17, and now sorted tables with 5 million rows are up to 100x faster, resulting in a smoother experience. Table load time, scrolling, sorting, filtering, and editing on Baserow.io is up to 3x faster. We also added new features to Calendar view and ARM Docker images.

Have you noticed the difference in Baserow’s speed? Let us know in the comments (our developers would be extremely happy to hear your feedback).

Speed improvements overview by @bram

As we already mentioned, Baserow now is up to 3x faster thanks to a long list of under-the-hood upgrades. In this video, our co-founder and CEO, @bram, delves into what changes and improvements we made. Give Bram some feedback in the comments regarding how you like this type of video

Baserow AI: what value does it bring to YOU?

To ensure that Baserow delivers real value with AI, we want to hear directly from you. How do you envision AI working in Baserow? Where do you see it being most useful?

Please fill out this 2-question form to help us determine the best application of AI in Baserow.

Ambassador program

Are you interested in becoming a Baserow Ambassador to help educate others on how to get the most out of Baserow?

Sounds interesting.
Too busy to take it on.
Need more details.

0 voters

We’re curious to see how you feel about this possible initiative

Collab with AppSmith

We partnered with AppSmith to build a PDF generator app, so you can create PDF docs with the Baserow data in minutes.

Start building your first PDF files from the data you have in Baserow data right now.

Leave us a review on Capterra

Help us to stand out in the crowd of SaaS tools listed on Capterra, and give others the encouragement they need to start using Baserow. We’d appreciate if you leave us a review.

Baserow in Mallorca

As a remote-only company without offices, we love meeting in person and spending quality time together. We spent the first week of May in Mallorca to get to know each other, connect, and exchange ideas. These were wonderful days of fun activities and lively conversations.

Baserow is hiring full-stack developers! Apply here.

Other useful resources to check out

Listen to a podcast with our co-founder and CRO, Olivier Maes, about Baserow leveraging open source for creating scalable no-code database. (Heads up: the podcast is in French )
Check out how Baserow compares to Airtable pricing.
Discover how to generate unique IDs with Baserow formulas.
Check out a new workflow on how to create new rows when Amazon Alexa trigger phrases are spoken.
Learn how to sync Google Sheets and Baserow with Zapier.
Figure out how to use Baserow data in your Appsmith application.

That’s it for today! Have a great weekend and take care

Warm regards,
Olga

lucw · May 27, 2023, 12:07am

Thanks for the continued focus on performance @bram it’s very appreciated.

As far as baserow + AI, the most obvious use case in my opinion is to run chatbot prompts based on field data. Here’s what it would work:

Field A contains a paragraph of text.
Field B could be added as a derived field , meant to contain the result of a chatbot API call on prompt “${prompt} ${field_data}”.

So what can the baserow team do to help facilitate this kind of workflow ? It might be tempting to hardcode calls to certain APIs such as OpenAI and support them directly out of the box on Baserow (you’d have an OpenAI ChatGPT field type for example, the user would supply their own API key). And it might still be the right thing to do from a marketing, exposure point of view for Baserow. (and it might be the perfect use case for what i’m describing below)

But what I would really like as a developer is an actual framework for running derived field calculations in background threads. Because in my project, the derived fields will call other APIs such as translation, dictionary lookup, which take 300ms-2000ms to respond.

Here’s what I’d love to see (and i’m 100% open to doing some of the work myself or somehow financially sponsoring some of the development with my meager funds)

the FieldType base class will have support for “long running computations”
on row update, the derived field would be amended at a later time (using one of the celery workers), to avoid blocking the HTTP PATCH call for long periods of time (particularly useful if you’re running a GPT-4 query).
very importantly, there would be support for adding a new field on an already populated table. Let’s say I have 100 rows of data. I add a OpenAI GPT field. It has to run on 100 rows. A celery task should be started which periodically updates the table data.
care should be taken when adding a new such derived field on a massive table. Let’s say we have 10k rows. Do we want to let the user casually add a GPT4 field ? First of all, it’ll block one of the celery workers for a long time, second, it’ll quickly exhaust the user’s OpenAI credits without them necessarily knowing about it.
There should be a “retry” button in case one of the query failed, or maybe the user is not satisfied with the output, they could quickly retry and get a different result.
cherry on top of the cake, there could be a visual progress indicator telling the user that background tasks are running.

I’ve tried to tackle these issues in my plugin. @nigel has been extremely helpful in pointing me in the right direction and writing some of the code. I rolled out some solutions which work for me, but I suspect sooner or later, other people will want the same functionality for their own derived fields.

Here’s how I do it:
my field type, the call to update all rows:

github.com

Language-Tools/baserow-vocabai-plugin/blob/main/plugins/baserow_vocabai_plugin/backend/src/baserow_vocabai_plugin/fields/vocabai_fieldtypes.py#L269

      
        
                    source_field_language = field.source_field.language
                    target_language = field.target_language
                    translation_service = field.service          
                    source_field_id = f'field_{field.source_field.id}'
                    target_field_id = f'field_{field.id}'
            
            
        table_id = field.table.id
            
            
        logger.info(f'after_update table_id: {table_id} source_field_id: {source_field_id} target_field_id: {target_field_id}')
            
            
        run_clt_translation_all_rows.delay(table_id, 
                                                       source_field_language, 
                                                       target_language,
                                                       translation_service,
                                                       source_field_id, 
                                                       target_field_id,
                                                       self.get_usage_user_id(field))
            
            

            
class TransliterationFieldType(TransformationFieldType):
                type = "transliteration"

the corresponding celery task:

github.com

Language-Tools/baserow-vocabai-plugin/blob/main/plugins/baserow_vocabai_plugin/backend/src/baserow_vocabai_plugin/cloudlanguagetools/tasks.py#L119

      
        
            # noinspection PyUnusedLocal
            @app.task(
                bind=True,
                soft_time_limit=EXPORT_SOFT_TIME_LIMIT,
                time_limit=EXPORT_TIME_LIMIT,
            )
            def run_clt_translation_all_rows(self, table_id, source_language, target_language, service, source_field_id, target_field_id, usage_user_id):
                # populating all rows is still a single celery task, but we break it up so that we can notify the user
                # about work in progress
            
            
    try:
                    for row_id_list in iterate_row_id_buckets(table_id):
                        for row in process_row_id_bucket_iterate_rows(table_id, row_id_list):
                            text = getattr(row, source_field_id)
                            if text != None and len(text) > 0:
                                translated_text = clt_interface.get_translation(text, source_language, target_language, service, usage_user_id)
                                setattr(row, target_field_id, translated_text)
                                row.save()
                except QuotaOverUsage:
                    logger.exception(f'could not complete translation for user {usage_user_id}')

I don’t have an answer for the following questions:

what happens when a large number of users add derived fields on massive tables, clogging up the celery tasks ?
how to give a warning that adding derived fields on massive table will exhaust the user’s credits.
how to retry (for example in case of a timeout).

I suspect having support for these long-running calculations will open up a ton of other use cases. For example at work (financial industry), we have a spreadsheet-like app to do pricing on derivative products. We could replace with with a custom baserow instance which calls the pricing code once all pricing parameters are configured. Baserow would be a much much nicer UI than what we have right now.

Alex · May 30, 2023, 7:34am

It might be tempting to hardcode calls to certain APIs such as OpenAI

There are plans in the not so far away future to support connecting Baserow to APIs and building the infrastructure to support authentication with those third party apis.

This wasn’t necessarily related to the Baserow tables but could still be useful here if we chose to build a more generic field type that essentially let's you call an api given some data from another field type. We could even go as far as making this possible in formulas so that you have much more flexibility in what that request might look like. Formulas are also calculated in the backend so we wouldn’t need to worry about having to add that.

Not sure how feasible the formula idea is ultimately, there might be caveats that make it impossible that I am not aware of

bram · May 30, 2023, 10:09am

Hi @lucw, I really like the idea of having fields that are slow in updating cell values, and might even depend on an external service to do so. @peter_baserow currently working on improving the search performance, and this actually works similar. Here we’ve introduced a tsvector search column in all tables, which basically holds the human-readable format for search queries. If a row has been changed, either directly, or via a formula update, it will debounce a celery task to actually update that column. The code can be found here: Draft: Resolve "Implement Postgres full-text search" (!1390) · Merge requests · Baserow / baserow · GitLab. However, if we want to implement slow field types properly, we also need to inform the user when something is being updated. The UX is important here.

lucw · May 30, 2023, 2:13pm

@Alex in fact I had some discussions with @nigel early on about my project and one of the suggestions was to implement the field transformations in the formula framework. However because those run inside postgres (in my understanding), and my logic was more complicated, I decided to just create new field types and do slow tasks in celery.

@bram thanks for pointing me to this merge request, I will take a look at it. I agree with you having a loading indication will be important for the user. Some of the LLM APis are particularly slow. ChatGPT4 is quite slow.
I actually need to work on a special marketing workflow for my customers where I look at their activity and generate a custom personalized GPT4 email. I want to do it with minimal coding and I want to use Baserow. Right now I might need to wire everything with a tool like n8n or some python. It would make sense for me to have more of this processing inside a baserow plugin. I can tell you that many other people out there are looking to integrate their CRMs with chatbots to automate reaching out to leads, etc.

The good news is that the plugin support in baserow is a good playground to experiment. I’m actually working on a minimal “translate” plugin showing how language translation can be integrated into baserow (which will be subject of a blog post on the baserow blog).
The minimal code required looks like this:

the field type: baserow-translate-plugin/field_types.py at master · Language-Tools/baserow-translate-plugin · GitHub
the celery task to mass-update all rows when the field is updated: baserow-translate-plugin/translation.py at master · Language-Tools/baserow-translate-plugin · GitHub

The approach is the same whether calling a language translation API or a chatbot API (or any other REST API / heavy python computation).