Custom field types: I need a simple explanation, what are get_serializer_field and get_model_field?

Technical Help
1

In my plugin, i’m introducing a new field type for which I want a complex storage type, it won’t be a string, it’ll be a list of possible choices, and the user will select one among the possible choices. The background is for my language learning spreadsheet, i’m adding a “chinese romanization” field type which allows going from chinese text to romanized (latin alphabet). However there are multiple such possible transformations, and the user will be allowed to pick one.
For this reason, storing the data as JSON makes sense.

Here’s what I have for my model:

class ChineseRomanizationField(Field):
    source_field = models.ForeignKey(
        LanguageField,
        on_delete=models.CASCADE,
        help_text="The field to transliterate.",
        null=True,
        blank=True,
        related_name='+'
    )    
    correction_table = models.ForeignKey(
        "database.Table",
        on_delete=models.CASCADE,
        help_text="The correction table for pinyin/jyutping",
        null=True,
        blank=True,
    )
    transformation = models.CharField(
        max_length=64,
        default=CHOICE_PINYIN,
        choices=CHINESE_ROMANIZATION_CHOICES,
        help_text="Pinyin or Jyutping",
    )
    tone_numbers = models.BooleanField(
        default=False,
        help_text="Whether to use tone numbers in pinyin/jyutping",
    )
    spaces = models.BooleanField(
        default=False,
        help_text="Whether to use space between each syllable",
    )

And here’s the FieldType:

class ChineseRomanizationFieldType(TransformationFieldType):
    type = "chinese_romanization"
    model_class = ChineseRomanizationField
    allowed_fields = [
        'source_field_id',
        'correction_table_id',
        'transformation',
        'tone_numbers',
        'spaces'
    ]

    can_be_primary_field = False

    def prepare_value_for_db(self, instance, value):
        return value

    # def get_serializer_field(self, instance, **kwargs):
    #     return serializers.JSONField(**kwargs)

    # def get_model_field(self, instance, **kwargs):
    #     return models.JSONField(null=True, blank=True, default={}, **kwargs)

    def get_serializer_field(self, instance, **kwargs):
        return serializers.JSONField(**kwargs)

    def get_model_field(self, instance, **kwargs):
        pprint.pprint(kwargs)
        return ChineseRomanizationField(
            default=None,
            blank=True, 
            null=True, 
            **kwargs
        )


    def transform_value(self, field, source_value, usage_user_id):
        if field.transformation == CHOICE_PINYIN:
            return clt_interface.get_pinyin(source_value, field.tone_numbers, field.spaces)
        elif field.transformation == CHOICE_JYUTPING:
            return clt_interface.get_jyutping(source_value, field.tone_numbers, field.spaces)

    def row_of_dependency_updated(
        self,
        field,
        starting_row,
        update_collector,
        field_cache: "FieldCache",
        via_path_to_starting_table,
    ):

        self.process_transformation(field, starting_row)

        ViewHandler().field_value_updated(field)     

        super().row_of_dependency_updated(
            field,
            starting_row,
            update_collector,
            field_cache,
            via_path_to_starting_table,
        )        


    def update_all_rows(self, field):
        # TODO write this
        pass

Here’s the issue: when I try to create a field of type chinese_romanization (in a regression test), I get the following error:

    # add pinyin field
    # ================

    response = api_client.post(
        reverse("api:database:fields:list", kwargs={"table_id": table_id}),
        {
            "name": "pinyin", 
            "type": "chinese_romanization", 
            "source_field_id": chinese_field_id, 
            'tone_numbers': False,
            'spaces': False,
        },
        format="json",
        HTTP_AUTHORIZATION=f"JWT {token}",
    )
    response_json = response.json()
    pprint.pprint(response_json)
    assert response.status_code == HTTP_200_OK

The error message:

Traceback (most recent call last):
  File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/baserow/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/sentry_sdk/integrations/django/views.py", line 67, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/baserow/venv/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/baserow/backend/src/baserow/api/decorators.py", line 280, in func_wrapper
    return func(*args, **kwargs)
  File "/baserow/backend/src/baserow/api/decorators.py", line 90, in func_wrapper
    return func(*args, **kwargs)
  File "/baserow/backend/src/baserow/contrib/database/api/fields/views.py", line 282, in post
    field, updated_fields = action_type_registry.get_by_type(
  File "/baserow/backend/src/baserow/contrib/database/fields/actions.py", line 335, in do
    result = FieldHandler().create_field(
  File "/baserow/backend/src/baserow/contrib/database/fields/handler.py", line 305, in create_field
    to_model = instance.table.get_model(field_ids=[], fields=[instance])
  File "/baserow/backend/src/baserow/contrib/database/table/models.py", line 547, in get_model
    field_attrs = self._fetch_and_generate_field_attrs(
  File "/baserow/backend/src/baserow/contrib/database/table/models.py", line 699, in _fetch_and_generate_field_attrs
    field_attrs[field_name] = field_type.get_model_field(
  File "/baserow/data/plugins/baserow_vocabai_plugin/backend/src/baserow_vocabai_plugin/fields/vocabai_fieldtypes.py", line 523, in get_model_field
    return ChineseRomanizationField(
  File "/baserow/backend/src/baserow/core/mixins.py", line 76, in __init__
    super().__init__(*args, **kwargs)
  File "/baserow/venv/lib/python3.9/site-packages/django/db/models/base.py", line 503, in __init__
    raise TypeError("%s() got an unexpected keyword argument '%s'" % (cls.__name__, kwarg))
TypeError: ChineseRomanizationField() got an unexpected keyword argument 'default'

My question: do I have the get_serializer_field and get_model_field methods configured properly on my fieldtype ? I’ll be honest, I don’t understand what they correspond to.
The previous field types that I created are similar and they don’t have that issue (baserow-vocabai-plugin/vocabai_fieldtypes.py at main · Language-Tools/baserow-vocabai-plugin · GitHub)

1 Like
2

Hey @lucw,

So one thing that might help with all of these FieldType methods is looking at the docstrings found in backend/src/baserow/contrib/database/fields/registries.py · develop · Bram Wiepjes / baserow · GitLab.

Additionally we have a bunch of field type examples found in backend/src/baserow/contrib/database/fields/field_types.py · develop · Bram Wiepjes / baserow · GitLab that might be useful to reference.

But in summary:
get_serializer_field This is used to generate a Django Rest Framework serializer specific to your new field type. So when POST/PATCH/GETing our field endpoints with your new field type this serializer will be used.

get_model_field - This is used when building a literal Django Model, you can imagine what we are doing under the hood when a user adds a field of this new type to a table as literally building and creating the following Django model:

class SomeUserCreatedTableInBaserowModel10(models.Model):
    field_10 = ChineseRomanizationFieldType.get_model_field()

So get_model_field needs to return a Django Model Field like models.TextField or models.CharField etc. Instead it looks like you are returning not a Django Model Field, but instead (and this is all rather confusing and badly named its not any problem with you) you are returning ChineseRomanizationField which is an actual Django Model and not a Django Model Field which Baserow will use to store metadata about your field.

TDLR: get_model_field returns the model field that we use to literally run the ALTER TABLE ADD FIELD my_new_field. Your ChineseRomanizationField(Field) model is instead a table in that actual database called say my_plugin_chinseromanizationfield which has columns describing the metadata for your field that Baserow then uses.

I hope this helps please let me know if I should go deeper into something or explain it a different way. Also if you ever would like some sort of plugin code review I’d love to help and hopefully give some more tips after looking at the actual code.

3

Hi @nigel , many thanks for your amazing explanation as always. I’m working on a new feature for my project, which involves inter-table dependencies so it’s going to be more complex than what I did before. I would love a code review, but even before I get there, I’ll have questions along the way.

Do you offer paid consulting services ? If you have interest, I’d love to get on a call to discuss the project to put me on the right path for implementation. If you’d rather we use forum posts that’s fine with me, I can explain my idea that way also.

edit: I got my basic example working, so I think i’m on the right track. Many thanks again !!

1 Like
4

No problem glad to hear it.

Would love to jump on a call to help you out, feel free to send over a meeting invite to nigel@baserow.io :slight_smile: I’m GMT timezone so any weekday working time should work out fine.

We do have a field dependency system used by formula, lookup and link row fields already in Baserow. It needs a bit of TLC so might be confusing to work with atm, but essentially we have model fields in a graph of dependencies already for formulas which perhaps you could also use.