Create storage for boefje definitions

[Donnype]

See the epic: #2736

TODO:

• Phase out the KATalogus Repository Model #2983
• Introduce the Boefje and Normalizer models in the KATalogus #2997
• Migrate the Setting model to a BoefjeConfig model and Introduce a NormalizerConfig model #3014
• Handle json schema's for copied boefjes #3052
• Improve env/settings/allowlist configuration logic for the boefjes #3356

Context

Creating/copying over boefjes in Rocky means we need to save such copies in the form of new boefje definitions.

Approach

This might be a good point to revise some of the KATalogus database models, in particular the PluginEnabled store that saves a PluginState and the Repository. The first is not the cleanest model, both in the domain and database, because this is not really a core concept we can extend and talk about. We do talk about Plugins a lot. Also with the new oci fields implementation for OCI images, we are seeing that boefje images in general will probably be uniquely identified by just a URL, that determines the API as well (e.g. Docker can pull OCI images from all OCI registries). This means that the Respository model is probably a bit redundant (or at least could be renamed to Registry).

Therefore my proposal is as follows:

Change:

classDiagram

class organisation {
   id
   name
   pk
}
class organisation_repository {
   organisation_pk
   repository_pk
}
class plugin_state {
   plugin_id
   enabled
   organisation_pk
   repository_pk
   id
}
class repository {
   id
   name
   base_url
   pk
}
class settings {
   values
   plugin_id
   organisation_pk
   id
}

organisation_repository  -->  organisation
organisation_repository  -->  repository
plugin_state  -->  organisation
plugin_state  -->  repository
settings  -->  organisation

Loading

Into:

classDiagram

class organisation {
  pk
  id
  name
}

class boefje_config {
    id
    settings
    enabled
    boefje_id
    organisation_pk
}

boefje_config --> organisation
boefje_config --> boefje

class normalizer_config {
    id
    enabled
    normalizer_id
    organisation_pk
}

normalizer_config --> organisation
normalizer_config --> normalizer

class boefje {
    id
    plugin_id
    scan_level
    consumes
    produces
    version
    created
    description
    name
    environment_keys
    oci_image
    oci_arguments

    organisations
}

class normalizer {
    id
    plugin_id
    scan_level
    consumes
    produces
    version
    created
    description
    name
    environment_keys

    organisations
}

Loading

(Where I removed the related and options fields, as I am not sure what they do and we never used them in the first place.)

Also see this thread post for a boefje/normalizer models vs. one generic plugin model discussion. The reason to separate the models would be to be able to freely develop both schema's and not use the produces and consumes for multiple formats (mime-types and OOI types) and be able to move e.g. the settings JSON into the many-to-many mapping between boefjes and organisations without giving normalizers an empty settins JSON always.

Benefits

This would turn the new functionality of creating a new Boefje from an image into simple CRUD calls to the KATalogus API for the boefje model. It also incidentally opens up the possibility to add remote boefjes: just add an entry with an oci_image pointing to an external repository. (Note that this current feature does not focus on the latter, but nonetheless is a nice addition from an API and/or DB perspective.) The settings are also now incorporated in the boefje-organization relation together with the enabled option, which collects all boefje-organization specific parameters in one place. Having no normalizer/boefje_config model would be equivalent to having it with enabled=False. We cannot drop the field in favor of the enabled field with this model because we would loose the sensitive settings field every time we'd disable the boefje.

Other consequences

Since we have local json files that contain the boefje/normalizer entity data, the models would somehow act as a mirror to these. But I believe that is an issue we already plan to face in the future when we have to register remote images, which makes the local files something that needs to be seeded in the database upon installation/startup.