Feature/boefje normalizer config models

[Donnype]

Disclaimer

Forgive me father, for I have sinned: I did refactoring in my feature branch. Although I think refactoring is often a crucial part of a feature, in this case these were a bit unrelated. Nevertheless, note that I moved the test-directory from the katalogus dir to the central boefjes directory for consistency with the other services (and the fact that we don't have that many test files), and the same holds for the dependencies directory which contains dependencies not strictly related to the KATalogus API. Some are used by the boefjes workers as well. Just don't let the amount of files changed distract you: most were just moved.

Changes

This feature adds the config models in this overview (as part of this epic):

classDiagram

class organisation {
  pk
  id
  name
}

class boefje_config {
    id
    settings
    enabled
    boefje_id
    organisation_pk
}

boefje_config --> organisation
boefje_config --> boefje

class normalizer_config {
    id
    enabled
    normalizer_id
    organisation_pk
}

normalizer_config --> organisation
normalizer_config --> normalizer

class boefje {
    id
    plugin_id
    scan_level
    consumes
    produces
    version
    created
    description
    name
    environment_keys
    oci_image
    oci_arguments

    organisations
}

class normalizer {
    id
    plugin_id
    scan_level
    consumes
    produces
    version
    created
    description
    name
    environment_keys

    organisations
}

Loading

These changes should be backward compatible, as has been tested with migration tests. The API is left intact as well.

Issue link

Closes #3014

QA notes

Please verify that the KATalogus works properly, in terms of enabling/disabling and changing/updating/copying settings. Also please check that upgrading to this branch does not disable previously disabled plugins or delete your settings. That shouldn't happen.

---

Code Checklist

• All the commits in this PR are properly PGP-signed and verified.
• This PR only contains functionality relevant to the issue.
• I have written unit tests for the changes or fixes I made.
• I have checked the documentation and made changes where necessary.
• I have performed a self-review of my code and refactored it to the best of my abilities.

• Tickets have been created for newly discovered issues.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[ammar92]

Looks good to me in general, but I didn't dive into it too deeply because of its vastness.

Overal refactoring (and move of the katalogus module tests) seemed necessary and the tests look good.

[stephanie0x00]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make reset.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.
• I checked the logs for errors and/or warnings and made issues where necessary

What works:

Looks good. Enabled all boefjes that do not require API keys and/or additional settings and they all run like a charm. Some normalizers throw errors, but those are different issues. Reports can be generated, all the links on the Katalogus details page are present and work.

What doesn't work:

n/a

Bug or feature?:

n/a