External asset database boefje

[zcrt]

Changes

This pull request is a generalization of https://github.com/minvws/nl-kat-coordination/tree/feature/dadb and introduces an external database boefje (previously DADB), where hostnames, IPs and NetBlocks can be fetched from an external API. See description.md for more details.

Proof

Related, not solved in this PR:

#188
#131

Code Checklist

• All the commits in this PR are properly PGP-signed and verified;
• This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
• I have written unit tests for the changes or fixes I made.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have made corresponding changes to the documentation, if necessary.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[Darwinkel]

Looks great! I'm not sure how to QA test this though, as it requires additional software which implements this API spec server-side. Is there any reference implementation which others can use?

Also, not 100% sure what our policy on unit-testing boefjes is (some, but not all appear to have them), but I'm thinking that it would be nice to have some basic tests for the parsing and yielding.

[ammar92]

I was testing it out but I had the following error with the input you gave me yesterday:

zcrt-normalizer-1  | [2023-06-28 08:16:58 +0000] [7] [INFO] [job_handler] Handling normalizer kat_external_db_normalize[6bf117ae397a46b5a385ae04ff68c969]
zcrt-normalizer-1  | [2023-06-28 08:16:59 +0000] [7] [INFO] [local] Running local normalizer plugin
zcrt-normalizer-1  | [2023-06-28 08:16:59 +0000] [7] [ERROR] [app] An error occurred handling scheduler item 6bf117ae397a46b5a385ae04ff68c969
zcrt-normalizer-1  | Traceback (most recent call last):
zcrt-normalizer-1  |   File "/app/boefjes/boefjes/app.py", line 105, in start_working
zcrt-normalizer-1  |     item_handler.handle(p_item.data)
zcrt-normalizer-1  |   File "/app/boefjes/boefjes/job_handler.py", line 183, in handle
zcrt-normalizer-1  |     results = self.job_runner.run(normalizer_meta, raw)
zcrt-normalizer-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
zcrt-normalizer-1  |   File "/app/boefjes/boefjes/local.py", line 73, in run
zcrt-normalizer-1  |     return self._parse_results(normalizer_meta, results)
zcrt-normalizer-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
zcrt-normalizer-1  |   File "/app/boefjes/boefjes/local.py", line 76, in _parse_results
zcrt-normalizer-1  |     parsed: List[NormalizerResult] = [self._parse(result) for result in results]
zcrt-normalizer-1  |                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
zcrt-normalizer-1  |   File "/app/boefjes/boefjes/local.py", line 76, in <listcomp>
zcrt-normalizer-1  |     parsed: List[NormalizerResult] = [self._parse(result) for result in results]
zcrt-normalizer-1  |                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
zcrt-normalizer-1  |   File "/app/boefjes/boefjes/plugins/kat_external_db/normalize.py", line 38, in run
zcrt-normalizer-1  |     for address_item in follow_path_in_dict(path=IP_ADDRESS_LIST_PATH, path_dict=results):
zcrt-normalizer-1  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
zcrt-normalizer-1  |   File "/app/boefjes/boefjes/plugins/kat_external_db/normalize.py", line 27, in follow_path_in_dict
zcrt-normalizer-1  |     raise KeyError(f"Key {key} not in {list(path_dict.keys())}")
zcrt-normalizer-1  | KeyError: "Key ip_addresses not in ['id', 'boefje_meta', 'mime_types', 'secure_hash', 'signing_provider_url', 'hash_retrieval_link']"

[zcrt]

I was testing it out but I had the following error with the input you gave me yesterday

The file I sent you contains two files (one with metadata and one with IP/domain data). This file was downloaded directly from the KAT interface on a succesfull normalizer run. Can it be that a normalizer download does not translate back to a normalizer input?

[ammar92]

I was testing it out but I had the following error with the input you gave me yesterday

The file I sent you contains two files (one with metadata and one with IP/domain data). This file was downloaded directly from the KAT interface on a succesfull normalizer run. Can it be that a normalizer download does not translate back to a normalizer input?

Ah I figured it out, I accidentally loaded the wrong file. My bad!

Seems to work and ready for merge.

Dismiss