Add Question ooi model and create the first bit that generates a question #1407

Donnype · 2023-07-17T12:57:45Z

Changes

Add Question ooi model and create the first bit that generates a question.

Issue link

(Partially) closes #288

Proof

Code Checklist

All the commits in this PR are properly PGP-signed and verified;
This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
I have written unit tests for the changes or fixes I made.
For any non-trivial functionality, I have added integration and/or end-to-end tests.
I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
I have made corresponding changes to the documentation, if necessary.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

… and answer field Signed-off-by: Donny Peeters <[email protected]>

…to include in ooi_detail.html

Submit json data from form in frontend for json schema form. Style update with css-based indentation. JS fixes: no identifier, remove input names to ignore them on submit and add "less" button.

…n right normalizer

ammar92

Just a few remarks, but looks good to me 👍

ammar92 · 2023-07-18T08:26:43Z

octopoes/bits/ask_port_specification/ask_port_specification.py

+    with (Path(__file__).parent / "question_schema.json").open() as f:
+        schema = json.load(f)


I was wondering. If this schema is not expected to change during a worker's lifetime or (statically) shipped with a bit, then you might want to do this reading outside of the function. The performance gain is obviously not significant, but it would make more sense if this schema variable is static.

It is quite nice to get the feedback of the changes without restarting Octopoes though

Yes agreed, but it's unlikely that this file be altered during development or production time right? And if it's changed during development, restarting shouldn't take that long.

Loading the file also doesn't take too long of course. And if there are issues with the file, perhaps we prefer a bit runtime error instead of an import-time error?

I prefer not to do unnecessary IO calls for things that never change. Changing bit (or plugins in general) code or input should always yield a newer version of that plugin.

Runtime errors are worse because they're much harder to catch. In case of the import-time error, that would mean someone developing an invalid plugin build which of course will be caught during a review.

Shoudn't we just put the schema in the questionOOI? As this would allow us to also support other questions in our next iterations of this functionality.

That QuestionOOI would then in turn need to be created by another bit which we can run on each 'network' being created. The flow would then be:
Bit: For each network, there must be a Ports configOOI, of not, create a QuestionOOI with the jsonschema from this PR. The Form is rendered using the schema value from this QuestionOOI.
The input on this form is then ingested by a normlizer specific for this schema, and creates the configOOI, which then also signals the question to be removed.

Bit: The config is used by the OpenPorts bit(s) to see if there are any issues.

Shoudn't we just put the schema in the questionOOI? As this would allow us to also support other questions in our next iterations of this functionality.

@underdarknl We are putting the schema in the QuestionOOI? We can just add new bits that parse different schema files or parse other schema files in this bit if we want to?

That QuestionOOI would then in turn need to be created by another bit which we can run on each 'network' being created. The flow would then be: Bit: For each network, there must be a Ports configOOI, of not, create a QuestionOOI with the jsonschema from this PR. The Form is rendered using the schema value from this QuestionOOI. The input on this form is then ingested by a normlizer specific for this schema, and creates the configOOI, which then also signals the question to be removed.

Bit: The config is used by the OpenPorts bit(s) to see if there are any issues.

I am not sure I understand. The bits don't do checks such as "if X does not exist yet", instead they are mostly idempotent which makes sense as we can manually retrigger them in the interface as well. The current QuestionOOI natural key setup means that is not an issue.

The flow you describe is also visualized in the issue and this flow is implemented in the subsequent PRs, the only difference being that I made the normalizer generic for now. Or do you see differences?

@ammar92 If we load all schema's for bits, one can also argue that we are leaving a memory footprint for a bit that runs only when we add a network (=almost never)?

The idea of loading schema's for future use is not relevant if you ask me, Its like asking us if we want to pre-load the values of other OOI's too.

octopoes/octopoes/models/ooi/question.py

octopoes/bits/ask_port_specification/question_schema.json

…ig OOI

…-kat-coordination into feature/add-question-ooi-model

Darwinkel · 2023-07-21T12:52:02Z

Checklist for QA:

I have checked out this branch, and successfully ran a fresh make reset.
I confirmed that there are no unintended functional regressions in this branch:
- I have managed to pass the onboarding flow
- Objects and Findings are created properly
- Tasks are created and completed properly
I confirmed that the PR's advertised feature or hotfix works as intended.

What works:

No apparent regressions
A new Question OOI is created when a network is added with clearance > L2

* main: (95 commits) Translations for release 1.11 - EN -> NL, PAP (#1439) Add Question ooi model and create the first bit that generates a question (#1407) make port classification configurable (#1418) KATalogus API filtering and pagination (#1405) Fix robot test (#1420) Use the correct clearance level variable in organization member list template (#1427) Fix translation in Debian package (#1432) Reschedule tasks when no results in bytes are found after grace period (#1410) Don't scan hostname nmap in nmap boefje (#1415) Add and use our own CVE API (#1383) Add `task_id` as a query parameter to the `GET /origins` endpoint (#1414) Remove member group checks and check for permission instead (#1275) Bump cryptography from 41.0.0 to 41.0.2 in /boefjes/boefjes/plugins/kat_ssl_certificates (#1396) Bump cryptography from 41.0.1 to 41.0.2 in /bytes (#1397) Build the Debian build image on the main branch (#1387) Add explicit `black` config to all modules (#1395) Fix <no title> in the user guide docs (#1391) Add configurable octpoes request timeout (#1382) Remove hardcoded clearance level in member list for superusers (#1390) Add Debian build depends for CVE API package (#1384) ...

Donnype added 13 commits July 17, 2023 14:50

First sketch of a FormQuestionAnswer OOI and validation on the schema…

ce064e2

… and answer field Signed-off-by: Donny Peeters <[email protected]>

First experiments with questions and form rendering.

baff1d2

More setup work for question ooi

94feb45

Removed specialization and add 0 as min-severity for Questions

d429784

Cleanup: remove dead forms code and add reverse relation name for oois

dd8e7ee

Use bit_id in schema id for now

6235c7c

Merge branch 'main' into feature/question-answer-ooi

5a79190

Remove json schema field max length and add json schema form partial …

0b0e72e

…to include in ooi_detail.html

Add missing json file

5d797c7

Merge fixes.

047cdc6

Submit json data from form in frontend for json schema form. Style update with css-based indentation. JS fixes: no identifier, remove input names to ignore them on submit and add "less" button.

Fix javascript type parsing, submit to bytes and use mime-types to ru…

7ed9b7c

…n right normalizer

Cleanup

d74b98f

Revert config dict field type annotation change

129c295

Donnype requested a review from a team as a code owner July 17, 2023 12:57

Donnype and others added 2 commits July 17, 2023 15:25

Cleanup question schema to use a comma separated list

29d63f2

Fix open call

7be7910

Donnype changed the title ~~Add Question ooi model and create the first bit that generates a question.~~ Add Question ooi model and create the first bit that generates a question Jul 17, 2023

Donnype mentioned this pull request Jul 17, 2023

Add Question OOI form rendering on the object detail page #1408

Merged

7 tasks

ammar92 previously approved these changes Jul 18, 2023

View reviewed changes

praseodym reviewed Jul 18, 2023

View reviewed changes

octopoes/bits/ask_port_specification/question_schema.json Outdated Show resolved Hide resolved

Donnype and others added 5 commits July 19, 2023 10:38

Merge branch 'main' into feature/add-question-ooi-model

af32931

Merge branch 'main' into feature/add-question-ooi-model

169232d

Merge branch 'main' into feature/add-question-ooi-model

c8d79a0

Update json schema and bit id according to the new bit using the Conf…

129e980

…ig OOI

Merge branch 'feature/add-question-ooi-model' of github.com:minvws/nl…

4759677

…-kat-coordination into feature/add-question-ooi-model

Donnype dismissed ammar92’s stale review via 4759677 July 20, 2023 10:05

Donnype requested review from underdarknl and ammar92 July 20, 2023 10:38

Merge branch 'main' into feature/add-question-ooi-model

7bf7c4e

Darwinkel mentioned this pull request Jul 24, 2023

make port classification configurable #1418

Merged

7 tasks

Merge branch 'main' into feature/add-question-ooi-model

cf0e95a

dekkers approved these changes Jul 25, 2023

View reviewed changes

dekkers merged commit 1e4f2ab into main Jul 25, 2023

dekkers deleted the feature/add-question-ooi-model branch July 25, 2023 12:20

Donnype mentioned this pull request Jul 27, 2023

Documentation about question OOIs #1471

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Question ooi model and create the first bit that generates a question #1407

Add Question ooi model and create the first bit that generates a question #1407

Donnype commented Jul 17, 2023

ammar92 left a comment

ammar92 Jul 18, 2023

Donnype Jul 18, 2023

ammar92 Jul 18, 2023

Donnype Jul 19, 2023

ammar92 Jul 19, 2023

underdarknl Jul 19, 2023

underdarknl Jul 19, 2023

Donnype Jul 19, 2023 •

edited

Loading

Donnype Jul 19, 2023

underdarknl Jul 24, 2023

Darwinkel commented Jul 21, 2023

		with (Path(__file__).parent / "question_schema.json").open() as f:
		schema = json.load(f)

Add Question ooi model and create the first bit that generates a question #1407

Add Question ooi model and create the first bit that generates a question #1407

Conversation

Donnype commented Jul 17, 2023

Changes

Issue link

Proof

Code Checklist

Communication

Checklist for code reviewers:

Checklist for QA:

ammar92 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Donnype Jul 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Darwinkel commented Jul 21, 2023

Checklist for QA:

What works:

Donnype Jul 19, 2023 •

edited

Loading