Question OOIs with json-schema

[Donnype]

See #130
Draft PR: #1394

Some considerations:

• We want questions to be formulated by json-schemas and answers as json
• In the frontend we can create forms from json-schema's and json from form submissions
• We might need to generate id's for questions Unique Questions per OOI are probably fine?
• Schema's are defined in Bits
• Per answer, the right Config needs to be created. Determining which Config can be done through:
  • The mime types, by adding an id per schema (best practice for json-schema's) and triggering the right normalizer. However, this would also be communicating the bit_id for Configs, so we are somewhat abusing notation here.
  • Adding a identifier field in the answer
  • Adding an argument (but I'm not a fan of the mostly unused arguments field, see Consider removing the arguments field of BoefjeMeta #763)
  • Try to infer it through the input_ooi This will not work if we start attaching multiple configs to a Network.

Still, Rocky (saving the raw file) needs to know what the "type" of Question is, so somehow the Question object (either as a field or in its schema) needs to carry an identifier.

Questions:

• Do we need to add a new page with an overview? Quickly seeing unanswered question would be nice I'd say.

Config flow and cardinality:

classDiagram

direction RL
    class Question
    class Config
    class JsonAnswer
    class JsonAnswer2["JsonAnswer"]

    Question : ooi[OOI]
    Question : schema[str]

    Config : ooi[OOI]
    Config : config[str]
    Config : bit_id[str]

    OOI <|-- Question
    Question <|-- JsonAnswer
    Question <|-- JsonAnswer2
    JsonAnswer <|-- Config
    JsonAnswer2 <|-- Config

Loading

Resulting ERD:

classDiagram

direction RL
    class Question
    class Config
    class OOI

    Question : ooi[OOI]
    Question : schema[str]

    Config : ooi[OOI]
    Config : config[str]
    Config : bit_id[str]

    OOI <|-- Question
    OOI <|-- Config

Loading

The Flow

sequenceDiagram
    participant User
    participant Rocky
    participant Normalizer
    participant Octopoes
    participant Bits
    participant Bytes

    Normalizer->>Octopoes: Add Network
    Bits->>Octopoes: Add Question["What ports may be open for this Network?"]
    Rocky->>Octopoes: Get Question
    Rocky->>User: Prompt Question to user
    User->>Rocky: Give answer (form)
    Rocky->>Bytes: Add answer (json) to Bytes
    Bytes->>Normalizer: Read answer
    Normalizer->>Octopoes: Create Config
    Bits->>Octopoes: Read Config

Loading

---

Generating Question (forms)

To create question OOI's:

• Rebuild Rocky and the boefjes/normalizers (Or: make kat)
• Add a Network object
• Increase its clearance level to at least L2
• Wait a few seconds and refresh the objects page (perhaps filter on Question objects)
• Open the Question detail page and see the json form rendered.

[underdarknl]

For our first usecase we will create a bit that does the following:

For each network (currently only the Internet is added to the graph regularly) trigger a bit.
In that bit create a QuestionOOI with the given schema as defined in that BIT as the payload.

{
    "$schema": "https://json-schema.org/draft/2019-09/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "default": {},
    "Port Configuration": "Root Schema",
    "required": [],
    "properties": {
        "common": {
            "type": "object",
            "default": {},
            "Common Ports": "Allowed ports",
            "required": [],
            "properties": {
                "udp": {
                    "title": "Allowed UDP ports",
                    "type": "array",
                    "default": [
                        53
                    ],
                    "items": {
                        "type": "integer",
                        "examples": [
                            53
                        ]
                    }
                },
                "tcp": {
                    "type": "array",
                    "default": [25, 53,
                            110,
                            143,
                            993,
                            995,
                            80,
                            443],
                    "title": "Allowed TCP ports",
                    "items": {
                        "type": "integer",
                        "examples": [
                            25,
                            53,
                            110,
                            143,
                            993,
                            995,
                            80,
                            443
                        ]
                    }                    
                }
            }
        },
        
        "admin": {
            "type": "object",
            "default": {},
            "Common Ports": "Disallowed admin ports",
            "required": [],
            "properties": {
                "udp": {
                    "title": "Disallowed UDP ports",
                    "type": "array",
                    "default": [
                        53
                    ],
                    "items": {
                        "type": "integer",
                        "examples": [
                            53
                        ]
                    }
                },
                "tcp": {
                    "type": "array",
                    "default": [21,
                            22,
                            23,
                            3389,
                            5900
                        ],
                    "title": "Disallowed TCP ports",
                    "items": {
                        "type": "integer",
                        "examples": [
                            21,
                            22,
                            23,
                            3389,
                            5900
                        ]
                    }                    
                }
            }
        },        
        
        "database": {
            "type": "object",
            "default": {},
            "Common Ports": "Disallowed database ports",
            "required": [],
            "properties": {
                "tcp": {
                    "type": "array",
                    "default": [1433,
                            1434,
                            3050,
                            3306,
                            5432
                        ],
                    "title": "Disallowed TCP ports",
                    "items": {
                        "type": "integer",
                        "examples": [
                            1433,
                            1434,
                            3050,
                            3306,
                            5432
                        ]
                    }                    
                }
            }
        },        
    }
}

Render this json schema as a form with 3 questions, with 2 sub-questions, etc.

When the user submits this form the following data (for example) should be posted to Rocky and stored in Bytes:

{
    "common": {
        "udp": [
            53
        ],
        "tcp": [
            25,
            53,
            110,
            143,
            993,
            995,
            80,
            443
        ]

    },
    "admin": {
        "tcp": [
            21,
            22,
            23,
            3389,
            5900
        ]
    },
    "database": {
        "tcp": [
            1433,
            1434,
            3050,
            3306,
            5432
        ]
    }
}

Which in turn is ingested by a normaliser who understands the context for this raw file (eg, which bit created the original questionOOI), and this would in turn produce ConfigBIT bound to the original Network the QuestionOOI was bound to.
The raw file's meta should hold a reference to the original network's PK, and the name of the BIT/form which allows us to trigger the correct normaliser.

[noamblitz]

After discussion with @underdarknl :

The configOOIs that come from normalizing answers will be saved in the graph. For example, the configOOI of the above vulnerable ports will be attached to the Network. In the bit config we will add a parameter that indicates which ooi has the config. In the example of the ports, it will say: ip_port.ip_address.network. The bit runner can then query the config attached to this ooi and pass it to the bit.

In the future, we can make these locations into in prioritized list so that the config ooi attached to the ip_address has priority over the ooi attached to the network.

In the first version we will make the location of the config a relation of the input ooi (ip_port.ip_address.network), in the future we can make it queriable (io_service.ip_port). OctopoesV3 might also make this easier with graphql queries.

[Donnype]

As a note: this means that the set of OOI's that can be used to attach a Config is limited to the connected component of the input OOI of the bit and must be higher up the hierarchy (or the input OOI itself).

[noamblitz]

Correct

[Donnype]

I wrote a query that could resolve to the right Config object immediately for the first use-case:

What we are trying to accomplish in this first example in the PR:

classDiagram

direction LR
    class Network
    class Hostname
    class Website
    class HTTPResource
    class Config

    Config : bit_id
    Config : ooi
    Network : name
    Hostname : network
    Website : hostname
    HTTPResource : website

    Network <|-- Config
    Network <|-- Hostname
    Hostname <|-- Website
    Website <|-- HTTPResource

Loading

is to get the Config object when we know both the primary key of the HTTPResource and the path towards the Network through the config_ooi_relation_path field. I did some testing and concluded that given the primary key pkey of a HTTPResource, this query should retrieve all Config objects with bit id bit_id pointing to the network without all the loops:

{:query
    {:find [(pull config [*])]
        :where [
            [e :xt/id pkey]
            [e :HTTPResource/website f]
            [f :Website/hostname g]
            [g :Hostname/network h]
            [config :Config/ooi h]
            [config :Config/bid_id bit_id]
        ]   
    }   
}

I am not sure if this would be possible to do with the current query functionality in Octopoes (@Lisser ?).

[Lisser]

We need to be careful with paths that traverse through edges on abstract classes:

IPPort.address.network traverses through the IPAddress abstract class.

{:query
    {:find [(pull port [*])]
        :where [
            [port :xt/id port_id]
            [port :IPPort/address ip_addr]
            (or [ip_addr :IPV4Address/network network]
                [ip_addr :IPV6Address/network network])
            [config :Config/ooi network]
            [config :Config/bid_id bit_id]
        ]   
    }   
}

[Donnype]

This worked to find a Finding on internet from an IPPort (V6 and V4):

{:query
    {:find [(pull finding [*])]
        :where [
            [e :xt/id "IPPort|internet|xxxx::xxxxx|tcp|23"]
            [e :IPPort/address f]
            ( or 
                [f :IPAddressV4/network n]
                [f :IPAddressV6/network n]
            )
            [finding :Finding/ooi n]
            [finding :Finding/finding_type "KATFindingType|KAT-500"]
        ]
    }
}

[Donnype]

#851

[Donnype]

Per the discussion of today with @underdarknl we decided to not allow more than 2 levels deep for the json schema Questions. @dekkers posed this question during standup again yesterday, and I have to say that I agree given:

• We don't have a full view yet of what we want to support
• We can build more complex decision trees from composed questions when needed (as discussed today).

We will never be able to remove some the element of trusting the user on their answers, so we will have to find a middle ground between muting findings with plain text reasons and having a full adaptive procedure workflow as described in this ticket. This means that we can probably soon finish this ticket. We do still have to consider:

• How/where to collect results from a Question (in place update of some results field on the Question object?).
• Fix the rendered json schema indentation on the ooi detail page.

[dekkers]

It should be possible to use KAT with hunderds of organisations. How can a user make changes to the configuration for all organisations at the same time? How can a default be configured so that new organisations get the right config? When we support multiple network in the future, if a new network is added, how will a network get the right default config?

[Donnype]

@dekkers Valid points, and I think adding Questions in response to adding a Network by default is not the way to go here in the end. If we decide that any configuration management should be done in XTDB we could look into "template organizations" to copy (configurable) default configurations from. Then changes to these configurations could be applied to multiple graphs.

One assumption here is that any configuration needed to run Bits must live in XTDB, and in particular the graph relevant to the current organisation, which forces us to apply some variant of a composite pattern with respect to configuration management for Bits. This is not the case for Boefjes/Normalizers of course (Katalogus), where we actually implemented this copy-from-organization as well. If we loosen this assumption we could also consider some "central" configuration service/database that is consulted at runtime. I personally think having XTDB, the Katalogus ánd env vars as configuration options is far from optimal in the first place.

This issue has turned a bit twofold: on the one hand Config OOIs and on the other Question OOIs that could be used to generate Config OOIs, but are for a large part also brought to live for handling procedures.

[noamblitz]

I want to add that most questions are only relevant if you want to deviate from default. For most people, the default would be okay.