Modify JSON filenames to include technique IDs

[joshswimlane]

I will submit a PR but wanted to create an issue to include technique IDs in the filename of each json file - I think that's easier then modifying the document jsons.

[sbousseaden]

@joshswimlane cool, but what if the captured technique is not documented in MITRE ? (which is often the case for macOS)

[joshswimlane]

Good question @sbousseaden. If you don't mind, I actually propose a change to the data structure which would help with this.

Would a simple json structure work for you and your purposes? I think it covers both the ability to categorize the tactic it belongs in as well as any potential techniques that may or may not have official technique IDs. Also you can have a list of hits and/or documents defined (based on other jsons within the repository). Feedback definitely welcome!

{
    "tactic": "Collection",
    "technique_names": [
        {
            "Clipboard": "T115",
            "Osascript": null,
            "pbpaste": null
        }
    ],
    "hits": [],
    "documents": [
        {
            "_index": ".ds-logs-endpoint.events.process-default-000003",
            "_id": "LGp3AHcBimKzADJjaHso",
            "_source": {
                "agent": {
                    "id": "c2d9ce9a-fdef-a405-125c-171a91d0e54a",
                    "type": "endpoint",
                    "version": "7.10.1"
                },
                "process": {
                    "Ext": {
                        "ancestry": []
                    },
                    "args": [
                        "osascript",
                        "-e",
                        "get the clipboard"
                    ],
                    "parent": {
                        "args": [],
                        "entity_id": ""
                    },
                    "name": "osascript",
                    "pid": 25623,
                    "args_count": 3,
                    "entity_id": "YzJkOWNlOWEtZmRlZi1hNDA1LTEyNWMtMTcxYTkxZDBlNTRhLTI1NjIzLTEzMjU1MDk0MTUwLjUyMDcwODAwMA==",
                    "command_line": "osascript -e get the clipboard",
                    "executable": "/usr/bin/osascript",
                    "hash": {
                        "sha1": "9f938559a0956dfae4ba48eaf7378dcb799761b5",
                        "sha256": "0ca8c6f4a574c803d68439de2565e85f2e2572b4480ef245ff1293fb4dc0c06f",
                        "md5": "22997dd0b65f7f96d99225788584c88f"
                    }
                },
                "message": "Endpoint process event",
                "@timestamp": "2021-01-14T10:35:50.520708Z",
                "ecs": {
                    "version": "1.5.0"
                },
                "data_stream": {
                    "namespace": "default",
                    "type": "logs",
                    "dataset": "endpoint.events.process"
                },
                "elastic": {
                    "agent": {
                        "id": "bbc973a1-6626-414a-88e5-43be8d909777"
                    }
                },
                "host": {
                    "hostname": "Sisis-MacBook-Pro.local",
                    "os": {
                        "Ext": {
                            "variant": "macOS"
                        },
                        "kernel": "Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64",
                        "name": "macOS",
                        "family": "macos",
                        "version": "10.15.3",
                        "platform": "macos",
                        "full": "macOS 10.15.3"
                    },
                    "ip": [
                        "127.0.0.1",
                        "::1",
                        "fe80::1",
                        "fe80::aede:48ff:fe00:1122",
                        "192.168.178.59",
                        "fe80::1830:c57a:1313:130b",
                        "2a02:a210:2302:b100:101c:31bc:4822:42cb",
                        "2a02:a210:2302:b100:f047:1980:a3c3:93c9",
                        "fe80::7cba:b6ff:feee:dd07",
                        "fe80::c19f:ad51:9312:adfd",
                        "fe80::7610:f180:3dda:a15f"
                    ],
                    "name": "Sisis-MacBook-Pro.local",
                    "architecture": "x86_64"
                },
                "event": {
                    "sequence": 206732,
                    "ingested": "2021-01-14T10:36:01.446033744Z",
                    "created": "2021-01-14T10:35:50.520708Z",
                    "kind": "event",
                    "module": "endpoint",
                    "action": "exec",
                    "id": "LzkFKysBSmcwOUEj++++/4b3",
                    "category": [
                        "process"
                    ],
                    "type": [
                        "start"
                    ],
                    "dataset": "endpoint.events.process"
                },
                "user": {
                    "Ext": {
                        "real": {
                            "name": "sisi",
                            "id": 501
                        }
                    },
                    "name": "sisi",
                    "id": 501
                },
                "group": {
                    "Ext": {
                        "real": {
                            "name": "staff",
                            "id": 20
                        }
                    },
                    "name": "staff",
                    "id": 20
                }
            }
        },
        {
            "_index": ".ds-logs-endpoint.events.process-default-000003",
            "_id": "_3EX_HYBORzSN0EUsZdP",
            "_source": {
                "agent": {
                    "id": "c2d9ce9a-fdef-a405-125c-171a91d0e54a",
                    "type": "endpoint",
                    "version": "7.10.1"
                },
                "process": {
                    "Ext": {
                        "ancestry": []
                    },
                    "args": [
                        "pbpaste"
                    ],
                    "parent": {
                        "args": [],
                        "entity_id": ""
                    },
                    "name": "pbpaste",
                    "pid": 18884,
                    "args_count": 1,
                    "entity_id": "YzJkOWNlOWEtZmRlZi1hNDA1LTEyNWMtMTcxYTkxZDBlNTRhLTE4ODg0LTEzMjU1MDE4NTc0LjkxMzEwNDAwMA==",
                    "command_line": "pbpaste",
                    "executable": "/usr/bin/pbpaste",
                    "hash": {
                        "sha1": "4a95bff43a6932164e1a799f822e618c7a921c0e",
                        "sha256": "dc1360e4303492afd79a40cfc3d6535e0f854a1680c5467d26f2df395be396b4",
                        "md5": "c76b93114fcb5b133c8e0c582cd94f18"
                    }
                },
                "message": "Endpoint process event",
                "@timestamp": "2021-01-13T13:36:14.913104Z",
                "ecs": {
                    "version": "1.5.0"
                },
                "data_stream": {
                    "namespace": "default",
                    "type": "logs",
                    "dataset": "endpoint.events.process"
                },
                "elastic": {
                    "agent": {
                        "id": "bbc973a1-6626-414a-88e5-43be8d909777"
                    }
                },
                "host": {
                    "hostname": "Sisis-MacBook-Pro.local",
                    "os": {
                        "Ext": {
                            "variant": "macOS"
                        },
                        "kernel": "Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64",
                        "name": "macOS",
                        "family": "macos",
                        "version": "10.15.3",
                        "platform": "macos",
                        "full": "macOS 10.15.3"
                    },
                    "ip": [
                        "127.0.0.1",
                        "::1",
                        "fe80::1",
                        "fe80::aede:48ff:fe00:1122",
                        "192.168.178.59",
                        "fe80::1830:c57a:1313:130b",
                        "2a02:a210:2302:b100:101c:31bc:4822:42cb",
                        "2a02:a210:2302:b100:6c5a:7245:7828:3a1f",
                        "fe80::7cba:b6ff:feee:dd07",
                        "fe80::c19f:ad51:9312:adfd",
                        "fe80::7610:f180:3dda:a15f"
                    ],
                    "name": "Sisis-MacBook-Pro.local",
                    "architecture": "x86_64"
                },
                "event": {
                    "sequence": 146561,
                    "ingested": "2021-01-13T14:12:59.822898535Z",
                    "created": "2021-01-13T13:36:14.913104Z",
                    "kind": "event",
                    "module": "endpoint",
                    "action": "exec",
                    "id": "LzkFKysBSmcwOUEj+++++lyV",
                    "category": [
                        "process"
                    ],
                    "type": [
                        "start"
                    ],
                    "dataset": "endpoint.events.process"
                },
                "user": {
                    "Ext": {
                        "real": {
                            "name": "sisi",
                            "id": 501
                        }
                    },
                    "name": "sisi",
                    "id": 501
                },
                "group": {
                    "Ext": {
                        "real": {
                            "name": "staff",
                            "id": 20
                        }
                    },
                    "name": "staff",
                    "id": 20
                }
            }
        }
    ]
}   

[sbousseaden]

@joshswimlane looks good, thanks for the effort.

[jaimeatwork]

I'm interested in helping out with this. I've completed most of the work since I'll need this anyway in a fork here - https://github.com/jaimeatwork/macOS-ATTACK-DATASET/tree/dev

If you're interested, when I'm done, I can either do a pull request branch to branch or open individual pull requests by folder (I'd like to avoid file by file since that might feel tedious 🤣 ).

Some things to note:

• Most files will show as completely rewritten since adding top level keys and formatting the json caused everything to get reindented, but data wise, there were very few changes except for the structure.
• There is probably plenty of room to disagree on the specific techniques I mapped to each of the event collections, I likely missed some or misapplied some labels but they should be pretty close.
• There were some lines where events were triple quoted and I converted those to single sets of double quotes and escaped double quotes within the data

This is an outline of the schema

{
    "tactic": "Collection",
    "techniques": [
    "T1000",
    "T1001"
    ],
    "documents": [
    < original content from elastic events, unchanged except for above >
    ]
}

When I complete the rewrite of the data files, I'm going to make some tooling (attack-navigator heat map? not sure entirely what yet) and that can be included or not too.

[Samirbous]

@jaimeatwork that's awesome, I think a PR branch to branch will be good.