Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create osint Profile #992

Merged
merged 33 commits into from
May 31, 2024
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
a9d21f2
add `threat_intelligence`, expand `dictionary`
jonrau-at-queryai Mar 18, 2024
a68cc78
add `subdomains`, some `threat_intelligence` fixes
jonrau-at-queryai Mar 18, 2024
9012027
Add `whois` and `contract` objects, formatting
jonrau-at-queryai Mar 18, 2024
edf4d6f
Add `stix_object` and `stix_campaign`
jonrau-at-queryai Mar 19, 2024
a39851e
Add `stix_malware` and support attrs
jonrau-at-queryai Mar 19, 2024
572e5b3
Add `stix_malware_analysis` and related attrs
jonrau-at-queryai Mar 19, 2024
46f7129
Add `stix_threat_actor` and `stix_tool`
jonrau-at-queryai Mar 19, 2024
913c0b8
Add `stix_intrusion_set`, small updates and fixes
jonrau-at-queryai Mar 19, 2024
996dda7
create threat intel profile
jonrau-at-queryai Mar 19, 2024
9b7f6a0
Update CHANGELOG.md
jonrau-at-queryai Mar 19, 2024
59b1114
Update CHANGELOG.md
jonrau-at-queryai Mar 19, 2024
81db723
removing STIX references
jonrau-at-queryai May 2, 2024
94c630e
change `contact` to `domain_contact`
jonrau-at-queryai May 2, 2024
f88599a
`domain_contact` fix, resolve conflicts
jonrau-at-queryai May 2, 2024
a97629f
more deconflicts
jonrau-at-queryai May 2, 2024
80ef4e7
Add `email`, `email_auth`, caption changes
jonrau-at-queryai May 2, 2024
f829690
Update CHANGELOG.md
jonrau-at-queryai May 2, 2024
f22cf5b
OSINt changes, remove STIX, fix checks
jonrau-at-queryai May 7, 2024
1d74944
more Changelog fixes
jonrau-at-queryai May 7, 2024
714c216
apparently `classifications` was in 1.2?
jonrau-at-queryai May 7, 2024
1737794
remove more stix refs from dictionary
jonrau-at-queryai May 7, 2024
52e317e
Add `tlp` - Traffic Light Protocol
jonrau-at-queryai May 7, 2024
032987e
formatted, add additional type_id to osint
jonrau-at-queryai May 7, 2024
9cd7613
purge remaining STIX references
jonrau-at-queryai May 21, 2024
353be11
Update dictionary.json
jonrau-at-queryai May 21, 2024
cc77fd7
Update CHANGELOG.md
jonrau-at-queryai May 21, 2024
735e4e4
Dictionary fixes for WHOIS
jonrau-at-queryai May 23, 2024
50562de
more validator fixes
jonrau-at-queryai May 23, 2024
72b89b3
small dict fix, revert file activity fix
jonrau-at-queryai May 23, 2024
d370745
update changelog...for the 17th time
jonrau-at-queryai May 23, 2024
d687239
Merge branch 'main' into threat-intel-profile
jonrau-at-queryai May 29, 2024
18176bd
final fixes on profile & formatting
jonrau-at-queryai May 29, 2024
1ebb8df
Update base_event.json
jonrau-at-queryai May 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
206 changes: 150 additions & 56 deletions CHANGELOG.md
jonrau-at-queryai marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -11,68 +11,66 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

Thankyou! -->

## [Unreleased]
<!-- All available sections in the Changelog:

### Added
* #### Categories
* #### Event Classes
1. Added `Data Security Finding` event class. #953
* #### Profiles
* #### Objects
1. Added `auth_factor` object. #949
2. Added `data_security` object. #953
3. Added `autonomous_system` object. #978
* #### Platform Extensions

### Improved
* #### Categories
* #### Event Classes
1. Added `auth_factors` array to Authentication event class. #949
2. Modified all classes such that primary attributes are at least recommended. #974
3. Added `src_endpoint`, `http_request` attributes to all IAM category classes. #976
4. Added `autonomous_system` to `network_endpoint` objects. #978
* #### Profiles
* #### Objects
1. Expanded `type_id` enum in `analytic` object to account for more use-cases: #953
- `5 - Fingerprinting`
- `6 - Tagging`
- `7 - Keyword Match`
- `8 - Regular Expressions`
- `9 - Exact Data Match`
- `10 - Partial Data Match`
- `11 - Indexed Data Match`
2. Added `lat`, `long`, `geohash` attributes to `location` object. #971.
3. Added `risk_score`, `risk_level_id`, `risk_level` to `user` object. Issue #972.
4. Added `app_name`, `app_uid` to `actor` object. Issue #966, PR #979.
* #### Objects
* #### Platform Extensions

### Bugfixes
1. Changed datatype of `priority` attribute, from `integer_t` to `string_t` #959

### Deprecated
1. Deprecated `coordinates` attribute in favor of specific `lat`, `long` attributes. #971
2. Deprecated `invoked_by` attribute in the `Actor` object in favor of `app_name`. #979.

### Breaking changes

### Misc
1. New Extension registration for Sedara. #951
2. Add new ways to define observables to metaschema. #982

<!-- All available sections in the Changelog:
-->

## [Unreleased]

### Added
* #### Categories
* #### Event Classes
1. Added `Event Log Activity` event class. #1014
2. Added `Remediation Activity` `File Remediation Activity` `Process Remediation Activity` `Network Remediation Activity` event classes. #1066
* #### Profiles
1. Added `osint` Profile based on `osint` object. #992
* #### Objects
1. Added `d3fend` `d3f_tactic` `d3f_technique` MITRE objects. #1066
2. Added `ja4_fingerprint` object. #834
3. Added `ja4_fingerprint_list` as a list of `ja4_fingerprint` objects. #834
4. Added `ticket` object. #1068
5. Added `osint` object. #992
6. Added `signatures` object, an array of `signature` objects. #992
7. Added `whois` object. #992
8. Added `domain_contact` and array-typed `domain_contacts` object for use with `whois` object. #992
* #### Platform Extensions

### Improved
* #### Categories
* #### Event Classes
1. Added `file_result` to File Hosting Activity. #1045
2. Added entries to `injection_type_id` enum (`Process Activity`) and `activity_id` enum (`Memory Activity`). #1060
3. Added a `Restart`, `Enable`, `Disable`, and `Update` `activity_id` to the `Application Lifecycle` class. #1064
4. Added `ja4_fingerprint_list` to base network event class. #834
* #### Profiles
* #### Objects
1. Added `ext` to `File` object. #1046
2. Added `account`, `device`, `email`, `url`, `user` to `evidences` in detection finding. #1000
3. Added `state_id`, `state` to `Digital Signature` object. #1069
4. Added `ticket` to `Incident Finding` object. ticket. #1068
5. Added `domain` to `Uniform Resource Locator` object. #1096
* #### Platform Extensions

### Bugfixes
Expand All @@ -82,8 +80,121 @@ Thankyou! -->
### Breaking changes

### Misc
1. Colorized validator output #1048
* Updated the GitHub workflow for the `ocsf-validator` to print colorized output.
2. Clarify how to reference profiles in metadata #1056
* Updated the description of `metadata.profiles` to clarify the correct way to reference a profile in that list.
3. Added a `gitignore` file. #1071
4. New Extension registration for Cisco #1074
5. Cleaned up MITRE trademarks and registrations for captions and descriptions.

-->
## [v1.2.0] - April 23rd, 2024

### Added
* #### Categories
n/a
* #### Event Classes
1. Added `Data Security Finding` event class. #953
2. Added `File Query` event class. #967
3. Added `Folder Query` event class. #967
4. Added `Group Query` event class. #967
5. Added `Job Query` event class. #967
6. Added `Kernel Object Query` event class. #967
7. Added `Module Query` event class. #967
8. Added `Network Connection Query` event class. #967
9. Added `Networks Query` event class. #967
10. Added `Peripheral Device Query` event class. #967
11. Added `Prefetch Query` event class. #967
12. Added `Process Query` event class. #967
13. Added `Registry Key Query` event class. #967
14. Added `Registry Value Query` event class. #967
15. Added `Service Query` event class. #967
16. Added `Session Query` event class. #967
17. Added `User Query` event class. #967
18. Added `Tunnel Activity` event class. #1012

* #### Profiles
1. Added `data_classification` profile. #998

* #### Objects
1. Added `auth_factor` object. #949
2. Added `data_security` object. #953
3. Added `autonomous_system` object. #978
4. Added `agent` object. #987
5. Added `data_classification` object. #998

* #### Observables
1. Added `port_t` `subnet_t` `cmd_line` `country` `pid` `cwe.uid` `cve.uid` `user_agent` enum items. #1035

* #### Platform Extensions
n/a

### Improved
* #### Categories
* #### Event Classes
1. Added `auth_factors` array to Authentication event class. #949
2. Modified all classes such that primary attributes are at least recommended. #974
3. Added `src_endpoint`, `http_request` attributes to all IAM category classes. #976
4. Added `autonomous_system` to `network_endpoint` objects. #978
5. Added `List`, `Encrypt` and `Decrypt` activities to `datastore` event class. #989
6. Added `file` attribute to `http`, `rdp`, `ssh`, and `ftp` event classes. #985
7. Added a `Preauth` `activity_id` to the `Authentication` class. #1018
8. Added the `Security Control` profile to the `Datastore Activity` class. #1030
9. Added `risk_details` to Detection Finding. #1032

* #### Profiles
n/a
* #### Objects
1. Expanded `type_id` enum in `analytic` object to account for more use-cases: #953
- `5 - Fingerprinting`
- `6 - Tagging`
- `7 - Keyword Match`
- `8 - Regular Expressions`
- `9 - Exact Data Match`
- `10 - Partial Data Match`
- `11 - Indexed Data Match`
2. Added `lat`, `long`, `geohash` attributes to `location` object. #971.
3. Added `risk_score`, `risk_level_id`, `risk_level` to `user` object. Issue #972.
4. Added `app_name`, `app_uid` to `actor` object. Issue #966, PR #979.
5. Added `container`, `database`, `databucket` to the `evidences` object. #984
6. Added `owner` to `endpoint` object. #987
7. Added `is_applied` Boolean attribute to `policy` object. #987
8. Added `agent_list` as an array of `agent` objects. #987
9. Added `policies` object as an array of `policy` objects. #987
10. Added `agent_list` to `endpoint` object. #987
11. Added `labels` to the `Account` object. #1028
12. Added `data_classification` profile to `database`, `databucket`, `email`, `file`, `metadata`, `product`, `resource_details` and `web_resource` objects. #998

* #### Platform Extensions
n/a

### Bugfixes
1. Changed datatype of `priority` attribute, from `integer_t` to `string_t` #959
2. Extended `email_t` regexp to allow characters from RFC5322 before @.
3. Updated `logon_type_id` enum to include `0` as `Unknown`. Added enum item `1` as `System`. #1055

### Deprecated
1. Deprecated `coordinates` attribute in favor of specific `lat`, `long` attributes. #971
2. Deprecated `invoked_by` attribute in the `Actor` object in favor of `app_name`. #979.

### Breaking changes
n/a

### Misc
1. New Extension registration for Sedara. #951
2. Corrected punctuation for the `transmit_time` attribute. #1001
3. New ways to define observables in the metaschema. #982 and #993
* (Current) Dictionary types using `observable` property in dictionary types. This allows defining all occurrences of attributes of this type as an observable.
* (Current) Objects using top-level `observable` property. This allows defining all occurrences attributes whose type is this object as an observable.
* _**(New)**_ Dictionary attributes using `observable` property in attribute. This allows defining all occurrences of this attribute as an observable.
* _**(New)**_ Object-specific attributes using `observable` property class's attributes. This allows defining object attributes as observables _only_ within instances of this specific object.
* _**(New)**_ Event class-specific attributes using `observable` property class's attributes. This allows defining class attributes as observables _only_ within instances of this specific class.
* _**(New)**_ Event class-specific attribute _paths_ using top-level `observables` property. The `observables` property holds an object mapping from an dotted attribute path to an observable `type_id`. This allows defining an observables _only_ within instances of this specific class, and only for the attributes at these paths, even for attributes that are within nested objects and arrays. This can also be used for top-level class attributes, which can be more convenient that defining a class attribute observable for classes that extend another, but don't otherwise change a attribute definition.
4. Metaschema improvements. #993
* Detect unexpected top-level properties in object and event class definitions. This was added at this point to detect invalid observable definitions: invalid `observable` property in event classes, and invalid `observables` property in objects.
* Remove hard-coded list of categories from `metaschema/categories.schema.json`, leaving this to the `ocsf-validator`. This change makes testing with alternate schemas that may add extra categories easier, as well as making it possible to validate private extensions that contain new categories.
5. Metaschema error reporting #1027
* Updated the definition of `object` and `event` so that metaschema errors reported by the validator with nested properties correctly attribute the error to the property with the error, rather than the top-level class.

## [v1.1.0] - January 25th, 2024

Expand All @@ -92,33 +203,16 @@ Thankyou! -->
`n/a`
* #### Event Classes
1. Added `User Inventory Info` event class. #667
2. Added `Vulnerability Finding` event class. #698
2. Added `NTP Activity` event class #705
3. Added `OS Patch State` event class. #746
4. Added `Datastore Activity` event class 6005. #874
5. Added `Detection Finding` event class. #877
6. Added `Incident Finding` event class. #903
7. Added `Device Config Sate Change` event class. #914
8. Added `Scan Activity` event class. #915
9. Added `File Hosting Activity` event class. #917
10. Added `File Query` event class. #967
11. Added `Folder Query` event class. #967
12. Added `Group Query` event class. #967
13. Added `Job Query` event class. #967
14. Added `Kernel Object Query` event class. #967
15. Added `Module Query` event class. #967
16. Added `Network Connection Query` event class. #967
17. Added `Networks Query` event class. #967
18. Added `Peripheral Device Query` event class. #967
19. Added `Prefetch Query` event class. #967
20. Added `Process Query` event class. #967
21. Added `Registry Key Query` event class. #967
22. Added `Registry Value Query` event class. #967
23. Added `Service Query` event class. #967
24. Added `Session Query` event class. #967
25. Added `Startup Application Query` event class. #967
26. Added `User Query` event class. #967

2. Added `Vulnerability Finding` event class. #698
3. Added `NTP Activity` event class #705
4. Added `OS Patch State` event class. #746
5. Added `Datastore Activity` event class 6005. #874
6. Added `Detection Finding` event class. #877
7. Added `Incident Finding` event class. #903
8. Added `Device Config Sate Change` event class. #914
9. Added `Scan Activity` event class. #915
10. Added `File Hosting Activity` event class. #917

* #### Profiles
1. Added `Network Proxy` Profile for the `Network Activity` and `Application Activity` classes. #705
2. Added `Load Balancer` Profile for the Network Activity classes. #897
Expand Down Expand Up @@ -193,4 +287,4 @@ Thankyou! -->

## [v1.0.0]

Initial release of OCSF.
Initial release of OCSF.
Loading