Implemented ListIOCs API. #1064

AWSHurneyt · 2024-06-11T05:24:53Z

Description

Implemented ListIOCs API.

This PR overlaps with PR #1060, so leaving it as a draft for now.

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: AWSHurneyt <[email protected]>

src/main/java/org/opensearch/securityanalytics/action/ListIOCsActionResponse.java

…red at the IOC level. Signed-off-by: AWSHurneyt <[email protected]>

Signed-off-by: AWSHurneyt <[email protected]>

… IOC level. Signed-off-by: AWSHurneyt <[email protected]>

…otFoundException when calling ListIOCs API. Signed-off-by: AWSHurneyt <[email protected]>

AWSHurneyt · 2024-06-17T19:17:22Z

src/main/java/org/opensearch/securityanalytics/action/ListIOCsAction.java

+
+public class ListIOCsAction extends ActionType<ListIOCsActionResponse> {
+    public static final ListIOCsAction INSTANCE = new ListIOCsAction();
+    public static final String NAME = "cluster:admin/opensearch/securityanalytics/iocs/list";


Including #1077 (comment)

should we call this ioc/_search or just cluster:admin/opensearch/securityanalytics/iocs is also good enough i guess

let's standardize all list apis...

@eirsep I believe we plan to have a separate SearchIOCs API which will support receiving a query in the request body (similar to the SearchMonitor API), so I was reserving the _search route for that. iocs/list follows the pattern used by the ListCorrelations API, but we can change this if needed.
https://github.com/opensearch-project/security-analytics/blob/feature/threat_intel/src/main/java/org/opensearch/securityanalytics/action/ListCorrelationsAction.java#L11

we need the search Ioc’s api.. that’s more flexible and if we are starting out with only one of these it should be the search API

Will implement a search IOC API separately.

AWSHurneyt · 2024-06-17T19:22:41Z

src/main/java/org/opensearch/securityanalytics/action/ListIOCsActionRequest.java

+    private int size;
+    private SortOrder sortOrder;
+    private String sortString;


Including #1077 (comment)

we need feedId as a parameter
check with @amsiglan if UX would be better suited to work with a search request rather than params?

@eirsep Added feedId to the request object. I believe @amsiglan mentioned that a list API would be more useful for the UI table, but I'll sync with him to confirm.

Signed-off-by: AWSHurneyt <[email protected]>

…red at the IOC level. Signed-off-by: AWSHurneyt <[email protected]>

Signed-off-by: AWSHurneyt <[email protected]>

…otFoundException when calling ListIOCs API. Signed-off-by: AWSHurneyt <[email protected]>

Signed-off-by: AWSHurneyt <[email protected]>

# Conflicts: # src/main/java/org/opensearch/securityanalytics/SecurityAnalyticsPlugin.java # src/main/java/org/opensearch/securityanalytics/model/STIX2IOC.java # src/main/java/org/opensearch/securityanalytics/model/STIX2IOCDto.java # src/main/java/org/opensearch/securityanalytics/services/STIX2IOCConsumer.java # src/main/java/org/opensearch/securityanalytics/services/STIX2IOCFeedStore.java # src/main/java/org/opensearch/securityanalytics/services/STIX2IOCFetchService.java # src/main/resources/mappings/stix2_ioc_mapping.json # src/test/java/org/opensearch/securityanalytics/resthandler/ListIOCsRestApiIT.java # src/test/java/org/opensearch/securityanalytics/util/STIX2IOCGenerator.java

Signed-off-by: AWSHurneyt <[email protected]>

src/main/java/org/opensearch/securityanalytics/SecurityAnalyticsPlugin.java

src/main/java/org/opensearch/securityanalytics/transport/TransportListIOCsAction.java

eirsep · 2024-06-17T23:42:46Z

src/main/java/org/opensearch/securityanalytics/resthandler/RestListIOCsAction.java

+        int size = request.paramAsInt(ListIOCsActionRequest.SIZE_FIELD, 10);
+        String sortOrder = request.param(ListIOCsActionRequest.SORT_ORDER_FIELD, ListIOCsActionRequest.SortOrder.asc.toString());
+        String sortString = request.param(ListIOCsActionRequest.SORT_STRING_FIELD, STIX2.NAME_FIELD);
+        String search = request.param(ListIOCsActionRequest.SEARCH_FIELD, "");


what is search field?

The search field is used to support the search bar (see screenshot). Currently, it'll search the name, value, severity, create time, and modified time fields of the docs.

eirsep · 2024-06-17T23:43:56Z

src/main/java/org/opensearch/securityanalytics/transport/TransportListIOCsAction.java

+                        QueryBuilders.queryStringQuery(request.getSearch())
+                                .defaultOperator(Operator.OR)
+//                            .field(STIX2_IOC_NESTED_PATH + STIX2IOC.ID_FIELD) // Currently not a column in UX table
+                                .field(STIX2_IOC_NESTED_PATH + STIX2IOC.NAME_FIELD)


can we remove nesting?

I think we still need to include it.

Elsewhere in the code, it looks like we need to provide the nested path; even when using QueryBuilders.nestedQuery().
https://github.com/opensearch-project/security-analytics/blob/feature/threat_intel/src/main/java/org/opensearch/securityanalytics/transport/TransportIndexCustomLogTypeAction.java#L449

The GetAlerts API doesn't need to provide a nested path
https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportGetAlertsAction.kt#L138

However, the alerts values aren't nested in an alert object in the mappings for that system index.
https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/resources/org/opensearch/alerting/alerts/alert_mapping.json#L28

src/main/java/org/opensearch/securityanalytics/action/ListIOCsActionRequest.java

eirsep · 2024-06-19T07:41:16Z

src/main/java/org/opensearch/securityanalytics/transport/TransportListIOCsAction.java

+        private ListIOCsActionRequest request;
+        private ActionListener<ListIOCsActionResponse> listener;
+
+        private final AtomicReference<Object> response;


what is this for?

src/main/java/org/opensearch/securityanalytics/transport/TransportListIOCsAction.java

* Removed unused imports. Removed redundant helper function. Signed-off-by: AWSHurneyt <[email protected]> * Added note about system index refactoring. Signed-off-by: AWSHurneyt <[email protected]> * Implemented draft of IocService. Signed-off-by: AWSHurneyt <[email protected]> * Made changes based on PR feedback. Signed-off-by: AWSHurneyt <[email protected]> * Fixed test helper function. Signed-off-by: AWSHurneyt <[email protected]> * Removed unused imports. Signed-off-by: AWSHurneyt <[email protected]> * Adjusted mappings based on PR feedback. Signed-off-by: AWSHurneyt <[email protected]> * Continuation of fetch IOC service implementation. Signed-off-by: AWSHurneyt <[email protected]> * Implemented ListtIOCs API. Signed-off-by: AWSHurneyt <[email protected]> * Removed "enabled" field from ListIOCs API as that will not be configured at the IOC level. Signed-off-by: AWSHurneyt <[email protected]> * Renamed response keys. Signed-off-by: AWSHurneyt <[email protected]> * Removed "enabled" field mapping as that will not be configured at the IOC level. Signed-off-by: AWSHurneyt <[email protected]> * Added feedId as a filter for LiistIOCs API. Added handling for IndexNotFoundException when calling ListIOCs API. Signed-off-by: AWSHurneyt <[email protected]> * Implemented ListtIOCs API. Signed-off-by: AWSHurneyt <[email protected]> * Removed "enabled" field from ListIOCs API as that will not be configured at the IOC level. Signed-off-by: AWSHurneyt <[email protected]> * Renamed response keys. Signed-off-by: AWSHurneyt <[email protected]> * Removed unused test suite. Signed-off-by: AWSHurneyt <[email protected]> * Added feedId as a filter for LiistIOCs API. Added handling for IndexNotFoundException when calling ListIOCs API. Signed-off-by: AWSHurneyt <[email protected]> * Added feedId as a filter for ListIOCs API. Signed-off-by: AWSHurneyt <[email protected]> * Fixed merge conflict. Signed-off-by: AWSHurneyt <[email protected]> * Removed unused test suite. Signed-off-by: AWSHurneyt <[email protected]> * Fixed test case. Signed-off-by: AWSHurneyt <[email protected]> * Fixed test index mappings. Signed-off-by: AWSHurneyt <[email protected]> --------- Signed-off-by: AWSHurneyt <[email protected]>

AWSHurneyt added 9 commits June 7, 2024 14:36

Removed unused imports. Removed redundant helper function.

363a4b0

Signed-off-by: AWSHurneyt <[email protected]>

Added note about system index refactoring.

7dfbf27

Signed-off-by: AWSHurneyt <[email protected]>

Implemented draft of IocService.

a94f218

Signed-off-by: AWSHurneyt <[email protected]>

Made changes based on PR feedback.

85a680b

Signed-off-by: AWSHurneyt <[email protected]>

Fixed test helper function.

49d8e6c

Signed-off-by: AWSHurneyt <[email protected]>

Removed unused imports.

c0bc366

Signed-off-by: AWSHurneyt <[email protected]>

Adjusted mappings based on PR feedback.

186264a

Signed-off-by: AWSHurneyt <[email protected]>

Continuation of fetch IOC service implementation.

3ee454e

Signed-off-by: AWSHurneyt <[email protected]>

Implemented ListtIOCs API.

6fb2ae7

Signed-off-by: AWSHurneyt <[email protected]>

eirsep reviewed Jun 11, 2024

View reviewed changes

src/main/java/org/opensearch/securityanalytics/action/ListIOCsActionResponse.java Show resolved Hide resolved

eirsep reviewed Jun 11, 2024

View reviewed changes

src/main/java/org/opensearch/securityanalytics/action/ListIOCsActionResponse.java Outdated Show resolved Hide resolved

AWSHurneyt added 3 commits June 12, 2024 16:13

Removed "enabled" field from ListIOCs API as that will not be configu…

e7cd6cd

…red at the IOC level. Signed-off-by: AWSHurneyt <[email protected]>

Renamed response keys.

710ba2a

Signed-off-by: AWSHurneyt <[email protected]>

Removed "enabled" field mapping as that will not be configured at the…

5800517

… IOC level. Signed-off-by: AWSHurneyt <[email protected]>

AWSHurneyt mentioned this pull request Jun 14, 2024

Assets for IOC fetch and index service. #1077

Merged

5 tasks

Added feedId as a filter for LiistIOCs API. Added handling for IndexN…

6911262

…otFoundException when calling ListIOCs API. Signed-off-by: AWSHurneyt <[email protected]>

AWSHurneyt commented Jun 17, 2024

View reviewed changes

AWSHurneyt added 10 commits June 17, 2024 15:43

Implemented ListtIOCs API.

b5a667d

Signed-off-by: AWSHurneyt <[email protected]>

Removed "enabled" field from ListIOCs API as that will not be configu…

fda73b6

…red at the IOC level. Signed-off-by: AWSHurneyt <[email protected]>

Renamed response keys.

27bac34

Signed-off-by: AWSHurneyt <[email protected]>

Removed unused test suite.

fa9637a

Signed-off-by: AWSHurneyt <[email protected]>

Added feedId as a filter for LiistIOCs API. Added handling for IndexN…

94683b3

…otFoundException when calling ListIOCs API. Signed-off-by: AWSHurneyt <[email protected]>

Added feedId as a filter for ListIOCs API.

d184625

Signed-off-by: AWSHurneyt <[email protected]>

Fixed merge conflict.

d4e1937

Signed-off-by: AWSHurneyt <[email protected]>

Removed unused test suite.

74d14ba

Signed-off-by: AWSHurneyt <[email protected]>

Fixed test case.

a850219

Signed-off-by: AWSHurneyt <[email protected]>

AWSHurneyt marked this pull request as ready for review June 17, 2024 23:28

AWSHurneyt requested a review from amsiglan as a code owner June 17, 2024 23:28

AWSHurneyt requested review from getsaurabh02, lezzago, praveensameneni, sbcd90, jowg-amazon, engechas, goyamegh and riysaxen-amzn as code owners June 17, 2024 23:28

Fixed test index mappings.

6e08115

Signed-off-by: AWSHurneyt <[email protected]>

eirsep reviewed Jun 17, 2024

View reviewed changes

src/main/java/org/opensearch/securityanalytics/SecurityAnalyticsPlugin.java Show resolved Hide resolved

eirsep reviewed Jun 17, 2024

View reviewed changes

src/main/java/org/opensearch/securityanalytics/transport/TransportListIOCsAction.java Show resolved Hide resolved

eirsep reviewed Jun 17, 2024

View reviewed changes

eirsep requested changes Jun 17, 2024

View reviewed changes

AWSHurneyt commented Jun 18, 2024

View reviewed changes

src/main/java/org/opensearch/securityanalytics/action/ListIOCsActionRequest.java Show resolved Hide resolved

eirsep reviewed Jun 19, 2024

View reviewed changes

eirsep approved these changes Jun 19, 2024

View reviewed changes

eirsep merged commit d71579c into opensearch-project:feature/threat_intel Jun 19, 2024
1 check passed

eirsep reviewed Jun 19, 2024

View reviewed changes

src/main/java/org/opensearch/securityanalytics/transport/TransportListIOCsAction.java Show resolved Hide resolved

AWSHurneyt deleted the 3.0-threat-intel-list-iocs branch June 19, 2024 19:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented ListIOCs API. #1064

Implemented ListIOCs API. #1064

AWSHurneyt commented Jun 11, 2024 •

edited

Loading

AWSHurneyt Jun 17, 2024

eirsep Jun 17, 2024

AWSHurneyt Jun 18, 2024

AWSHurneyt Jun 17, 2024

eirsep Jun 17, 2024

AWSHurneyt Jun 18, 2024

eirsep Jun 17, 2024

AWSHurneyt Jun 18, 2024

eirsep Jun 19, 2024

Implemented ListIOCs API. #1064

Implemented ListIOCs API. #1064

Conversation

AWSHurneyt commented Jun 11, 2024 • edited Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AWSHurneyt commented Jun 11, 2024 •

edited

Loading