Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[7.x] [SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists (#72337) #72483

Merged
merged 1 commit into from
Jul 20, 2020
Merged

[7.x] [SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists (#72337) #72483

merged 1 commit into from
Jul 20, 2020

Conversation

FrankHassanabad
Copy link
Contributor

Backports the following commits to 7.x:

@FrankHassanabad
[SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts im…
18d251d 
…provements to lists (elastic#72337)

## Summary

* Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities.
* Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types
* Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things
* Removes extra types that were causing confusion which was an idiom from io-ts
* Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point.

NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted.


Output example of posting an exception list:

❯ ./post_exception_list.sh
```ts
{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "Wzk4NiwxXQ==",
  "created_at": "2020-07-17T18:59:22.872Z",
  "created_by": "yo",
  "description": "This is a sample endpoint type exception",
  "id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
  "type": "detection",
  "updated_at": "2020-07-17T18:59:22.891Z",
  "updated_by": "yo"
}
```

Output example of posting an exception list item
❯ ./post_exception_list_item.sh
```ts
{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "Wzk4NywxXQ==",
  "comments": [],
  "created_at": "2020-07-17T18:59:30.286Z",
  "created_by": "yo",
  "description": "This is a sample endpoint type exception",
  "entries": [
    {
      "field": "actingProcess.file.signer",
      "operator": "excluded",
      "type": "exists"
    },
    {
      "field": "host.name",
      "operator": "included",
      "type": "match_any",
      "value": [
        "some host",
        "another host"
      ]
    }
  ],
  "id": "a4f2b800-c85f-11ea-b1a6-c155df988a92",
  "item_id": "simple_list_item",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f",
  "type": "simple",
  "updated_at": "2020-07-17T18:59:30.304Z",
  "updated_by": "yo"
}
```

Output example of when you get an exception list:

❯ ./get_exception_list.sh simple_list
```ts
{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "WzEwNzcsMV0=",
  "created_at": "2020-07-17T18:59:22.872Z",
  "created_by": "yo",
  "description": "Different description",
  "id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
  "type": "endpoint",
  "updated_at": "2020-07-17T20:01:24.958Z",
  "updated_by": "yo"
}
```

Example of the error you get if you do an update of an exception list and someone else has changed it:
```ts
{
  "message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }",
  "status_code": 409
}
```

Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts:

```ts
./post_list.sh
./post_list_item.sh
./find_list.sh
./find_list_item.sh
```



### Checklist

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack API Integration Tests.x-pack/test/api_integration/apis/management/ingest_pipelines/ingest_pipelines·ts.apis management Ingest Node Pipelines Pipelines Delete should delete multiple pipelines

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook
[00:04:31]           └-: management
[00:04:31]             └-> "before all" hook
[00:04:59]             └-: Ingest Node Pipelines
[00:04:59]               └-> "before all" hook
[00:04:59]               └-: Pipelines
[00:04:59]                 └-> "before all" hook
[00:04:59]                 └-: Delete
[00:04:59]                   └-> "before all" hook
[00:04:59]                   └-> should delete a pipeline
[00:04:59]                     └-> "before each" hook: global before each
[00:04:59]                     └- ✓ pass  (60ms) "apis management Ingest Node Pipelines Pipelines Delete should delete a pipeline"
[00:04:59]                   └-> should delete multiple pipelines
[00:04:59]                     └-> "before each" hook: global before each
[00:05:00]                     └- ✖ fail: apis management Ingest Node Pipelines Pipelines Delete should delete multiple pipelines
[00:05:00]                     │       Error: expected [ { name: 'test_delete_pipeline_1',
[00:05:00]                     │     error: 
[00:05:00]                     │      { msg: '[resource_not_found_exception] pipeline [test_delete_pipeline_1] is missing',
[00:05:00]                     │        path: '/_ingest/pipeline/test_delete_pipeline_1',
[00:05:00]                     │        query: {},
[00:05:00]                     │        statusCode: 404,
[00:05:00]                     │        response: '{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"pipeline [test_delete_pipeline_1] is missing"}],"type":"resource_not_found_exception","reason":"pipeline [test_delete_pipeline_1] is missing"},"status":404}' } } ] to sort of equal []
[00:05:00]                     │       + expected - actual
[00:05:00]                     │ 
[00:05:00]                     │       -[
[00:05:00]                     │       -  {
[00:05:00]                     │       -    "error": {
[00:05:00]                     │       -      "msg": "[resource_not_found_exception] pipeline [test_delete_pipeline_1] is missing"
[00:05:00]                     │       -      "path": "/_ingest/pipeline/test_delete_pipeline_1"
[00:05:00]                     │       -      "query": {}
[00:05:00]                     │       -      "response": "{\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"pipeline [test_delete_pipeline_1] is missing\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"pipeline [test_delete_pipeline_1] is missing\"},\"status\":404}"
[00:05:00]                     │       -      "statusCode": 404
[00:05:00]                     │       -    }
[00:05:00]                     │       -    "name": "test_delete_pipeline_1"
[00:05:00]                     │       -  }
[00:05:00]                     │       -]
[00:05:00]                     │       +[]
[00:05:00]                     │       
[00:05:00]                     │       at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
[00:05:00]                     │       at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
[00:05:00]                     │       at Context.it (test/api_integration/apis/management/ingest_pipelines/ingest_pipelines.ts:276:27)
[00:05:00]                     │ 
[00:05:00]                     │

Stack Trace

{ Error: expected [ { name: 'test_delete_pipeline_1',
    error: 
     { msg: '[resource_not_found_exception] pipeline [test_delete_pipeline_1] is missing',
       path: '/_ingest/pipeline/test_delete_pipeline_1',
       query: {},
       statusCode: 404,
       response: '{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"pipeline [test_delete_pipeline_1] is missing"}],"type":"resource_not_found_exception","reason":"pipeline [test_delete_pipeline_1] is missing"},"status":404}' } } ] to sort of equal []
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/api_integration/apis/management/ingest_pipelines/ingest_pipelines.ts:276:27)
  actual:
   '[\n  {\n    "error": {\n      "msg": "[resource_not_found_exception] pipeline [test_delete_pipeline_1] is missing"\n      "path": "/_ingest/pipeline/test_delete_pipeline_1"\n      "query": {}\n      "response": "{\\"error\\":{\\"root_cause\\":[{\\"type\\":\\"resource_not_found_exception\\",\\"reason\\":\\"pipeline [test_delete_pipeline_1] is missing\\"}],\\"type\\":\\"resource_not_found_exception\\",\\"reason\\":\\"pipeline [test_delete_pipeline_1] is missing\\"},\\"status\\":404}"\n      "statusCode": 404\n    }\n    "name": "test_delete_pipeline_1"\n  }\n]',
  expected: '[]',
  showDiff: true }

Build metrics

page load bundle size

id value diff baseline
lists 256.7KB +503.0B 256.2KB

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@FrankHassanabad FrankHassanabad merged commit a20a807 into elastic:7.x Jul 20, 2020
@FrankHassanabad FrankHassanabad deleted the backport/7.x/pr-72337 branch July 20, 2020 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backport
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@FrankHassanabad @kibanamachine