[Endpoint] ERT-82 ERT-83 ERT-84: Alert list API with pagination

[madirey]

Summary

This is the initial checkin of the Alert List API, which proxies a Kibana request through to Elasticsearch, returning a paginated set of alerts.

Technique used is simple pagination requiring a page index and page size. This will not scale with deep pagination through large result sets. A follow-up issue to implement cursor-based pagination can be found here: https://github.com/elastic/endpoint-app-team/issues/145 ...

Screenshot

Issues Addressed

https://github.com/elastic/endpoint-app-team/issues/82
https://github.com/elastic/endpoint-app-team/issues/83
https://github.com/elastic/endpoint-app-team/issues/84

Checklist

This PR supports the following:

GET /api/endpoint/alerts?page_size=2&page_index=2

... OR ...

POST /api/endpoint/alerts
{
      page_size: 2,
      page_index: 2
}

Example Response:

{
    "alerts": [
        {
            "@timestamp": 1542789423000,
            "agent": {
                "id": "5085268f-7443-4f15-85d2-bf14b2a69c60",
                "type": "endpoint",
                "version": "3.0.0"
            },
            "ecs": {
                "version": "1.1.0"
            },
            "event": {
                "action": "creation",
                "category": "malware",
                "dataset": "endpoint",
                "kind": "alert",
                "module": "endpoint",
                "type": "creation"
            },
            "file": {
                "accessed": 1542789400,
                "created": 1542789400,
                "hash": {
                    "imphash": "835d619dfdf3cc727cebd91300ab3462",
                    "md5": "4ace3baaa509d08510405e1b169e325b",
                    "sha1": "27fb21cf5db95ffca43b234affa99becc4023b9d",
                    "sha256": "6ed1c836dbf099be7845bdab7671def2c157643761b52251e04e9b6ee109ec75"
                },
                "mtime": 1542789400,
                "owner": "Administrators",
                "path": "C:\\Windows\\TEMP\\tmp0000008f\\tmp00001c56",
                "size": 188416
            },
            "file_classification": {
                "captured_file": false,
                "is_signature_trusted": false,
                "malware_classification": {
                    "compressed_malware_features": {
                        "data_buffer": "eAHtnU1oHHUUwHsQ7MGDiIIUD4sH8WBBxJtopiLoUY0pYo2ZTbJJ0yQ17m4+ms/NRzeVWpuUWCL4sWlEYvFQ8KJQ6NCTEA8eRD30sIo3PdSriLi7837Pko3LbHZ2M5m+XObHm/d/X////83O7jCZvzacHBpPplNdfalkdjSdyty674Ft59dN71Dpb9v5eKh8LMEHjsCF2wIfVlRKsHROYPGkQO5+gY2vBSYYdWZFYGwEO/cITHMqkxPYnBBY+07gtCuQ9gSGigJ5lPPYGXcE+jA4z3Ad1ZtAUiDUyrEEPYzqRnIKgxd/Rgc7gygPo5wn95PouN7OeEYJ1UXiJgRmvscgp/LOziIkkSyT+xRVnXhZ4DKh5goCkzidRHkGO4uvCyw9LDDtCay8ILCAzrJOJaGuZwUuvSewivJVIPsklq8JbL4qMJsTSCcExrGs83WKU295ZFo5lr2TaZbcUw5FeJy8tgTeLpCy2iGeS67ABXzlgbEi1UC5FxcZnA4y/CLK82Qxi847FGGZRTLsCUxR1aWEwOp1AmOjDRYYzgwusL9WfqBiGJxnVAanixTq7Dp22LBdlWMJzlOx8wmBK2Rx5WmBLJIRwtAijOQE+ooCb2B5xBOYRtlfNeXpLpA7oyZRTqHzGenkmIJPnhBIMrzTwSA6H93CO5l+c1NA99f6IwLH8fUKdjTmDpTbgS50+gGVnECnE4PpooC2guPoaPADSHrcncNHmEHtAFkq3+EI+A37zsrrTvH3WTkvJLoOTyBp10wx2JcgVCRahA4NrICE4a+hrMXsA3qAHItW188E8ejO7XV3eh/KCYwxlamEwCgL8lN2wTntfrhY/U0g/5KAdvUpT+AszWqBdqH7VLeeZrExK9Cv1UgIDKA8g/cx7QAEP+AhAfRaMKB2HOJh+BSFSqKjSytNGBlc6PrpxvK7lCVDxbSG3Z7AhCMwx6gelwgLAltXBXJUTH29j+U1LHdipx/QprfKfGnF0sBpdBYxmEQyTzW0h6/0khcuhhJYRufym+i4VKMocJMs/KvfoW3/UJb4PeZOSZVONThZz4djP/75TAXa/CVfOvX3RgVLIDreLPN1pP1osW7lGmHsEhjBOzf+EPBE4vndvWz5xb/cChxGcv1LAb+tluALKnZ47isf1MXvz1ZMlsCXbXtPceqhrcp1ps6YHwQeBXLEPCf7q23tl9uJui0bGBgYRAccv7uXr/g5Af+2oNTrpgTa/vnpjBvpLAwM4gRBPvIZGBgYGBgYGBgYGBgYGBgYGBgYGBgYNAOc9oMXs4GBgYFBcNBnww5QzDXgRtPSaZ5lg/itsRaslgZ3bnWEEVnhMetIBwiiVnlbCbWrEftrt11zdwWnseFW1QO63w1is3ptD1pV9xG0t+zvfUrzrvh380qwXWAVCw6h78GIfG7ZlzltXu6hd+y92fECRFhjuH3bXG8N43oXEHperdzvUbteaDxhVTUeq25fqhG1X6Ai8mtF6BDXz2wR+dzSgg4Qsxls5T11XMG+82y8GkG+b7kL69xg7mF1SFvhBgYGsYH/Xi7HE+PVkiB2jt1bNZxT+k4558jR53ydz5//1m1KOgYGBgYGBgYGEQfnsYaG2z1sdPJS79XQSu91ndobOAHCaN5vNzUk1bceQVzUpbw3iOuT+UFmR18bHrp3gyhDC56lCd1y85w2+HSNUwVhhdGC7blLf+bV/fqtvhMg1NDjCcugB1QXswbs8ekj/v1BgzFHBIIsyP+HfwFdMpzu",
                        "decompressed_size": 27831,
                        "encoding": "zlib"
                    },
                    "identifier": "endpointpe",
                    "prevention_threshold": 0.66,
                    "score": 1,
                    "threshold": 0.66,
                    "version": "3.0.33"
                },
                "signature_signer": "",
                "temp_file_path": "C:\\Windows\\TEMP\\dc20f6f7-abe9-4460-8b9a-ca16f7ab1972",
                "user_blacklisted": false
            },
            "host": {
                "hostname": "HD-ssm-0b0d26ad",
                "ip": "10.81.164.74",
                "name": "HD-ssm-0b0d26ad",
                "os": {
                    "name": "Windows",
                    "platform": "windows",
                    "version": "6.1"
                }
            },
            "process": {
                <TRUNCATED>
            },
            "thread": {
                "id": 1640,
                "service_name": "CybereasonAntiMalware",
                "start": 1542788400,
                "start_address": 8791698721056,
                "start_address_module": "C:\\Program Files\\Cybereason ActiveProbe\\gzfltum.dll"
            },
            "user": {
                "group": {}
            }
        },
        {
            "@timestamp": 1542789423000,
            "agent": {
                "id": "ced9c68e-b94a-4d66-bb4c-6106514f0a2f",
                "type": "endpoint",
                "version": "3.0.0"
            },
            "ecs": {
                "version": "1.1.0"
            },
            "event": {
                "action": "creation",
                "category": "malware",
                "dataset": "endpoint",
                "kind": "alert",
                "module": "endpoint",
                "type": "creation"
            },
            "file": {
                "accessed": 1542789400,
                "created": 1542789400,
                "hash": {
                    "imphash": "835d619dfdf3cc727cebd91300ab3462",
                    "md5": "4ace3baaa509d08510405e1b169e325b",
                    "sha1": "27fb21cf5db95ffca43b234affa99becc4023b9d",
                    "sha256": "6ed1c836dbf099be7845bdab7671def2c157643761b52251e04e9b6ee109ec75"
                },
                "mtime": 1542789400,
                "owner": "Administrators",
                "path": "C:\\Windows\\TEMP\\tmp0000008f\\tmp00001c56",
                "size": 188416
            },
            "file_classification": {
                "captured_file": false,
                "is_signature_trusted": false,
                "malware_classification": {
                    "compressed_malware_features": {
                        "data_buffer": "eAHtnU1oHHUUwHsQ7MGDiIIUD4sH8WBBxJtopiLoUY0pYo2ZTbJJ0yQ17m4+ms/NRzeVWpuUWCL4sWlEYvFQ8KJQ6NCTEA8eRD30sIo3PdSriLi7837Pko3LbHZ2M5m+XObHm/d/X////83O7jCZvzacHBpPplNdfalkdjSdyty674Ft59dN71Dpb9v5eKh8LMEHjsCF2wIfVlRKsHROYPGkQO5+gY2vBSYYdWZFYGwEO/cITHMqkxPYnBBY+07gtCuQ9gSGigJ5lPPYGXcE+jA4z3Ad1ZtAUiDUyrEEPYzqRnIKgxd/Rgc7gygPo5wn95PouN7OeEYJ1UXiJgRmvscgp/LOziIkkSyT+xRVnXhZ4DKh5goCkzidRHkGO4uvCyw9LDDtCay8ILCAzrJOJaGuZwUuvSewivJVIPsklq8JbL4qMJsTSCcExrGs83WKU295ZFo5lr2TaZbcUw5FeJy8tgTeLpCy2iGeS67ABXzlgbEi1UC5FxcZnA4y/CLK82Qxi847FGGZRTLsCUxR1aWEwOp1AmOjDRYYzgwusL9WfqBiGJxnVAanixTq7Dp22LBdlWMJzlOx8wmBK2Rx5WmBLJIRwtAijOQE+ooCb2B5xBOYRtlfNeXpLpA7oyZRTqHzGenkmIJPnhBIMrzTwSA6H93CO5l+c1NA99f6IwLH8fUKdjTmDpTbgS50+gGVnECnE4PpooC2guPoaPADSHrcncNHmEHtAFkq3+EI+A37zsrrTvH3WTkvJLoOTyBp10wx2JcgVCRahA4NrICE4a+hrMXsA3qAHItW188E8ejO7XV3eh/KCYwxlamEwCgL8lN2wTntfrhY/U0g/5KAdvUpT+AszWqBdqH7VLeeZrExK9Cv1UgIDKA8g/cx7QAEP+AhAfRaMKB2HOJh+BSFSqKjSytNGBlc6PrpxvK7lCVDxbSG3Z7AhCMwx6gelwgLAltXBXJUTH29j+U1LHdipx/QprfKfGnF0sBpdBYxmEQyTzW0h6/0khcuhhJYRufym+i4VKMocJMs/KvfoW3/UJb4PeZOSZVONThZz4djP/75TAXa/CVfOvX3RgVLIDreLPN1pP1osW7lGmHsEhjBOzf+EPBE4vndvWz5xb/cChxGcv1LAb+tluALKnZ47isf1MXvz1ZMlsCXbXtPceqhrcp1ps6YHwQeBXLEPCf7q23tl9uJui0bGBgYRAccv7uXr/g5Af+2oNTrpgTa/vnpjBvpLAwM4gRBPvIZGBgYGBgYGBgYGBgYGBgYGBgYGBgYNAOc9oMXs4GBgYFBcNBnww5QzDXgRtPSaZ5lg/itsRaslgZ3bnWEEVnhMetIBwiiVnlbCbWrEftrt11zdwWnseFW1QO63w1is3ptD1pV9xG0t+zvfUrzrvh380qwXWAVCw6h78GIfG7ZlzltXu6hd+y92fECRFhjuH3bXG8N43oXEHperdzvUbteaDxhVTUeq25fqhG1X6Ai8mtF6BDXz2wR+dzSgg4Qsxls5T11XMG+82y8GkG+b7kL69xg7mF1SFvhBgYGsYH/Xi7HE+PVkiB2jt1bNZxT+k4558jR53ydz5//1m1KOgYGBgYGBgYGEQfnsYaG2z1sdPJS79XQSu91ndobOAHCaN5vNzUk1bceQVzUpbw3iOuT+UFmR18bHrp3gyhDC56lCd1y85w2+HSNUwVhhdGC7blLf+bV/fqtvhMg1NDjCcugB1QXswbs8ekj/v1BgzFHBIIsyP+HfwFdMpzu",
                        "decompressed_size": 27831,
                        "encoding": "zlib"
                    },
                    "identifier": "endpointpe",
                    "prevention_threshold": 0.66,
                    "score": 1,
                    "threshold": 0.66,
                    "version": "3.0.33"
                },
                "signature_signer": "",
                "temp_file_path": "C:\\Windows\\TEMP\\dc20f6f7-abe9-4460-8b9a-ca16f7ab1972",
                "user_blacklisted": false
            },
            "host": {
                "hostname": "HD-c15-bc09190a",
                "ip": "10.179.244.14",
                "name": "HD-c15-bc09190a",
                "os": {
                    "name": "Windows",
                    "platform": "windows",
                    "version": "6.1"
                }
            },
            "process": {
                <TRUNCATED>
            },
            "thread": {
                "id": 1640,
                "service_name": "CybereasonAntiMalware",
                "start": 1542788400,
                "start_address": 8791698721056,
                "start_address_module": "C:\\Program Files\\Cybereason ActiveProbe\\gzfltum.dll"
            },
            "user": {
                "group": {}
            }
        }
    ],
    "request_page_index": 2,
    "request_page_size": 2,
    "result_from_index": 4,
    "total": 132
}

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

• This was checked for cross-browser compatibility, including a check against IE11
• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for keyboard-only and screenreader accessibility

For maintainers

- [ ] This was checked for breaking API changes and was labeled appropriately
- [ ] This includes a feature addition or change that requires a release note and was labeled appropriately

Follow-Up Issues

• https://github.com/elastic/endpoint-app-team/issues/145

[madirey]

@oatkiller Reduced sample data size by more than half...

[madirey]

Functional tests added.

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 2d0843a0a8bd6a2580fe91fc8ca44ca4cfbb39af

History

• 💛 Build #24682 was flaky be63e4a3759edbe01b7b03456eb200c48c581f8e
• 💚 Build #24505 succeeded e6cac13
• 💚 Build #24408 succeeded 28f7baa
• 💔 Build #24397 failed 426502c
• 💔 Build #24220 failed bb37b8d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: d513a43

History

• 💚 Build #25039 succeeded 2d0843a0a8bd6a2580fe91fc8ca44ca4cfbb39af
• 💛 Build #24682 was flaky be63e4a3759edbe01b7b03456eb200c48c581f8e
• 💚 Build #24505 succeeded e6cac13
• 💚 Build #24408 succeeded 28f7baa
• 💔 Build #24397 failed 426502c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream