[Entity Analytics] Internal API to upload asset criticality records via CSV file

[hop-dev]

Summary

Adds a new internal API to provide a CSV file of asset criticality records to bulk upload.

The CSV has some constraints:

• It must use a comma as the delimiter
• its must have 3 columns in the following order:
  • entity type: host or user
  • entity ID:any string up to 1000 characters
  • criticality level: one of low_impact, medium_impact, high_impact, extreme_impact
• we do not support header rows at the moment

Here is an example of a good file:

host,host-01,low_impact
host,host-02,medium_impact
host,host-03,high_impact
user,user-04,extreme_impact

We send the updates to elasticsearch using the bulk API stream helper, this allows us to configure the max request size and the number of retries on error. I have mapped these to config just incase we ever need to change them for a certain env. Here are the keys and their defaults:

xpack.securitySolution.entityAnalytics.assetCriticality.csvUpload.errorRetries: 1
xpack.securitySolution.entityAnalytics.assetCriticality.csvUpload.maxBulkRequestBodySizeBytes: 100000

Example Curl with response:

curl -X POST -F "file=@./1k.csv;type=text/csv" http: //<kibana>/internal/asset_criticality/upload_csv -H "kbn-xsrf:hello" -H "elastic-api-version:1"
{
    "errors": [],
    "stats": {
        "successful": 1000,
        "failed": 0,
        "total": 1000
    }
}

Errors are reported line by line, here are some examples:

{
    "errors": [
        {
            "index": 0,
            "message": "Invalid criticality level invalid_criticality"
        },
        {
            "index": 1,
            "message": "Invalid entity type invalid_entity"
        },
        {
            "index": 2,
            "message": "Missing ID"
        },
        {
            "index": 3,
            "message": "Missing criticality level"
        },
    ]
}

[maximpn]

Detection and Response LGTM

@hop-dev The changes here aren't really owned by Detection and Response. I left comments about moving the code from common files to entity analytics domain where ownership is limited to entity analytics.

[maximpn]

While it's possible to register individual routes here it's much better to incapsulate the details. I recommend to move all assetCriticality... route registrations into registerAssetCriticalityRoutes() function. This function file should have explicit ownership.

[maximpn]

I recommend to move ASSET_CRITICALITY... constants into Asset criticality domain. Create common/asset_criticality folder with constants.ts. It will help to define proper ownership avoid expanding common/constants.ts with not really common constants.

[hop-dev]

@maximpn thanks for the really useful recommendations, I don't want to grow this PR any more so I have created an issue for our team to implement them separately soon 👍 #180531

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 539ecc7

Failed CI Steps

• FTR Configs #72

Test Failures

• [job] [logs] FTR Configs #72 / transform /internal/transform/schedule_now_transforms bulk schedule "after each" hook for "should schedule multiple transforms by transformIds, even if one of the transformIds is invalid"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	5302	5303	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	17.0MB	17.0MB	+18.0B

History

• 💚 Build #202593 succeeded 8630d07
• 💚 Build #202444 succeeded 9a70627
• 💔 Build #202224 failed feb235b
• 💛 Build #202094 was flaky 46fb379
• 💔 Build #202065 failed e391d0f
• 💔 Build #202051 failed 5ee0498

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @hop-dev

[JDKurma]

SDA LGTM

[tiansivive]

LGTM