Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Entity Store] [Asset Inventory] Universal entity definition (#202888)
## Summary This PR adds a universal entity definition. A universal entity uses `related.entity` as an identifier field and includes an extra processor step that parses the field `entities.keyword` and extracts all the entities in said field (whose original data comes from `related.entities`). See this [doc](https://docs.google.com/document/d/1D8xDtn3HHP65i1Y3eIButacD6ZizyjZZRJB7mxlXzQY/edit?tab=t.0#heading=h.9fz3qtlfzjg7) for more details. To accomplish this, we need to allow describing an entity along with extra entity store resources required for that entity's engine. This PR reworks the current entity store by introducing an `Entity Description`, which has all that required information. From it, we can build an `EntityEngineDescription` which adds all the needed data that must be computed (as opposed to hardcoded) and is then used to generate all the resources needed for that Entity's engine (entity definition, pipeline, enrich policy, index mappings, etc). <img width="3776" alt="EntityDescriptions" src="https://github.com/user-attachments/assets/bdf7915f-1981-47e6-a815-31163f24ad03"> This required a refactoring of the current `UnitedEntityDefinition`, which has now been removed in favour of more contextual functions for all the different parts. The intention is to decouple the Entity Description schema from the schemas required for field retention, entity manager and pipeline. We can then freely expand on our Entity Description as required, and simply alter the conversion functions when needed. ## How to test 1. On a fresh ES cluster, add some entity data * For hosts and user, use the [security documents generator](https://github.com/elastic/security-documents-generator) * For universal, there are a few more steps: 1. Create the `entity.keyword` builder pipeline 2. Add it to a index template 3. Post some docs to the corresponding index 2. Initialise the universal entity engine via: `POST kbn:/api/entity_store/engines/universal/init {}` * Note that using the UI does not work, as we've specifically removed the Universal engine from the normal Entity Store workflow 3. Check the status of the store is `running` via `GET kbn:/api/entity_store/status` 4. Once the transform runs, you can query `GET entities*/_search` to see the created entities Note that universal entities do not show up in the dashboard Entities List. ### Code to ingest data <details> <summary>Pipeline</summary> ```js PUT _ingest/pipeline/entities-keyword-builder { "description":"Serialize entities.metadata into a keyword field", "processors":[ { "script":{ "lang":"painless", "source":""" String jsonFromMap(Map map) { StringBuilder json = new StringBuilder("{"); boolean first = true; for (entry in map.entrySet()) { if (!first) { json.append(","); } first = false; String key = entry.getKey().replace("\"", "\\\""); Object value = entry.getValue(); json.append("\"").append(key).append("\":"); if (value instanceof String) { String escapedValue = ((String) value).replace("\"", "\\\"").replace("=", ":"); json.append("\"").append(escapedValue).append("\""); } else if (value instanceof Map) { json.append(jsonFromMap((Map) value)); } else if (value instanceof List) { json.append(jsonFromList((List) value)); } else if (value instanceof Boolean || value instanceof Number) { json.append(value.toString()); } else { // For other types, treat as string String escapedValue = value.toString().replace("\"", "\\\"").replace("=", ":"); json.append("\"").append(escapedValue).append("\""); } } json.append("}"); return json.toString(); } String jsonFromList(List list) { StringBuilder json = new StringBuilder("["); boolean first = true; for (item in list) { if (!first) { json.append(","); } first = false; if (item instanceof String) { String escapedItem = ((String) item).replace("\"", "\\\"").replace("=", ":"); json.append("\"").append(escapedItem).append("\""); } else if (item instanceof Map) { json.append(jsonFromMap((Map) item)); } else if (item instanceof List) { json.append(jsonFromList((List) item)); } else if (item instanceof Boolean || item instanceof Number) { json.append(item.toString()); } else { // For other types, treat as string String escapedItem = item.toString().replace("\"", "\\\"").replace("=", ":"); json.append("\"").append(escapedItem).append("\""); } } json.append("]"); return json.toString(); } def metadata = jsonFromMap(ctx['entities']['metadata']); ctx['entities']['keyword'] = metadata; """ } } ] } ``` </details> <details> <summary>Index template</summary> ```js PUT /_index_template/entity_store_index_template { "index_patterns":[ "logs-store" ], "template":{ "settings":{ "index":{ "default_pipeline":"entities-keyword-builder" } }, "mappings":{ "properties":{ "@timestamp":{ "type":"date" }, "message":{ "type":"text" }, "event":{ "properties":{ "action":{ "type":"keyword" }, "category":{ "type":"keyword" }, "type":{ "type":"keyword" }, "outcome":{ "type":"keyword" }, "provider":{ "type":"keyword" }, "ingested":{ "type": "date" } } }, "related":{ "properties":{ "entity":{ "type":"keyword" } } }, "entities":{ "properties":{ "metadata":{ "type":"flattened" }, "keyword":{ "type":"keyword" } } } } } } } ``` </details> <details> <summary>Example source doc</summary> ```js POST /logs-store/_doc/ { "@timestamp":"2024-11-29T10:01:00Z", "message":"Eddie", "event": { "type":[ "creation" ], "ingested": "2024-12-03T10:01:00Z" }, "related":{ "entity":[ "AKIAI44QH8DHBEXAMPLE" ] }, "entities":{ "metadata":{ "AKIAI44QH8DHBEXAMPLE":{ "entity":{ "id":"AKIAI44QH8DHBEXAMPLE", "category":"Access Management", "type":"AWS IAM Access Key" }, "cloud":{ "account":{ "id":"444455556666" } } } } } } ``` </details> ### To do - [x] Add/Improve [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) - [x] Feature flag ---- #### Update: Added `assetInventoryStoreEnabled` Feature Flag. It is disabled by default and even when enabled, the `/api/entity_store/enable` route does not initialize the Universal Entity Engine. `/api/entity_store/engines/universal/init` needs to be manually called to initialize it --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Rômulo Farias <[email protected]> Co-authored-by: jaredburgettelastic <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
- Loading branch information
5 people
authored
Jan 3, 2025
1 parent
e51b581
commit c6b0a31