[asset manager] merge obsasset signals collection #162222
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closes elastic#156757 Introduce background implicit collection in the asset_manager plugin. The process can be configured with the following kibana settings: ``` xpack.assetManager: implicitCollection: enabled: true interval: 30s # elasticsearch cluster we should extract signals from input: hosts: http://input:9200 username: ... password: ... # elasticsearch cluster we should write assets to output: hosts: http://output:9200 username: ... password: ... ``` --------- Co-authored-by: kibanamachine <[email protected]>
## Summary Closes elastic#157377 Records performances of implicit collection queries using apm transactions ### Testing - configure implicit collection in kibana settings ``` xpack.assetManager: implicitCollection: enabled: true interval: 30s # elasticsearch cluster we should extract signals from input: hosts: http://input:9200 username: ... password: ... # elasticsearch cluster we should write assets to output: hosts: http://output:9200 username: ... password: ... ``` - (for elastic maintainers) start kibana with `ELASTIC_APM_ACTIVE=true yarn start` - look for `transaction.type : "asset_manager-implicit_collection"` in dev cluster --------- Co-authored-by: Jason Rhodes <[email protected]>
## Summary Implicit collection stores segmentation values on `asset.kind` instead of on `asset.type`, like originally planned. This PR makes those changes so that `asset.kind` is a valid filter. It leaves `asset.type` in place for the moment.
Closes elastic#158864 This PR introduces a getHosts method and a GET /assets/hosts endpoint, both of which switch between querying from the assets indices vs. signals indices depending on the value of `xpack.assetManager.lockedSource`, a config value introduced in elastic#159338 . To test this PR, pull this branch and point it at an oblt-cli created cluster that uses cross-cluster search to read from the edge cluster. Then, add the following to your kibana.yml file: ```yml server.basePath: '/assets-demo' xpack.assetManager: alphaEnabled: true sourceIndices: metrics: remote_cluster:metricbeat*,remote_cluster:metrics-* logs: remote_cluster:filebeat*,remote_cluster:logs-* traces: remote_cluster:traces-* serviceMetrics: remote_cluster:metrics-apm* serviceLogs: remote_cluster:logs-apm* lockedSource: signals implicitCollection: enabled: true interval: 30s ``` Run the following curl request (assuming you have the `jq` utility installed, otherwise omit that part): ```sh $ curl -u {user}:{password} http://localhost:5601/assets-demo/api/asset-manager/assets/hosts | jq '.hosts[]."asset.id"' ``` which will print a list of the edge cluster's host assets, e.g. ```sh % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2549 100 2549 0 0 307 0 0:00:08 0:00:08 --:--:-- 625 [ "gke-edge-oblt-edge-oblt-pool-095c801b-8xp7" ] [ "gke-edge-oblt-edge-oblt-pool-095c801b-k4pz" ] [ "gke-edge-oblt-edge-oblt-pool-095c801b-jg87" ] [ "gke-edge-oblt-edge-oblt-pool-095c801b-xr9t" ] [ "opbeans-java-otel-867ddf766-8r7xv" ] [ "o11y-edge-windows" ] [ "o11y-edge-linux" ] [ "opbeans-php-5f765c854-8fh9v" ] ``` To test it against the implicitly-collected asset docs, change your kibana.yml file: ```diff - lockedSource: signals + lockedSource: assets ``` Perform the same curl request as above and you should receive a very similar response, although _much_ faster. --------- Co-authored-by: kibanamachine <[email protected]>
Closes elastic#157376 Adds pagination to implicit collector queries. To get pagination collapse queries now sort results by the same field we collapse on. Instead of returning the `n` most recent results we'll now get the `n` first hosts by hostname when querying signals. Baring the limited sorting flexibility this sounds like a sane contract for an API if we send back the page cursor so that consumers (implicit collector or kibana UI) can loop through, but we may send incomplete results when UI consumers rely on sliding time ranges (ie last 15mn). If that is a concern we could have the API accumulating pages in a single call and return the entire dataset but are there use cases for that ? Alternatively UI consumer can also do it by locking the time range, similar to implicit collection - create a host dataset with `n` cardinality. [slingshot](https://github.com/elastic/slingshot) can help - (optional) update [QUERY_MAX_SIZE](https://github.com/elastic/kibana/blob/feat/obs-asset-manager-demo/x-pack/plugins/asset_manager/server/lib/implicit_collection/collectors/index.ts#L13) to be less than `n` - run host collector and verify that paginated queries are executed to retrieve the full set in a single run --------- Co-authored-by: kibanamachine <[email protected]>
…lastic#162054) ## Summary Moving asset collectors up a level since they can also be used without periodic implicit collection wrapper
## Summary Remove unneeded apm span Part of cleanup work for elastic#161887
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
klacabane
commented
Jul 19, 2023
|
@elasticmachine merge upstream |
|
This looks good to me. The assets index template was created depending on the value of |
neptunian
approved these changes
Aug 15, 2023
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
History
To update your PR or re-run it, just comment with: |
|
@neptunian I've created two ftr configurations, one for each data source, and only enabled the |
klacabane
added
the
release_note:skip
klacabane
added
Team:Observed Asset Management
neptunian
approved these changes
Aug 16, 2023
kibanamachine
added
v8.11.0
backport:skip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Closes #161887
Merges most of the functionality from
feat/obs-asset-manager-demobranch. We remove implicit collection code while including:assetAccessorlogic that determines which indices to queryThe change also enables ftr test suite. We'll also merge the services endpoint #160294 when approved.