Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Asset Manager] Run implicit collection periodically #156757

Closed
Tracked by #157375
klacabane opened this issue May 4, 2023 · 0 comments · Fixed by #156830
Closed
Tracked by #157375

[Asset Manager] Run implicit collection periodically #156757

klacabane opened this issue May 4, 2023 · 0 comments · Fixed by #156830
Assignees
@klacabane
Labels
enhancement New value added to drive a business result Feature:Asset Manager Team:Observed Asset Management Label used for engineers working on various parts of observed asset management

Comments

@klacabane
Copy link
Contributor

Summary

#155614 follow up

A working etl job transforming signals into asset documents is available in this change. We're now looking into a way to run this job automatically, in a frictionless way (read without onboarding existing mechanisms).
One approach is to execute the job on a fire-and-forget interval in the asset manager plugin. This will keep the logic contained in our alpha plugin and limit the amount of dependencies to get something up and running quickly. We can port the etl function, adapting anything that is relevant along the way.

Note: we are not looking at a production-ready implementation, but a prototype that allows populating assets-* indices periodically and testing/assessing quality of the data

AC

  • implicit collection runs periodically when asset-manager plugin is enabled
  • implicit collection period is configurable
  • implicit collection should get data from the elasticsearch it is connected to by default but this should be configurable
  • (optional) each query running time is logged (plus any relevant metrics like no of documents). Can we use apm agent ?
@klacabane klacabane added enhancement New value added to drive a business result Team:Observed Asset Management Label used for engineers working on various parts of observed asset management Feature:Asset Manager labels May 4, 2023
@klacabane klacabane self-assigned this May 5, 2023
This was referenced May 5, 2023
Merged
Closed
klacabane added a commit that referenced this issue May 11, 2023
@klacabane @kibanamachine
[Asset Manager] run implicit collection periodically (#156830)
19c222a 
## Summary

Closes #156757

Introduce background implicit collection in the asset_manager plugin.
The process can be configured with the following kibana settings:

```
xpack.assetManager:
  implicitCollection:
    enabled: true
    interval: 30s
    
    # elasticsearch cluster we should extract signals from
    input:
      hosts: http://input:9200
      username: ...
      password: ...
    
    # elasticsearch cluster we should write assets to
    output:
      hosts: http://output:9200
      username: ...
      password: ...
```

---------

Co-authored-by: kibanamachine <[email protected]>
klacabane added a commit to klacabane/kibana that referenced this issue Jul 19, 2023
@klacabane @kibanamachine
[Asset Manager] run implicit collection periodically (elastic#156830)
0b23894 
Closes elastic#156757

Introduce background implicit collection in the asset_manager plugin.
The process can be configured with the following kibana settings:

```
xpack.assetManager:
  implicitCollection:
    enabled: true
    interval: 30s

    # elasticsearch cluster we should extract signals from
    input:
      hosts: http://input:9200
      username: ...
      password: ...

    # elasticsearch cluster we should write assets to
    output:
      hosts: http://output:9200
      username: ...
      password: ...
```

---------

Co-authored-by: kibanamachine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@klacabane klacabane

Labels
enhancement New value added to drive a business result Feature:Asset Manager Team:Observed Asset Management Label used for engineers working on various parts of observed asset management
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Asset Manager] run implicit collection periodically klacabane/kibana
1 participant
@klacabane