Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[data views] telemetry is too resource intensive, skip field loading #151248

Closed
mattkime opened this issue Feb 15, 2023 · 4 comments
Closed

[data views] telemetry is too resource intensive, skip field loading #151248

mattkime opened this issue Feb 15, 2023 · 4 comments
Assignees
Labels
bug Fixes for quality problems that affect the customer experience Feature:Data Views Data Views code and UI - index patterns before 8.0 Feature:Kibana Management Feature label for Data Views, Advanced Setting, Saved Object management pages Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL.

Comments

@mattkime
Copy link
Contributor

From #151064

Some customers have thousands of data views and thousands of fields so running telemetry can be very resource intensive. Lets skip loading fields as the primary culprit is likely json.parse which can block the main thread when parsing large json.

src/plugins/data_views/server/register_index_pattern_usage_collection.ts:85 - loads full data views even though just scripted and runtime fields are needed.

x-pack/plugins/maps/server/maps_telemetry/index_pattern_stats/index_pattern_stats_collector.ts - only needs geo fields

@mattkime mattkime added bug Fixes for quality problems that affect the customer experience Feature:Data Views Data Views code and UI - index patterns before 8.0 Feature:Kibana Management Feature label for Data Views, Advanced Setting, Saved Object management pages Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. labels Feb 15, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@kertal
Copy link
Member

kertal commented Feb 15, 2023

+1 one for optional & selective data view fields fetching, also other areas in Kibana using data views would benefit from that.
On top of that, much more effort, but also something to think about, combining data views, fetch fields with one request, this could improve the following stats collection:

await asyncForEach(indexPatternIds, async (indexPatternId) => {

@mattkime
Copy link
Contributor Author

I'm hoping this is the start of a proposal to address this - #152159

@kertal
Copy link
Member

kertal commented Apr 13, 2023

@mattkime I guess we can close this, right?

@kertal kertal closed this as completed Apr 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Feature:Data Views Data Views code and UI - index patterns before 8.0 Feature:Kibana Management Feature label for Data Views, Advanced Setting, Saved Object management pages Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL.
Projects
None yet
Development

No branches or pull requests

3 participants