[Usage Collection] Report nodes feature usage #70108

TinaHeiligers · 2020-06-26T18:19:06Z

Summary

This PR allows us to track different types of aggregations performed on cluster nodes as well as determine the rest actions that were issued. Note that this PR only handles the case of usage data being collected locally. The case for when monitoring is enabled will be addressed in a different issue.

The original request was to add aggregations to the rest_actions from the nodes feature usage data but we weren't reporting any of that data.
The data is reported pretty much as-is, although the structure is modified slightly from the raw response.

raw response:

{
  cluster_name: 'test cluster',
  nodes: {
    some_node_id: {
      timestamp: 1588617023177,
      since: 1588616945163,
      rest_actions: {
        nodes_usage_action: 1,
        ...
      },
      aggregations: {
        terms: {
          bytes: 2,
        },
        scripted_metric: {
          other: 7,
        },
        ...
      }
    }
  }
}

format of response reported _within `cluster_stats.nodes`_:

usage: [
  {
    node_id: 'some_node_id',
    timestamp: 1588617023177,
    since: 1588616945163,
    rest_actions: {
      nodes_usage_action: 1,
      ...
    },
    aggregations: {
      terms: {
        bytes: 2,
      },
      scripted_metric: {
        other: 7,
      },
      ...
    }
  },
  {
    node_id: 'some_other_node',
    ...
  }
]

screenshots:

Preview of part of the response from POST /api/telemetry/v2/clusters/_stats:

How it renders in the UI (note that this is a one-node cluster running locally):

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

…local_stats

elasticmachine · 2020-06-26T18:19:09Z

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

TinaHeiligers · 2020-06-26T18:23:47Z

cc @mindbat this PR adds the response from a GET request to /_nodes/usage for a cluster and reports it un-aggregated. We will need to aggregate the data in the indexing layer, as discussed in #68603 (comment).

TinaHeiligers · 2020-06-26T18:31:32Z

@imotov @polyfractal, here's the PR for adding nodes feature usage aggregations (and rest actions) to the usage collection payload. The work only covers the case for when usage collection is local, i.e. monitoring is not enabled.
The data reported is unaggregated.

afharo

It's looking great! I noticed a couple of gotchas. But happy to approve it if they are intended 👍

src/plugins/telemetry/server/telemetry_collection/get_local_stats.ts

src/plugins/telemetry/server/telemetry_collection/get_nodes_usage.test.ts

src/plugins/telemetry/server/telemetry_collection/get_nodes_usage.ts

...collection_xpack/server/telemetry_collection/__snapshots__/get_stats_with_xpack.test.ts.snap

…ats.ts Using native merge without mutating the original object. Co-authored-by: Alejandro Fernández Haro <[email protected]>

…-usage-local-collection

kibanamachine · 2020-06-29T21:14:38Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: e43b412
Flaky suites:
- kibana-ciGroup1

Test Failures

Kibana Pipeline / kibana-oss-agent / Plugin Functional Tests.test/plugin_functional/test_suites/core_plugins/logging·ts.core plugins plugin logging writes info_json context to custom JSON appender

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/70151

[00:00:00]       │
[00:01:15]         └-: core plugins
[00:01:15]           └-> "before all" hook
[00:03:46]           └-: plugin logging
[00:03:46]             └-> "before all" hook
[00:03:46]             └-> does not write to custom appenders when not configured
[00:03:46]               └-> "before each" hook: global before each
[00:03:46]               └-> "before each" hook
[00:03:46]               │ proc [kibana]   log   [19:53:30.759] [info][core_logging][debug_json][plugins] i go to the default appender!
[00:03:46]               └- ✓ pass  (71ms) "core plugins plugin logging does not write to custom appenders when not configured"
[00:03:46]             └-> writes debug_json context to custom JSON appender
[00:03:46]               └-> "before each" hook: global before each
[00:03:46]               └-> "before each" hook
[00:03:46]               └- ✓ pass  (50ms) "core plugins plugin logging writes debug_json context to custom JSON appender"
[00:03:46]             └-> writes info_json context to custom JSON appender
[00:03:46]               └-> "before each" hook: global before each
[00:03:46]               └-> "before each" hook
[00:03:46]               │ info Taking screenshot "/dev/shm/workspace/kibana/test/functional/screenshots/failure/core plugins plugin logging writes info_json context to custom JSON appender.png"
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [dev_tools] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[INFO] http://localhost:6121/9007199254740991/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-06-29T19:53:31Z
[00:03:47]               │        Adding connection to http://localhost:6121/elasticsearch
[00:03:47]               │
[00:03:47]               │      "
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [management] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [rendering] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [custom-app-route] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [chromeless] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [foo] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [bar] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [appleave1] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[WARNING] http://localhost:6121/9007199254740991/bundles/core_legacy_compat.bundle.js 2495:20 "App [appleave2] is using deprecated mount context. Use core.getStartServices() instead."
[00:03:47]               │ debg browser[DEBUG] http://localhost:6121/9007199254740991/bundles/plugin/usageCollection/usageCollection.plugin.js 950:16 "Reporting user-agent."
[00:03:47]               │ debg browser[DEBUG] http://localhost:6121/9007199254740991/bundles/plugin/usageCollection/usageCollection.plugin.js 950:16 "Reporting application changed to core_legacy_compat"
[00:03:47]               │ info Current URL is: http://localhost:6121/app/core_legacy_compat
[00:03:47]               │ info Saving page source to: /dev/shm/workspace/kibana/test/plugin_functional/failure_debug/html/core plugins plugin logging writes info_json context to custom JSON appender.html
[00:03:47]               └- ✖ fail: "core plugins plugin logging writes info_json context to custom JSON appender"
[00:03:47]               │

Stack Trace

{ Error: expected [] to sort of equal [ { level: 'INFO',
    context: 'plugins.core_logging.info_json',
    message: 'log2' } ]
    at Assertion.assert (packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (packages/kbn-expect/expect.js:244:8)
    at Context.it (test/plugin_functional/test_suites/core_plugins/logging.ts:96:40)
  actual: '[]',
  expected:
   '[\n  {\n    "context": "plugins.core_logging.info_json"\n    "level": "INFO"\n    "message": "log2"\n  }\n]',
  showDiff: true }

Build metrics

✅ unchanged

History

💔 Build #57258 failed 5acbcdb
💚 Build #57235 succeeded a6bd9bd
💔 Build #57178 failed 853710b
💚 Build #56862 succeeded 345310c
💔 Build #56807 failed 048bfe5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

afharo

LGTM!

afharo · 2020-06-30T12:29:40Z

src/plugins/telemetry/server/telemetry_collection/get_nodes_usage.ts

-  }
+  const result = await fetchNodesUsage(callCluster);
+  const transformedNodes = Object.entries(result?.nodes || {}).map(([key, value]) => ({
+    ...(value as NodeObj),


NIT: I think there is no need for the cast now that fetchNodesUsage is properly typed :)

* Adds nodes feature usage stats merged into cluster_stats.nodes when usage collection is local

* master: [ML] Modifies page title to Create job (elastic#70191) [APM] Add API test for service maps (elastic#70185) [DOCS] Adds glossary to documentation (elastic#69721) [Usage Collection] Report nodes feature usage (elastic#70108) chore: improve support for mjs file extension (elastic#70186) [ML] Anomaly Detection: ensure 'Category examples' tab in the expanded table row can be seen (elastic#70241) [Maps] Add maps telemetry saved object in with mappings disabled (elastic#69995) Fix typo in bootstrap command (elastic#69976) [code coverage] ingest correct coveredFilePath for mocha (elastic#70215) [Dashboard] Add visualization by value to dashboard (elastic#69898) updates wording in Cases connectors (elastic#70298) [ML] Fix license subscription race condition. (elastic#70074) [Logs UI] [Alerting] "Group by" functionality (elastic#68250)

* Adds nodes feature usage stats merged into cluster_stats.nodes when usage collection is local

elasticmachine · 2021-10-01T21:40:39Z

Pinging @elastic/kibana-core (Team:Core)

TinaHeiligers added 3 commits June 26, 2020 09:48

Adds nodes feature usage stats merged into cluster_stats.nodes

de25409

Adds jest unit test for get_nodes_usage, adds to mocha tests for get_…

c75b23e

…local_stats

adds assertion for nodes usage in cluster stats api integration test

048bfe5

TinaHeiligers added Feature:Telemetry v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:KibanaTelemetry v7.10.0 labels Jun 26, 2020

TinaHeiligers requested review from afharo and Bamieh June 26, 2020 18:19

TinaHeiligers requested a review from a team as a code owner June 26, 2020 18:19

TinaHeiligers removed request for afharo and Bamieh June 26, 2020 18:21

TinaHeiligers added 2 commits June 26, 2020 13:04

resolves linter problem

3e62592

Handles unit test failures

345310c

afharo reviewed Jun 29, 2020

View reviewed changes

TinaHeiligers and others added 6 commits June 29, 2020 07:39

Update src/plugins/telemetry/server/telemetry_collection/get_local_st…

4b7e9a1

…ats.ts Using native merge without mutating the original object. Co-authored-by: Alejandro Fernández Haro <[email protected]>

review comments addressed

73bb233

removes unused merge import

853710b

Updates test snapshots

9a0962d

removes unused var

a6bd9bd

Merge branch 'master' of github.com:elastic/kibana into nodes-feature…

5acbcdb

…-usage-local-collection

TinaHeiligers requested a review from afharo June 29, 2020 17:52

Updates type

e43b412

afharo approved these changes Jun 30, 2020

View reviewed changes

TinaHeiligers removed the v7.10.0 label Jun 30, 2020

TinaHeiligers added the v7.9.0 label Jun 30, 2020

TinaHeiligers merged commit 93ef5c0 into elastic:master Jun 30, 2020

TinaHeiligers mentioned this pull request Jun 30, 2020

[7.x] [Usage Collection] Report nodes feature usage (#70108) #70329

Merged

TinaHeiligers added a commit that referenced this pull request Jun 30, 2020

[Usage Collection] Report nodes feature usage (#70108) (#70329)

6795a4c

* Adds nodes feature usage stats merged into cluster_stats.nodes when usage collection is local

TinaHeiligers deleted the nodes-feature-usage-local-collection branch June 30, 2020 16:20

Bamieh pushed a commit to Bamieh/kibana that referenced this pull request Jul 1, 2020

[Usage Collection] Report nodes feature usage (elastic#70108)

082cc3c

* Adds nodes feature usage stats merged into cluster_stats.nodes when usage collection is local

lukeelmers added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Oct 1, 2021

lukeelmers removed the Team:KibanaTelemetry label Oct 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Usage Collection] Report nodes feature usage #70108

[Usage Collection] Report nodes feature usage #70108

TinaHeiligers commented Jun 26, 2020 •

edited

Loading

elasticmachine commented Jun 26, 2020

TinaHeiligers commented Jun 26, 2020

TinaHeiligers commented Jun 26, 2020

afharo left a comment

kibanamachine commented Jun 29, 2020

Standard Out

Stack Trace

afharo left a comment

afharo Jun 30, 2020

elasticmachine commented Oct 1, 2021

[Usage Collection] Report nodes feature usage #70108

[Usage Collection] Report nodes feature usage #70108

Conversation

TinaHeiligers commented Jun 26, 2020 • edited Loading

Summary

screenshots:

Checklist

For maintainers

elasticmachine commented Jun 26, 2020

TinaHeiligers commented Jun 26, 2020

TinaHeiligers commented Jun 26, 2020

afharo left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 29, 2020

💛 Build succeeded, but was flaky

Test Failures

Standard Out

Stack Trace

Build metrics

History

afharo left a comment

Choose a reason for hiding this comment

afharo Jun 30, 2020

Choose a reason for hiding this comment

elasticmachine commented Oct 1, 2021

TinaHeiligers commented Jun 26, 2020 •

edited

Loading