Stats API: implement the "kibana status" spec from the Monitoring data model for stats #20577

tsullivan · 2018-07-09T18:00:10Z

Addresses #20393

The /api/stats?extended=true API endpoint returns data in the same model as the kibana_stats documents in the .monitoring-kibana-* indices
The API data shares some metrics modules with the /api/status API endpoint, which will be marked as "deprecated" in 7.0 and removed in 8.0
Where the stats API needs data in a different format than the shared modules are providing, the workarounds to fix the format are done in the older status API code to continue legacy backwards compatibility

No release notes yet, but there should documentation for this API in the next major version.

Usage:

Get Kibana server system status metrics:
```
GET /api/stats
```
Get Kibana server system status metrics, along with the Elasticsearch clusterUuid, and plugin usage metrics. This will require authentication if Elasticsearch requires it:
```
GET /api/stats?extended=true
```

tsullivan · 2018-07-10T23:55:49Z

src/server/status/metrics_collector/metrics.js

      },
      response_times: {
-        // TODO: rename to use `_ms` suffix per beats naming conventions


collectorSet.toApiFieldNames() takes care of this TODO

- constantly accumulating stats over time does not align with the existing behavior, which is to reset the stats to 0 whenever they are pulled

- change the signature of sourceKibana

- to not clear the data when pulling via the api - fetching is a read-only thing

tsullivan · 2018-07-12T23:54:10Z

x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.combine_stats_legacy.test.js

@@ -69,11 +69,13 @@ const getInitial = () => {
  ];
 };

+// TODO use jest snapshotting


This TODO will not be done in this PR to avoid extra lines of code change. This PR does change this from a Mocha test to a Jest test though.

…st duplicates

tsullivan · 2018-07-18T19:31:34Z

CI passing and going green looks like a fluke. Jenkins has downtime right now. I'll run it when I hear that it's ready.

tsullivan · 2018-07-18T20:42:50Z

jenkins, test this

tsullivan · 2018-07-18T22:01:49Z

Had a failed build from:

 └- ✖ fail: "discover app discover app query save query should show toast message and display query name"
21:28:34    │ proc  [ftr]        │        Error: expected '' to equal 'Query # 1'
21:28:34    │ proc  [ftr]

ycombinator

LGTM!

elasticmachine · 2018-07-18T23:18:33Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

tsullivan · 2018-07-18T23:48:05Z

Thanks 1,000,000 @ycombinator and @chrisronline for helping me review and test this!

tsullivan · 2018-07-18T23:51:31Z

6.x/6.4.0: #20956

elasticmachine · 2018-07-19T01:50:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

…a model for stats (elastic#20577) * [Stats API] Set API field names per spec * fix jest tests * fix api integration test * trash the original metrics collector - constantly accumulating stats over time does not align with the existing behavior, which is to reset the stats to 0 whenever they are pulled * move some logic out of the collector types combiner into inline - change the signature of sourceKibana * Make a new stats collector for the API - to not clear the data when pulling via the api - fetching is a read-only thing * isolate data transforms for api data and upload data * no static methods * remove external in bytes * remove the _stats prefix for kibana and reporting * update jest test snapshot * fix collector_types_combiner test * fix usage api * add test suite todo comment * reduce some loc change * roll back mysterious change * reduce some more loc change * comment correction * reduce more loc change * whitespace * comment question * fix cluster_uuid * fix stats integration test * fix bulk uploader test, combineTypes is no longer external * very important comments about the current nature of stats represented and long-term goals * add stats api tests with/without authentication * fix more fields to match data model * fix more tests * fix jest test * remove TODO * remove sockets * use snake_case for api field names * restore accidental removal + copy/paste error * sourceKibana -> getKibanaInfoForStats * skip usage test on legacy endpoint * fix api tests * more comment * stop putting a field in that used to be omitted * fix the internal type to ID the usage data for bulk uploader * correct the kibana usage type value, which is shown as-is in the API * more fixes for the constants identifying collector types + test against duplicates * add a comment on a hack, and a whitespace fix

…a model for stats (#20577) (#20956) * [Stats API] Set API field names per spec * fix jest tests * fix api integration test * trash the original metrics collector - constantly accumulating stats over time does not align with the existing behavior, which is to reset the stats to 0 whenever they are pulled * move some logic out of the collector types combiner into inline - change the signature of sourceKibana * Make a new stats collector for the API - to not clear the data when pulling via the api - fetching is a read-only thing * isolate data transforms for api data and upload data * no static methods * remove external in bytes * remove the _stats prefix for kibana and reporting * update jest test snapshot * fix collector_types_combiner test * fix usage api * add test suite todo comment * reduce some loc change * roll back mysterious change * reduce some more loc change * comment correction * reduce more loc change * whitespace * comment question * fix cluster_uuid * fix stats integration test * fix bulk uploader test, combineTypes is no longer external * very important comments about the current nature of stats represented and long-term goals * add stats api tests with/without authentication * fix more fields to match data model * fix more tests * fix jest test * remove TODO * remove sockets * use snake_case for api field names * restore accidental removal + copy/paste error * sourceKibana -> getKibanaInfoForStats * skip usage test on legacy endpoint * fix api tests * more comment * stop putting a field in that used to be omitted * fix the internal type to ID the usage data for bulk uploader * correct the kibana usage type value, which is shown as-is in the API * more fixes for the constants identifying collector types + test against duplicates * add a comment on a hack, and a whitespace fix

tsullivan mentioned this pull request Jul 9, 2018

Provide a Kibana Stats API that can be consumed by a Kibana Metricbeat metricset giving 1:1 parity with current Kibana Monitoring data #20393

Closed

4 tasks

tsullivan force-pushed the usage/stats-api-fixes branch from 7395c8c to ba1d0e5 Compare July 10, 2018 23:53

elastic deleted a comment from elasticmachine Jul 10, 2018

tsullivan commented Jul 10, 2018

View reviewed changes

tsullivan force-pushed the usage/stats-api-fixes branch 2 times, most recently from 503ac41 to ca5f2fa Compare July 10, 2018 23:58

tsullivan added 2 commits July 11, 2018 09:30

[Stats API] Set API field names per spec

cd83e73

fix jest tests

acba93c

tsullivan force-pushed the usage/stats-api-fixes branch from ca5f2fa to acba93c Compare July 11, 2018 17:04

tsullivan added 3 commits July 11, 2018 16:08

fix api integration test

3478ed4

trash the original metrics collector

12718bc

- constantly accumulating stats over time does not align with the existing behavior, which is to reset the stats to 0 whenever they are pulled

move some logic out of the collector types combiner into inline

d59cd4d

- change the signature of sourceKibana

ycombinator mentioned this pull request Jul 12, 2018

Create (X-Pack Monitoring) stats metricset for Kibana module elastic/beats#7525

Merged

tsullivan added 3 commits July 12, 2018 10:38

Make a new stats collector for the API

027ef25

- to not clear the data when pulling via the api - fetching is a read-only thing

isolate data transforms for api data and upload data

b7f0766

no static methods

fb05f20

alexfrancoeur mentioned this pull request Jul 12, 2018

Add sample data telemetry #19319

Closed

tsullivan added 7 commits July 12, 2018 15:06

remove external in bytes

4fe1c9d

remove the _stats prefix for kibana and reporting

c1e8226

update jest test snapshot

d162284

fix collector_types_combiner test

489be42

Merge branch 'master' into usage/stats-api-fixes

6e36b3c

fix usage api

817314c

add test suite todo comment

dfb901b

elastic deleted a comment from elasticmachine Jul 12, 2018

tsullivan added 4 commits July 12, 2018 16:45

reduce some loc change

48d4987

roll back mysterious change

9be60c9

reduce some more loc change

244627f

comment correction

81f502b

tsullivan commented Jul 12, 2018

View reviewed changes

elastic deleted a comment from elasticmachine Jul 18, 2018

tsullivan added 2 commits July 18, 2018 11:24

more fixes for the constants identifying collector types + test again…

79111eb

…st duplicates

add a comment on a hack, and a whitespace fix

528cc0d

Merge branch 'master' into usage/stats-api-fixes

eebed0c

elastic deleted a comment from elasticmachine Jul 18, 2018

ycombinator approved these changes Jul 18, 2018

View reviewed changes

tsullivan merged commit 274617d into elastic:master Jul 18, 2018

tsullivan deleted the usage/stats-api-fixes branch July 18, 2018 23:37

tsullivan mentioned this pull request Jul 18, 2018

[6.x] Stats API: implement the "kibana status" spec from the Monitoring data model for stats (#20577) #20956

Merged

1 task

elastic deleted a comment from elasticmachine Jul 19, 2018

This was referenced Jul 20, 2018

Updates status page to reflect /api/stats changes #21055

Merged

[Status API] When the instance is running on Windows, the Status API should remove the OS Load fields instead of returning them as 0 #17548

Open

tsullivan mentioned this pull request Jul 25, 2018

[Monitoring] Retrofit the Bulk Uploader types combiner to require less maintenance as metrics/telemetry KPIs grow #21238

Closed

alexfrancoeur mentioned this pull request Aug 15, 2018

[Telemetry] Count of Visualization by Type #22010

Closed

alexfrancoeur mentioned this pull request Aug 30, 2018

Localization Telemetry #22532

Closed

Bamieh mentioned this pull request Nov 23, 2021

[telemetry] Stats api is slow on M1 chips #119477

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stats API: implement the "kibana status" spec from the Monitoring data model for stats #20577

Stats API: implement the "kibana status" spec from the Monitoring data model for stats #20577

tsullivan commented Jul 9, 2018 •

edited

Loading

tsullivan Jul 10, 2018 •

edited

Loading

tsullivan Jul 12, 2018 •

edited

Loading

tsullivan commented Jul 18, 2018

tsullivan commented Jul 18, 2018

tsullivan commented Jul 18, 2018

ycombinator left a comment

elasticmachine commented Jul 18, 2018

tsullivan commented Jul 18, 2018

tsullivan commented Jul 18, 2018

elasticmachine commented Jul 19, 2018

Stats API: implement the "kibana status" spec from the Monitoring data model for stats #20577

Stats API: implement the "kibana status" spec from the Monitoring data model for stats #20577

Conversation

tsullivan commented Jul 9, 2018 • edited Loading

Usage:

tsullivan Jul 10, 2018 • edited Loading

Choose a reason for hiding this comment

tsullivan Jul 12, 2018 • edited Loading

Choose a reason for hiding this comment

tsullivan commented Jul 18, 2018

tsullivan commented Jul 18, 2018

tsullivan commented Jul 18, 2018

ycombinator left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 18, 2018

💚 Build Succeeded

tsullivan commented Jul 18, 2018

tsullivan commented Jul 18, 2018

elasticmachine commented Jul 19, 2018

💚 Build Succeeded

tsullivan commented Jul 9, 2018 •

edited

Loading

tsullivan Jul 10, 2018 •

edited

Loading

tsullivan Jul 12, 2018 •

edited

Loading