From e15af9497493c8f733dd4a15db773c3aa0efe207 Mon Sep 17 00:00:00 2001 From: Timothy Sullivan Date: Tue, 18 Sep 2018 12:01:52 -0700 Subject: [PATCH 1/6] Developer documentation for integrating with the telemetry service --- src/server/usage/README.md | 92 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 src/server/usage/README.md diff --git a/src/server/usage/README.md b/src/server/usage/README.md new file mode 100644 index 0000000000000..679bac8e82edd --- /dev/null +++ b/src/server/usage/README.md @@ -0,0 +1,92 @@ +# Kibana Telemetry + +Telemetry is a service in Kibana that features and plugins can register with for tracking usage. The general term "telemetry" refers to multiple things: + +1. Integrating with the telemetry service to express how to collect usage data (Collecting). +2. Sending a payload of usage data up to Elastic's telemetry cluster, once per browser per day (Sending). +3. Viewing usage data in the Kibana instance of the telemetry cluster (Viewing). + +You, the feature or plugin developer, mainly need to worry about the first meaning: collecting. To integrate with the telemetry services for usage collection of your feature, there are 2 steps: + +1. Create a usage collector using a factory function +2. Register the usage collector with the Telemetry service + +NOTE: To a lesser extent, there's also a need to update the telemetry payload of Kibana stats and telemetry cluster field mappings to include your fields. This part is typically handled not by you, the developer, but different maintainers of the telemetry cluster. Usually, this step just means file an issue in the telemetry repo and have someone approve your data model. + +## Creating a Usage Collector + +A usage collector object is an instance of a class called `UsageCollector`. A factory function on `server.usage.collectorSet` object allows you to create an instance of this class. All you need to provide is a `type` for organizing your fields, and a `fetch` method for returning your usage data. + +Example: + +```js +const myCollector = server.usage.collectorSet.makeUsageCollector({ + type: MY_USAGE_TYPE, + fetch: async callCluster => { + + // query ES and get some data + // summarize the data into a model + // return the modeled object that includes whatever you want to track + + return { + my_objects: { + total: SOME_NUMBER + } + }; + }, +}); +``` + +Some background: The `callCluster` that gets passed to the `fetch` method is created in a way that's a bit tricky, to support multiple contexts the `fetch` method could be called. Your `fetch` method could get called as a result of an HTTP API request: in this case, the `callCluster` function wraps `callWithRequest`, and the request headers are expected to have read privilege on the entire `.kibana` index. The use case for this is stats pulled from a Kibana Metricbeat module, where the Beat calls Kibana's stats API in Kibana to invoke collection. + +The fetch method also might be called through an internal background task on the Kibana server, which currently lives in the `kibana_monitoring` module of the X-Pack Monitoring plugin, that polls for data and uploads it to Elasticsearch through a bulk API exposed by the Monitoring plugin for Elasticsearch. In this case, the `callCluster` method will be the internal system user and will have read privilege over the entire `.kibana` index. + +Note: there will be many cases where you won't need to use the `callCluster` function that gets passed in to your `fetch` method at all. Your feature might have an accumulating value in server memory, or read something from the OS. + +## Registering the Usage Collector + +After your `collector` object is created, you'll want to make the Telemetry service aware of it by registering it. This is one line of code: + +```js +server.usage.collectorSet.register(myCollector); +``` + +Typically, a plugin will create the collector object and register it with the Telemetry service from the `init` method of the plugin definition, or a helper module called from `init`. + +## Update the telemetry payload and telemetry cluster field mappings + +There is a module in the telemetry service that creates the payload of data that gets sent up to the telemetry cluster. + +As of the time of this writing (pre-6.5.0) there are a few unpleasant realities with this module. Today, this module has to be aware of all the features that have integrated with it, which it does from hard-coding. It does this because at the time of creation, the payload implemented a designed model where X-Pack plugin info went together regardless if it was ES-specific or Kibana-specific. In hindsight, all the Kibana data could just be put together, X-Pack or not, which it could do in a generic way. This is a known problem and a solution will be implemented in an upcoming refactoring phase, as this would break the contract for model of data sent in the payload. + +The second reality is that new fields added to the telemetry payload currently mean that telemetry cluster field mappings have to be updated, so they can be searched and aggregated in Kibana visualizations. This is also a short-term obligation. In the next refactoring phase, collectors will need to use a proscribed data model that eliminates maintenance of mappings in the telemetry cluster. + +## Testing + +There are a few ways you can test that your usage collector is working properly. + +1. The `/api/stats?extended=true` HTTP API in Kibana (added in 6.4.0) will call the fetch methods of all the registered collectors, and add them to a stats object you can see in a browser or in curl. To test that your usage collector has been registered correctly and that it has the model of data you expected it to have, call that HTTP API manually and you should see a key in the `usage` object of the response named after your usage collector's `type` field. This method tests the Metricbeat scenario described above where `callCluster` wraps `callWithRequest`. +2. There is a dev script in x-pack that will give a sample of a payload of data that gets sent up to the telemetry cluster for the sending phase of telemetry. Collected data comes from: + - The `.monitoring-*` indices, when Monitoring is enabled. Monitoring enhances the sent payload of telemetry by producing usage data potentially of multiple clusters that exist in the monitoring data. Monitoring data is time-based, and the time frame of collection is the last 15 minutes. + - Live-pulled from ES API endpoints. This will get just real-time stats without context of historical data. ✳ + - The dev script in x-pack can be run on the command-line with: + ``` + cd x-pack + node scripts/api_debug.js telemetry --host=http://localhost:5601 + ``` + Where `http://localhost:5601` is a Kibana server running in dev mode. If needed, authentication and basePath info can be provided in the command as well. +3. In Dev mode, Kibana will send telemetry data to a staging telemetry cluster. Assuming you have access to the staging cluster, you can log in and check the latest documents for your new fields. +4. If you catch the network traffic coming from your browser when a telemetry payload is sent, you can examine the request payload body to see the data. This can be tricky as telemetry payloads are sent only once per day per browser. Use incognito mode or clear your localStorage data to force a telemetry payload. + +✳ At the time of this writing, there is an open issue that in the sending phase, Kibana usage collectors are not "live-pulled" from Kibana API endpoints if Monitoring is disabled. The implementation on this depends on a new secure way to live-pull the data from the end-user's browser, as it would not be appropriate to supply only partial data if the logged-in user only has partial access to `.kibana`. + +## FAQ + +1. **Can telemetry track UI interactions, such as button click?** + Brief answer: no. Telemetry collection happens on the server-side so the usage data will only include information that the server-side is aware of. +2. **Does the telemetry service have a hook that I can call whenever some event happens in my feature?** + Brief answer: no. Telemetry collection is a fetch model, not a push model. Telemetry fetches info from your collector. +3. **How should I design my data model?** + Keep it simple, and keep it to a model that Kibana will be able to understand. In short, that means don't rely on nested fields (arrays with objects). Flat arrays, such as arrays of strings are fine. +4. **Who can I talk to with more questions?** + The Kibana Platform team is the owner of the telemetry service. You can bring questions to them. You can also talk to Tim Sullivan, who created the Kibana telemetry service, or Chris Earle, who set up the telemetry cluster and AWS Lambas for the upstream prod and staging endpoints that recieve the data sent from end-user browsers. From 551f0dcefe251631c7385100e33d65d1d48229a2 Mon Sep 17 00:00:00 2001 From: Timothy Sullivan Date: Tue, 18 Sep 2018 12:30:00 -0700 Subject: [PATCH 2/6] open with a bang --- src/server/usage/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/server/usage/README.md b/src/server/usage/README.md index 679bac8e82edd..550ccb4701c6d 100644 --- a/src/server/usage/README.md +++ b/src/server/usage/README.md @@ -1,6 +1,6 @@ -# Kibana Telemetry +# Kibana Telemetry Service -Telemetry is a service in Kibana that features and plugins can register with for tracking usage. The general term "telemetry" refers to multiple things: +Telemetry allows Kibana features to have usage tracked in the wild. The general term "telemetry" refers to multiple things: 1. Integrating with the telemetry service to express how to collect usage data (Collecting). 2. Sending a payload of usage data up to Elastic's telemetry cluster, once per browser per day (Sending). From 672521793b31a678c3b44d3db4fd9aaf614d1053 Mon Sep 17 00:00:00 2001 From: Timothy Sullivan Date: Tue, 18 Sep 2018 12:48:53 -0700 Subject: [PATCH 3/6] more faqs --- src/server/usage/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/server/usage/README.md b/src/server/usage/README.md index 550ccb4701c6d..097dc5b040e40 100644 --- a/src/server/usage/README.md +++ b/src/server/usage/README.md @@ -88,5 +88,9 @@ There are a few ways you can test that your usage collector is working properly. Brief answer: no. Telemetry collection is a fetch model, not a push model. Telemetry fetches info from your collector. 3. **How should I design my data model?** Keep it simple, and keep it to a model that Kibana will be able to understand. In short, that means don't rely on nested fields (arrays with objects). Flat arrays, such as arrays of strings are fine. -4. **Who can I talk to with more questions?** +4. **Can the telemetry payload include dynamic fields?** + Yes. In the issue you file in the telemetry repo, point out specifically which properties will have dynamic inner fields. +5. **If I accumulate an event counter in server memory, which my fetch method returns, won't it reset when the Kibana server restarts?** + Yes, but that is not a major concern. A visualization on such info might be a date histogram that gets events-per-second or something, which would be impacted by server restarts, so we'll have to offset the beginning of the time range when we detect that the latest metric is smaller than the earliest metric. That would be a pretty custom visualization, but perhaps future Kibana enhancements will be able to support that. +6. **Who can I talk to with more questions?** The Kibana Platform team is the owner of the telemetry service. You can bring questions to them. You can also talk to Tim Sullivan, who created the Kibana telemetry service, or Chris Earle, who set up the telemetry cluster and AWS Lambas for the upstream prod and staging endpoints that recieve the data sent from end-user browsers. From bc31607f5e6f14252508c5b9975a2d8b000445ce Mon Sep 17 00:00:00 2001 From: Timothy Sullivan Date: Tue, 18 Sep 2018 12:53:17 -0700 Subject: [PATCH 4/6] thing about tracking ui interactions --- src/server/usage/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/server/usage/README.md b/src/server/usage/README.md index 097dc5b040e40..6c507ba878ada 100644 --- a/src/server/usage/README.md +++ b/src/server/usage/README.md @@ -83,7 +83,7 @@ There are a few ways you can test that your usage collector is working properly. ## FAQ 1. **Can telemetry track UI interactions, such as button click?** - Brief answer: no. Telemetry collection happens on the server-side so the usage data will only include information that the server-side is aware of. + Brief answer: no. Telemetry collection happens on the server-side so the usage data will only include information that the server-side is aware of. There is no generic way to do this today, but UI-interaction KPIs can be tracked with a custom server endpoint that gets called for tracking when the UI event happens. 2. **Does the telemetry service have a hook that I can call whenever some event happens in my feature?** Brief answer: no. Telemetry collection is a fetch model, not a push model. Telemetry fetches info from your collector. 3. **How should I design my data model?** From a0d8f08375219248813bdd56bf15d52fb4d92633 Mon Sep 17 00:00:00 2001 From: Timothy Sullivan Date: Tue, 18 Sep 2018 13:02:19 -0700 Subject: [PATCH 5/6] talk to the plat team --- src/server/usage/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/server/usage/README.md b/src/server/usage/README.md index 6c507ba878ada..76cf4a585f379 100644 --- a/src/server/usage/README.md +++ b/src/server/usage/README.md @@ -11,7 +11,7 @@ You, the feature or plugin developer, mainly need to worry about the first meani 1. Create a usage collector using a factory function 2. Register the usage collector with the Telemetry service -NOTE: To a lesser extent, there's also a need to update the telemetry payload of Kibana stats and telemetry cluster field mappings to include your fields. This part is typically handled not by you, the developer, but different maintainers of the telemetry cluster. Usually, this step just means file an issue in the telemetry repo and have someone approve your data model. +NOTE: To a lesser extent, there's also a need to update the telemetry payload of Kibana stats and telemetry cluster field mappings to include your fields. This part is typically handled not by you, the developer, but different maintainers of the telemetry cluster. Usually, this step just means talk to the Platform team and have them approve your data model or added fields. ## Creating a Usage Collector @@ -89,7 +89,7 @@ There are a few ways you can test that your usage collector is working properly. 3. **How should I design my data model?** Keep it simple, and keep it to a model that Kibana will be able to understand. In short, that means don't rely on nested fields (arrays with objects). Flat arrays, such as arrays of strings are fine. 4. **Can the telemetry payload include dynamic fields?** - Yes. In the issue you file in the telemetry repo, point out specifically which properties will have dynamic inner fields. + Yes. When you talk to the Platform team about new fields being added, point out specifically which properties will have dynamic inner fields. 5. **If I accumulate an event counter in server memory, which my fetch method returns, won't it reset when the Kibana server restarts?** Yes, but that is not a major concern. A visualization on such info might be a date histogram that gets events-per-second or something, which would be impacted by server restarts, so we'll have to offset the beginning of the time range when we detect that the latest metric is smaller than the earliest metric. That would be a pretty custom visualization, but perhaps future Kibana enhancements will be able to support that. 6. **Who can I talk to with more questions?** From eea4aba0d97382d635f85f237d189257c82b97ac Mon Sep 17 00:00:00 2001 From: Timothy Sullivan Date: Wed, 19 Sep 2018 18:16:16 -0700 Subject: [PATCH 6/6] create and register --- src/server/usage/README.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/src/server/usage/README.md b/src/server/usage/README.md index 76cf4a585f379..0383ccdcc43a0 100644 --- a/src/server/usage/README.md +++ b/src/server/usage/README.md @@ -13,13 +13,14 @@ You, the feature or plugin developer, mainly need to worry about the first meani NOTE: To a lesser extent, there's also a need to update the telemetry payload of Kibana stats and telemetry cluster field mappings to include your fields. This part is typically handled not by you, the developer, but different maintainers of the telemetry cluster. Usually, this step just means talk to the Platform team and have them approve your data model or added fields. -## Creating a Usage Collector +## Creating and Registering Usage Collector -A usage collector object is an instance of a class called `UsageCollector`. A factory function on `server.usage.collectorSet` object allows you to create an instance of this class. All you need to provide is a `type` for organizing your fields, and a `fetch` method for returning your usage data. +A usage collector object is an instance of a class called `UsageCollector`. A factory function on `server.usage.collectorSet` object allows you to create an instance of this class. All you need to provide is a `type` for organizing your fields, and a `fetch` method for returning your usage data. Then you need to make the Telemetry service aware of the collector by registering it. Example: ```js +// create usage collector const myCollector = server.usage.collectorSet.makeUsageCollector({ type: MY_USAGE_TYPE, fetch: async callCluster => { @@ -35,6 +36,9 @@ const myCollector = server.usage.collectorSet.makeUsageCollector({ }; }, }); + +// register usage collector +server.usage.collectorSet.register(myCollector); ``` Some background: The `callCluster` that gets passed to the `fetch` method is created in a way that's a bit tricky, to support multiple contexts the `fetch` method could be called. Your `fetch` method could get called as a result of an HTTP API request: in this case, the `callCluster` function wraps `callWithRequest`, and the request headers are expected to have read privilege on the entire `.kibana` index. The use case for this is stats pulled from a Kibana Metricbeat module, where the Beat calls Kibana's stats API in Kibana to invoke collection. @@ -43,13 +47,6 @@ The fetch method also might be called through an internal background task on the Note: there will be many cases where you won't need to use the `callCluster` function that gets passed in to your `fetch` method at all. Your feature might have an accumulating value in server memory, or read something from the OS. -## Registering the Usage Collector - -After your `collector` object is created, you'll want to make the Telemetry service aware of it by registering it. This is one line of code: - -```js -server.usage.collectorSet.register(myCollector); -``` Typically, a plugin will create the collector object and register it with the Telemetry service from the `init` method of the plugin definition, or a helper module called from `init`.