Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs Data Telemetry] Report telemetry events to telemetry service #192868

Conversation

awahab07
Copy link
Contributor

@awahab07 awahab07 commented Sep 13, 2024

Summary

The PR defines the schema and registers the Usage Collector to report Logs Data telemetry events introduced in #189380.

This will introduce a telemetry subsection under stats.stack_stats.kibana.plugins.logs_data. This may likely be promoted to stats.logs_data given the nature of the data and as the schema matures.

How to test

  1. Make sure the task (task id: logs-data-telemetry:collect-and-report-task) has been run at least once. TELEMETRY_TASK_INTERVAL can be changed e.g. to 1 minutes to run it immediately.
  2. The queued data can be viewed by invoking the _stats api in browser console and inspecting the key [0].stats.stack_stats.kibana.plugins.logs_data:
await (await fetch("/<xyz>/internal/telemetry/clusters/_stats", {"body": JSON.stringify({"unencrypted":true,"refreshCache":true}), "method": "POST", "headers": {"kbn-xsrf": "x", "elastic-api-version": "2", "x-elastic-internal-origin": "x"}})).json()

Once the task (task id: logs-data-telemetry:collect-and-report-task) is run, the captured telemetry data can also be viewed in UI under Kibana -> Advanced Settings -> Global Settings -> Usage collection -> click "cluster data".
image

@obltmachine
Copy link

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@awahab07 awahab07 force-pushed the observability-dev-3772-report-events-to-telemetry branch from c0a2903 to ff34311 Compare September 25, 2024 05:13
@awahab07 awahab07 added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Team:obs-ux-logs Observability Logs User Experience Team labels Sep 25, 2024
@awahab07 awahab07 marked this pull request as ready for review September 25, 2024 05:43
@awahab07 awahab07 requested review from a team as code owners September 25, 2024 05:43
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

@botelastic botelastic bot added the ci:project-deploy-observability Create an Observability project label Sep 25, 2024
…2-report-events-to-telemetry

# Conflicts:
#	x-pack/plugins/observability_solution/dataset_quality/server/services/data_telemetry/constants.ts
Copy link
Member

@afharo afharo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Telemetry changes LGTM

Copy link
Contributor

@tonyghiani tonyghiani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kibana-ci
Copy link
Collaborator

kibana-ci commented Sep 26, 2024

💛 Build succeeded, but was flaky

  • Buildkite Build
  • Commit: 6d2acc2
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-192868-6d2acc2d0ef8

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #15 / Create renders correctly with optional fields
  • [job] [logs] Jest Tests #15 / CustomFields renders custom fields correctly
  • [job] [logs] FTR Configs #65 / Dashboard save a search sessions with relative time "before all" hook for "Saves and restores a session with relative time ranges"
  • [job] [logs] Jest Tests #15 / OpenLensButton calls navigateToPrefilledEditor correctly
  • [job] [logs] FTR Configs #72 / Rule execution logic API Detection Engine - Execution logic @ess @serverless Indicator match type rules, alert suppression Code execution path: events count is smaller than threats count should suppress alerts on large number of documents, more than 9,000

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@awahab07 awahab07 merged commit 5f49cb8 into elastic:main Sep 27, 2024
28 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11063638207

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 27, 2024
…lastic#192868)

Define the schema and register the [Usage
Collector](https://docs.elastic.dev/telemetry/collection/snapshot-telemetry#creating-and-registering-usage-collector)
to report Logs Data telemetry events introduced in
elastic#189380.

This introduces a telemetry
[subsection](https://docs.elastic.dev/telemetry/kibana-snapshot-subsections)
under `stats.stack_stats.kibana.plugins.logs_data`.

(cherry picked from commit 5f49cb8)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Sep 27, 2024
…ice (#192868) (#194244)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Logs Data Telemetry] Report telemetry events to telemetry service
(#192868)](#192868)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Abdul Wahab
Zahid","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-27T02:47:02Z","message":"[Logs
Data Telemetry] Report telemetry events to telemetry service
(#192868)\n\nDefine the schema and register the
[Usage\r\nCollector](https://docs.elastic.dev/telemetry/collection/snapshot-telemetry#creating-and-registering-usage-collector)\r\nto
report Logs Data telemetry events introduced
in\r\nhttps://github.com//pull/189380.\r\n\r\nThis
introduces a
telemetry\r\n[subsection](https://docs.elastic.dev/telemetry/kibana-snapshot-subsections)\r\nunder
`stats.stack_stats.kibana.plugins.logs_data`.","sha":"5f49cb897281bdd729e25873875c9f57aa492a47","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-logs"],"title":"[Logs
Data Telemetry] Report telemetry events to telemetry
service","number":192868,"url":"https://github.com/elastic/kibana/pull/192868","mergeCommit":{"message":"[Logs
Data Telemetry] Report telemetry events to telemetry service
(#192868)\n\nDefine the schema and register the
[Usage\r\nCollector](https://docs.elastic.dev/telemetry/collection/snapshot-telemetry#creating-and-registering-usage-collector)\r\nto
report Logs Data telemetry events introduced
in\r\nhttps://github.com//pull/189380.\r\n\r\nThis
introduces a
telemetry\r\n[subsection](https://docs.elastic.dev/telemetry/kibana-snapshot-subsections)\r\nunder
`stats.stack_stats.kibana.plugins.logs_data`.","sha":"5f49cb897281bdd729e25873875c9f57aa492a47"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192868","number":192868,"mergeCommit":{"message":"[Logs
Data Telemetry] Report telemetry events to telemetry service
(#192868)\n\nDefine the schema and register the
[Usage\r\nCollector](https://docs.elastic.dev/telemetry/collection/snapshot-telemetry#creating-and-registering-usage-collector)\r\nto
report Logs Data telemetry events introduced
in\r\nhttps://github.com//pull/189380.\r\n\r\nThis
introduces a
telemetry\r\n[subsection](https://docs.elastic.dev/telemetry/kibana-snapshot-subsections)\r\nunder
`stats.stack_stats.kibana.plugins.logs_data`.","sha":"5f49cb897281bdd729e25873875c9f57aa492a47"}}]}]
BACKPORT-->

Co-authored-by: Abdul Wahab Zahid <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:obs-ux-logs Observability Logs User Experience Team v8.16.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants