Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

telemetry: fix tememetry data and update reported fields #1327

Merged
merged 10 commits into from
Apr 25, 2021

Conversation

AstroProfundis
Copy link
Contributor

What problem does this PR solve?

  • Fix wrong "cluster UUID" used for telemetry reports for clusters, use hashed cluster name instead
  • Add "installation UUID" of the tiup instance to identify unique users
  • Add "telemetry secret" to use as salt when hashing field values before uploading telemetry info
  • Include tiup and tiup-cluster build info in the telemetry report
  • Hash more critical field values, hash everything with the "telemetry secret" as salt so that same values used by different users produces different hashes

Check List

Tests

  • Unit test

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has persistent data change

Side effects

  • Increased code complexity

Related changes

  • Need to cherry-pick to the release branch

Release notes:

telemetry: fix tememetry data and update reported fields

@AstroProfundis AstroProfundis added type/enhancement Categorizes issue or PR as related to an enhancement. component/tiup Issues about the TiUP package management component itself component/cluster Issues about the tiup-cluster component labels Apr 23, 2021
@AstroProfundis AstroProfundis requested a review from lucklove April 23, 2021 10:04
@AstroProfundis AstroProfundis self-assigned this Apr 23, 2021
@ti-chi-bot ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Apr 23, 2021
@codecov-commenter
Copy link

codecov-commenter commented Apr 23, 2021

Codecov Report

Merging #1327 (b18f605) into master (4bc9b95) will increase coverage by 28.88%.
The diff coverage is 8.04%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #1327       +/-   ##
===========================================
+ Coverage   26.54%   55.43%   +28.88%     
===========================================
  Files         264      290       +26     
  Lines       19149    21557     +2408     
===========================================
+ Hits         5084    11951     +6867     
+ Misses      13286     7718     -5568     
- Partials      779     1888     +1109     
Flag Coverage Δ
cluster 42.68% <4.94%> (?)
dm 25.55% <1.13%> (?)
integrate 49.61% <5.79%> (+33.33%) ⬆️
playground 14.01% <2.05%> (?)
tiup 15.75% <2.05%> (-0.53%) ⬇️
unittest 22.45% <4.95%> (-0.66%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
components/cluster/command/check.go 80.00% <0.00%> (+1.42%) ⬆️
pkg/utils/http_client.go 69.23% <0.00%> (+69.23%) ⬆️
pkg/telemetry/report.pb.go 0.97% <1.22%> (-0.21%) ⬇️
components/cluster/command/root.go 43.08% <11.53%> (+15.27%) ⬆️
pkg/cluster/report/report.go 50.00% <14.28%> (+3.33%) ⬆️
pkg/telemetry/meta.go 62.50% <33.33%> (-17.50%) ⬇️
components/cluster/command/deploy.go 65.11% <50.00%> (+34.16%) ⬆️
pkg/telemetry/scrub.go 75.55% <70.83%> (-11.12%) ⬇️
components/cluster/command/clean.go 87.50% <100.00%> (+22.98%) ⬆️
components/cluster/command/destroy.go 73.91% <100.00%> (+19.36%) ⬆️
... and 238 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4bc9b95...b18f605. Read the comment docs.

@AstroProfundis AstroProfundis added this to the v1.4.2 milestone Apr 23, 2021
@@ -4,16 +4,18 @@ package telemetry;
import "gogoproto/gogo.proto";

message Report {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a good practice to modify existed field in protobuf.
Make sure it's compatible with old servers/clients.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didin't use protobuf at all, it was only used to generate the structs definitions, and the actual data send to telemetry server is using json.Marshal() with those structs.

pkg/telemetry/scrub.go Outdated Show resolved Hide resolved
@lucklove
Copy link
Member

/lgtm

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • lucklove

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 25, 2021
@lucklove
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: abe705f

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 25, 2021
@ti-chi-bot ti-chi-bot merged commit f663f41 into pingcap:master Apr 25, 2021
@AstroProfundis AstroProfundis deleted the update-telemetry branch April 25, 2021 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/cluster Issues about the tiup-cluster component component/tiup Issues about the TiUP package management component itself size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. type/enhancement Categorizes issue or PR as related to an enhancement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants