Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement analytics on changes to the .gitpod.yml #6894

Closed
Tracked by #7671
loujaybee opened this issue Nov 25, 2021 · 14 comments · Fixed by #11653
Closed
Tracked by #7671

Implement analytics on changes to the .gitpod.yml #6894

loujaybee opened this issue Nov 25, 2021 · 14 comments · Fixed by #11653
Assignees
Labels
aspect: analytics Anything related to analytics aspect: growth Issues relating to product growth initiatives editor: code (browser) meta: never-stale This issue can never become stale team: IDE

Comments

@loujaybee
Copy link
Member

loujaybee commented Nov 25, 2021

Context: This one is important as it will give us data around when a user is updating this file as part of their onboarding journey. e.g. are users updating the file on their first visit? second visit? etc

We would like to know: if / when a user is changing or updating their gitpod.yml.

Ideally, we would gather fine-grained data on which properties the user is updating on the YAML file. This is so we can optimize how and when we present certain features to our users.

Considerations:

  • We could possibly implement this via a keystroke event within VS Code, but we'll need to set a reasonable debounce/throttle on the event to ensure we do not send many events.
  • We will need to decide which attributes/aspects of the gitpod.yml that are interesting for us to track and why.

Note: If necessary we can break this ticket down, simply knowing that the user has changed the file / added it would deliver value at this stage.

@loujaybee loujaybee changed the title Implement analytics that track changes to the .gitpod.yml Implement analytics on changes to the .gitpod.yml Nov 25, 2021
@loujaybee loujaybee added team: IDE editor: code (browser) aspect: growth Issues relating to product growth initiatives aspect: analytics Anything related to analytics labels Nov 25, 2021
@filiptronicek
Copy link
Member

@loujaybee this is a very interesting concept. I wonder if we could do more things on the client by just adding a tailored extension, which would take care of the whole gitpod.yml walkthrough process. This would mean nothing is sent back - good for our servers and also 0 latency. I imagine in this process there won't be too many features which we couldn't implement via an extension, but just a thought 💡!

@loujaybee
Copy link
Member Author

This one is a little tricky, so probably best if @akosyakov is able to collaborate with whoever picks this up to come up with a plan. As with all of the analytics tickets, having something working initially is better than trying to overthink it, and come up with something super comprehensive.

@loujaybee
Copy link
Member Author

loujaybee commented Dec 8, 2021

There is potentially some tracking added already here in the supervisor [1].

Let's investigate if we have that data, and if those events are working as expected.

@akosyakov
Copy link
Member

There is potentially some tracking added already here in the supervisor [1].

@jakobhero Could you check please whether we have config-changed in Segment, but which does not make to BigQuery for some reasons?

@jakobhero
Copy link
Contributor

whether we have config-changed in Segment, but which does not make to BigQuery for some reasons?

Hi @akosyakov, i had another look at the events coming into Segment and config-changed never makes it there. Also, the BigQuery destination is not configured to block any of the events. Are we sending the events to the source with the write key of production trusted?

@akosyakov
Copy link
Member

I think it is configured here:

switch os.Getenv("GITPOD_ANALYTICS_WRITER") {
case "log":
return &logAnalyticsWriter{}
case "segment":
return &segmentAnalyticsWriter{Client: segment.New(os.Getenv("GITPOD_ANALYTICS_SEGMENT_KEY"))}
and sent here:
func (s *segmentAnalyticsWriter) Track(m TrackMessage) {
defer recover()
_ = s.Client.Enqueue(segment.Track{
AnonymousId: m.AnonymousID,
UserId: m.UserID,
Event: m.Event,
Timestamp: m.Timestamp,
Properties: m.Properties,
})
}

@csweichel Could it be that GITPOD_ANALYTICS_WRITER and GITPOD_ANALYTICS_SEGMENT_KEY not configured in production?

@jakobhero
Copy link
Contributor

Could it be that GITPOD_ANALYTICS_WRITER and GITPOD_ANALYTICS_SEGMENT_KEY not configured in production?

For what it's worth, dashboard and server side analytics calls also depend on the configuration of those as far as i know and they don't seem to be affected.

@akosyakov
Copy link
Member

For what it's worth, dashboard and server side analytics calls also depend on the configuration of those as far as i know and they don't seem to be affected.

yes, i see it in each workspace

@jeanp413
Copy link
Member

@akosyakov are you working on this if not what still needs to be done?

@akosyakov
Copy link
Member

@jeanp413 not actively

I wanted to sync with @csweichel with how generally we can get analytics directly from workspace to Segment in secure way. If it is not possible then the fix is rather simple, use GitpodServer in supervisor to track this event. You can make a PR for it, but we delay merging after talking to Chris.

@jeanp413
Copy link
Member

@akosyakov is that related to this #7257 (comment). If so I'd be fine waiting as that would also help make straightforward sending telemetry in vscode core (instead of creating a web socket connection)

@akosyakov
Copy link
Member

@jeanp413 since we are fine to use untrusted, it would be good to investigate why supervisor cannot send to segment, we don't log any errors there to avoid spamming logs, but maybe it is good to enable them temporarily

@loujaybee
Copy link
Member Author

Removing from scheduled as not a great deal we can do whilst we're waiting for visualisation dashboards to be created.

Since I've now closed the following epic (which used to contain this ticket). I'll now move this out of groundwork.

@stale
Copy link

stale bot commented May 25, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the meta: stale This issue/PR is stale and will be closed soon label May 25, 2022
@loujaybee loujaybee added meta: never-stale This issue can never become stale and removed meta: stale This issue/PR is stale and will be closed soon labels May 27, 2022
@akosyakov akosyakov self-assigned this Jul 29, 2022
@akosyakov akosyakov moved this to In Review in 🚀 IDE Team Jul 29, 2022
@akosyakov akosyakov moved this from In Review to Awaiting Deployment in 🚀 IDE Team Aug 10, 2022
@akosyakov akosyakov moved this from Awaiting Deployment to Done in 🚀 IDE Team Aug 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aspect: analytics Anything related to analytics aspect: growth Issues relating to product growth initiatives editor: code (browser) meta: never-stale This issue can never become stale team: IDE
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

5 participants