Create an interface for viewing deployment analytics #288

manaswisaha · 2016-09-02T01:34:02Z

Things that we need for the analytics interface:

What percentage of new visitors click participate?
What percentage of folks that click participate finish on boarding?
What percentage finish one mission?
How many visitors the site is getting?
How many touchscreen users are we getting?

4 and 5 can be retrieved using Google Analytics API and the rest can be retrieved from the Sidewalk Database.

Options: Add to the admin interface? OR some other interface e.g. a script.

jonfroehlich · 2016-09-02T16:30:51Z

@Manaswis, can't you just calculate touchscreen users from Google Analytics? Would be good to know this stuff ASAP.

jonfroehlich · 2016-09-02T17:11:06Z

Looks like it's 30% touchscreen currently--27% phone and 3% tablet. That's a pretty high number. Unsurprisingly, the bounce rate is double for touchscreen. :)

kotarohara · 2016-09-02T17:19:07Z

I see you in a meeting yet you are commenting :)

jonfroehlich · 2016-09-02T17:28:24Z

Haha, I assure you I do this far less than the average prof!

misaugstad · 2017-06-07T17:28:48Z

Thanks for pointing this out @Manaswis

What percentage of new visitors click participate?

Of course this is now either the "start mapping" or "login". How important is this metric to us right now?

What percentage of folks that click participate finish on boarding?

This is included in the current admin interface.

What percentage finish one mission?

I'm not sure we need a number that says what percentage of people finish one mission... I think this is mostly captured by histograms showing distance covered per user (split by registered and anon), as mentioned in #351 and #342

How many visitors the site is getting?

We can get this from Google Analytics I'm assuming?

How many touchscreen users are we getting?

We got an answer to this from Google Analytics.

jonfroehlich · 2017-06-07T17:44:49Z

What percentage of folks that click participate finish on boarding?
This is included in the current admin interface.

It is? Where. I use the admin interface all the time and I've never seen this (or understood the current analytics for this if it exists)

What percentage finish one mission?

Why can't we have a histogram of number of missions completed complemented by descriptive stats like avg, median, mode, and SD of missions completed per person.

Also, I do think it would be useful to know:

The total number users who click on 'start mapping'
The total number who start tutorial
The total number who finish tutorial
The total number who start at least one mission
The total number who finish at least one mission

I would expect a dropoff at each step and tracking this seems important to informing future updates of our tools to better engage and sustain participation.

misaugstad · 2017-06-07T17:57:18Z

What percentage of folks that click participate finish on boarding?
This is included in the current admin interface.
It is? Where. I use the admin interface all the time and I've never seen this (or understood the current analytics for this if it exists)

Directly above the "Onboarding Completion Time" graph, the bounce rate is shown:

Why can't we have a histogram of number of missions completed complemented by descriptive stats like avg, median, mode, and SD of missions completed per person.

Whoops, that is what I meant to say :) Yes let's have all the things!

jonfroehlich · 2017-06-07T18:02:34Z

What's the x-axis on this graph? To me, this thing is not super legible.

Also, does this show how long it takes people to complete onboarding? If so, then it's not showing us information about people who drop out...

I now see the bounce rate stat--I seemed to have overlooked this before (poor UI :)). Is the bounce rate defined as the number of people who do not finish (so 26.6% who start the tutorial do not finish) or is it the number of people who do finish (so, 26.6% who start the tutorial do finish)?

misaugstad · 2017-06-07T18:13:54Z

What's the x-axis on this graph? To me, this thing is not super legible.

That would be a time interval in seconds (you can kind of see it in the bottom right corner, but above the x-axis).

I now see the bounce rate stat--I seemed to have overlooked this before (poor UI :)).

Much needs to be changed on this page :)

Is the bounce rate defined as the number of people who do not finish (so 26.6% who start the tutorial do not finish) or is it the number of people who do finish (so, 26.6% who start the tutorial do finish)?

That would be 26% do not finish, the relevant line of code is here

adevdash · 2017-06-19T19:19:56Z

Currently the way I'm calculating Visit Index and Start Mapping are from events logged in WebsiteActivityTable, Visit_Index and Visit_Audit. I count all of the Visit_Index and Visit_Audit activities (including those logged from the same IP address). The problem with doing it like this is that each time the homepage is refreshed, a new Visit_Index is logged, which obfuscates how many actual users are going on to audit. A possible solution could be to log page visits per session rather than per visit,

Onboarding beginnings and completions are counted from Onboarding_Transition events in AuditTaskInteractionTable where the note corresponds to an initialization or an outro. Initializations suffer from the same problem as the index in that refreshing on the tutorial page logs a new event corresponding to initialization.

misaugstad · 2017-06-19T19:24:52Z

I think that there are a lot of cases where we would want to keep track of the session. How hard would it be to start recording this?

jonfroehlich · 2017-06-19T19:27:06Z

I don't think homepage refreshes are very common, so not sure this should be a key concern. Though I understand (and appreciate) your point. And agree with it.

Re: the visualization itself. You need to print raw % above each bar. In addition, perhaps we need not show Visit Index since that is always 100%, right? This would help with the flattening that has occurred, obscuring our ability to understand the underlying data.

Re: @misaugstad's question about sessions. @Manaswis: aren't we currently doing session tracking?

misaugstad · 2017-06-19T19:29:26Z

What are some other reasons for the huge bounce rate then? According to Google Analytics, shown in this comment, the bounce rate is ~50%, and @Adash12 seems to be getting a bounce rate of ~99%. Something is up here!

misaugstad · 2017-06-19T19:30:46Z

Since we switched from "participate" to "start mapping", are those considered different activities?

jonfroehlich · 2017-06-19T19:31:18Z

Not sure. We will have to investigate further but I'm fairly confident that, in general, homepage refreshes are uncommon.

My suggestion to take out Visit Index stands (as do my other suggestions).

Since we switched from "participate" to "start mapping", are those considered different activities?

I doubt it. I think this is only a frontend change rather than a logging change.

manaswisaha · 2017-06-19T23:46:38Z

Jon: @Manaswis: aren't we currently doing session tracking?

We don't for anonymous users across different pages.

adevdash · 2017-06-20T15:56:12Z

Removed Visit_Index as suggested, and got percentages to display. Currently working on logging anonymous users completing their first missions.

jonfroehlich · 2017-06-20T16:07:40Z

Shouldn't the graph always go down? Or are you individually calculating retention at each step (i.e., the % shown is the % of users left in the system who finish that step)....

…

On Tue, Jun 20, 2017 at 11:56 AM, Aditya Dash ***@***.***> wrote: [image: image] <https://user-images.githubusercontent.com/13080218/27342861-5ad2f364-55af-11e7-8b22-8d1ec32c1f1f.png> Removed Visit_Index as suggested, and got percentages to display. Currently working on logging anonymous users completing their first missions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#288 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABi-9XKBPnG5dEhq2nMw56R-Vi9ZQav4ks5sF-uegaJpZM4JzSe4> .

-- Jon Froehlich Assistant Professor Computer Science University of Maryland, College Park http://www.cs.umd.edu/~jonf/ @jonfroehlich <https://twitter.com/jonfroehlich> - Twitter

adevdash · 2017-06-20T16:15:20Z

The latter - at the main page an anonymous user can either start mapping or log in (or do something else that isn't shown in this graph). Each subsequent bar should be smaller than Start Mapping, but I think Sign-In and Start Mapping are still independent of each other.

jonfroehlich · 2017-06-20T16:20:17Z

So, you are saying that 0.3% of our users click 'Start Mapping'? That seems incredibly small. And only 0.026% complete onboarding (that is, less than 3 users per 100 complete onboarding)? And then, basically, 0% complete their first mission (the label says '0.00000%').

In addition to completing first mission, how many: (i) apply at least one label; (ii) take at least one step?

manaswisaha · 2017-06-20T16:27:55Z

I think it will be useful to test this code using a larger dump. Since he has a small local dump and could be that mostly researchers worked on it, the data is such. I'll send the link to the larger dump (until Dec 2016) on slack.

adevdash · 2017-06-20T16:37:37Z

First mission has a placeholder value right now since I haven't implemented it. I think that the low values are due to the fact that refreshes on the home page log as new page visits under my method of calculation, and since the data dump I had was probably used by developers there were a lot of page refreshes. Hopefully the new data that Manaswi sent will provide more accurate results.

jonfroehlich · 2017-06-20T16:39:00Z

OK, thanks.

…

On Tue, Jun 20, 2017 at 12:37 PM, Aditya Dash ***@***.***> wrote: First mission has a placeholder value right now since I haven't implemented it. I think that the low values are due to the fact that refreshes on the home page log as new page visits under my method of calculation, and since the data dump I had was probably used by developers there were a lot of page refreshes. Hopefully the new data that Manaswi sent will provide more accurate results. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#288 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABi-9Ss-de9WwhXAMP7AyBw2d0gSKaJVks5sF_VTgaJpZM4JzSe4> .

-- Jon Froehlich Assistant Professor Computer Science University of Maryland, College Park http://www.cs.umd.edu/~jonf/ @jonfroehlich <https://twitter.com/jonfroehlich> - Twitter

manaswisaha changed the title ~~TODO: Deployment Analytics~~ Create an interface for viewing deployment analytics Sep 2, 2016

manaswisaha added the Feature Request label Sep 2, 2016

kotarohara added the Priority Medium label Sep 2, 2016

kotarohara added this to the Public Deployment v2 milestone Sep 2, 2016

kotarohara added the Admin Interface label Sep 2, 2016

kotarohara assigned manaswisaha Sep 13, 2016

manaswisaha removed the Priority Medium label Sep 17, 2016

manaswisaha added the Priority: Low label Oct 26, 2016

manaswisaha modified the milestones: Public Deployment v3, Public Deployment v2 Oct 28, 2016

manaswisaha assigned misaugstad and unassigned manaswisaha Jun 7, 2017

adevdash mentioned this issue Jun 20, 2017

Admin page: Onboarding time chart incorrect #716

Closed

misaugstad assigned adevdash Aug 13, 2017

misaugstad unassigned misaugstad and adevdash Jan 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create an interface for viewing deployment analytics #288

Create an interface for viewing deployment analytics #288

manaswisaha commented Sep 2, 2016 •

edited

Loading

jonfroehlich commented Sep 2, 2016

jonfroehlich commented Sep 2, 2016 •

edited

Loading

kotarohara commented Sep 2, 2016

jonfroehlich commented Sep 2, 2016 •

edited by misaugstad

Loading

misaugstad commented Jun 7, 2017 •

edited

Loading

jonfroehlich commented Jun 7, 2017 •

edited by misaugstad

Loading

misaugstad commented Jun 7, 2017

jonfroehlich commented Jun 7, 2017

misaugstad commented Jun 7, 2017

adevdash commented Jun 19, 2017 •

edited

Loading

misaugstad commented Jun 19, 2017

jonfroehlich commented Jun 19, 2017

misaugstad commented Jun 19, 2017

misaugstad commented Jun 19, 2017

jonfroehlich commented Jun 19, 2017 •

edited

Loading

manaswisaha commented Jun 19, 2017 •

edited

Loading

adevdash commented Jun 20, 2017

jonfroehlich commented Jun 20, 2017 via email

adevdash commented Jun 20, 2017

jonfroehlich commented Jun 20, 2017

manaswisaha commented Jun 20, 2017

adevdash commented Jun 20, 2017

jonfroehlich commented Jun 20, 2017 via email

Create an interface for viewing deployment analytics #288

Create an interface for viewing deployment analytics #288

Comments

manaswisaha commented Sep 2, 2016 • edited Loading

jonfroehlich commented Sep 2, 2016

jonfroehlich commented Sep 2, 2016 • edited Loading

kotarohara commented Sep 2, 2016

jonfroehlich commented Sep 2, 2016 • edited by misaugstad Loading

misaugstad commented Jun 7, 2017 • edited Loading

jonfroehlich commented Jun 7, 2017 • edited by misaugstad Loading

misaugstad commented Jun 7, 2017

jonfroehlich commented Jun 7, 2017

misaugstad commented Jun 7, 2017

adevdash commented Jun 19, 2017 • edited Loading

misaugstad commented Jun 19, 2017

jonfroehlich commented Jun 19, 2017

misaugstad commented Jun 19, 2017

misaugstad commented Jun 19, 2017

jonfroehlich commented Jun 19, 2017 • edited Loading

manaswisaha commented Jun 19, 2017 • edited Loading

adevdash commented Jun 20, 2017

jonfroehlich commented Jun 20, 2017 via email

adevdash commented Jun 20, 2017

jonfroehlich commented Jun 20, 2017

manaswisaha commented Jun 20, 2017

adevdash commented Jun 20, 2017

jonfroehlich commented Jun 20, 2017 via email

manaswisaha commented Sep 2, 2016 •

edited

Loading

jonfroehlich commented Sep 2, 2016 •

edited

Loading

jonfroehlich commented Sep 2, 2016 •

edited by misaugstad

Loading

misaugstad commented Jun 7, 2017 •

edited

Loading

jonfroehlich commented Jun 7, 2017 •

edited by misaugstad

Loading

adevdash commented Jun 19, 2017 •

edited

Loading

jonfroehlich commented Jun 19, 2017 •

edited

Loading

manaswisaha commented Jun 19, 2017 •

edited

Loading