Discord bot effectiveness metrics

[songz]

As part of this design doc, we want to see the effectiveness of messaging and start the process of measuring student engagement.

We currently have posthog analytics, perhaps we could fire off events to posthog and see if their tools could help us answer a few of these questions?

1. On average, how long does it take from a student's submission to receiving a review? - The shorter this time is, the faster the feedback hence higher engagement
2. Dropoff - at which point does the student submit, get feedback, and then left the platform? This could indicate the need for establishing proper mentoring / code review guidelines to make sure students are not discouraged

Plus any other efforts to answer engagement. Moving forward, I'd like metric considerations to be part of all future designs, perhaps this discord bot metrics could be a start.

[JasirZaeem]

I haven't yet looked at posthog, I'll explore it. But these feel like things that could probably be achieved with some sql queries since we should already have this data in the database now that we store all attempts at a challenge. I'll see if I can add some queries in our metabase dashboard to show these statistics as a start.

[JasirZaeem]

Metric for 1: https://metabase.c0d3.com/question/17?days=7 (requires login to c0d3 metabase account)

[songz]

#2 research done by @InerLeeLofgren

Pretty even dropoff at each lesson. Linear decrease in users.