Phase 1: Membership (reporting)

[cbowskill]

User story

As an organisation
We'd like to know how people are finding the membership journeys
So that we can improve and optimise the flow

Acceptance criteria

1. Ability to understand the membership flow and any key drop out points
2. Easy to view dashboard created around the key journeys
3. Understand and document what insights we're capturing, how we're capturing them and why

Measurement

• TBD

All Phase 1 stories

#7808 - Replicate existing form
#7809 - Confirmation of application
#7811 - Verification of email
#7812 - Update sign in
#7814 - Update editorial content
#7815 - Reporting
#7848 - Temporary member messaging - account

[jennpb]

Hi @melanierogan - do you know what kind of steer this requires? I think this is finished but want to double check as you last touched it.

[melanierogan]

@jennpb Ah yes, I remember why I had put this label on now: I think it needs some validation to verify that the reporting is returning what we need to deliver the metrics indicated. I know we were wanting to wait to see membership in use for a while before we looked, but I think it would be good to double check reporting is giving us what we need.

It might also be worth noting that we can get some reporting or some idea of membership numbers from auth0, so I guess that gives us another data point.

I raised the possible uses for the metrics on membership, and the data points we have in auth0, being a possibility for further iterations where we could do cool things e.g. engagement emails. I fully admit that may not be the direction we would take, but i'm a self confessed email-dork, so I thought that idea might be worth mentioning.

[taceybadgerbrook]

I have access to Auth0 but the data is not very illuminating. Mousing over the boxes reveals total numbers of successful daily logins. Darkest green = 9 logins. Lightest green = 1 login.

[melanierogan]

I guess it depends on what we focus on e.g. 'we want lots of people to sign up and be engaged', then what we have as our metrics for success, e.g. is 'x number of registrations per week' a good measure.

[alexwlchan]

I have access to Auth0 but the data is not very illuminating.

That's in the staging instance of Auth0, not prod – notice how it's heavier earlier in the year when we were doing lots of testing, but has dropped off now we're not working on it.

[taceybadgerbrook]

@alexwlchan How do I view Prod? I tried changing the address to wellcomecollection-prod but that doesn't seem to do anything.

[alexwlchan]

How do I view Prod?

You don't have access to prod; only a handful of people have access to our prod Auth0 tenant (because it contains PII for real library users).

I gave you access to stage as a way to experiment and see if you could get the data you need from it, and if there was useful data there we'd discuss how best to get it to you. There aren't different types of data in prod, but there's more activity in the graph.

[alexwlchan]

How do I view Prod?

You don't have access to prod; only a handful of people have access to our prod Auth0 tenant (because it contains PII for real library users).

I gave you access to stage as a way to experiment and see if you could get the data you need from it, and if there was useful data there we'd discuss how best to get it to you. There aren't different types of data in prod, but there's more activity in the graph.

[jennpb]

@jennpb Ah yes, I remember why I had put this label on now: I think it needs some validation to verify that the reporting is returning what we need to deliver the metrics indicated. I know we were wanting to wait to see membership in use for a while before we looked, but I think it would be good to double check reporting is giving us what we need.

It might also be worth noting that we can get some reporting or some idea of membership numbers from auth0, so I guess that gives us another data point.

Thanks @melanierogan

Given the ongoing conversation then, it looks like we have not yet met the Acceptance Criteria and this is still a totally to-be-done task. I agree with the AC included here. I'll keep in the backlog for now.

[taceybadgerbrook]

@alexwlchan Ok, that makes sense. I've seen the PII data from Tanya's reports so good call. It's difficult for me to know whether the data in Prod is useful if I don't know what it is.

Questions we're interested in	data we'd like to see
Do people have trouble logging in?	Daily totals of failed login attempts
Are more people signing up for library membership?	Daily totals of new memberships
Are more people making use of their membership?	Daily total of successful logins, daily totals of users

[alexwlchan]

That feels achievable. This is the stream of logs inside Auth0:

We already listen to it to get alerts in Slack about unexpected failures. We could hook up a second Lambda which siphons off a suitably anonymised version of these logs to the reporting cluster.

The tricky bit would be aggregating by user – e.g. a user who logs in three times in a day should count as +1 towards the daily user total, not +3. We'd have to think of a way to anonymise users.

This would probably take a few days to build out properly.