Determining the Orbit Level based on the Love

[joshed-io]

This issue is connected to #29.

"Love" is a formula that helps determine relative engagement of community members. #29 contains a proposal for, among other things, a new way to calculate Love.

Once Love is calculated, it can then be used to place members into 1 of the 4 Orbit Levels. How this happens, numerically, is the topic of this issue. Ultimately, the way that Love is calculated and how it is used to map members to Orbit Levels is up to each community that implements the model. However, it'd be great if we could provide some reasonable defaults and common logic for the general case.

Here are a few different ways I can see calculating Orbit Levels. Please add thoughts, feedback, and more ideas below.

1 - Step function

Pick 3 ranges for Love that represent the Orbit Levels.

Orbit 1: Love >= 10
Orbit 2: 5 < Love < 10
Orbit 3: 1 < Love < 5
Orbit 4: Love < 1

The ranges would be defined by each community, and take into account how their Love formula works. We could try to provide reasonable defaults for this method, but since communities differ widely in their amount of engagement, this might be hard.

2 - Ranking / percentile distribution

Orbit 1: Top 95% Love
Orbit 2: Top 75-95% Love
Orbit 3: Top 25-75% Love
Orbit 4: Bottom 25% Love

This approach doesn't require picking constants for Love, which might make it more flexible for diffferent communities. The actual %'s would be configurable. However, it could lead to issues like a member who should be in Orbit 1, according to how the community sees their engagement, but can't be because Orbit 1 is already "full" of members with higher Love.

Note: There's a variation of this approach with fixed numbers instead of percents, i.e. Top 100 members in Orbit 1, next 250 in Orbit 2, and so on. That still fixes the size of each level, but to a constant not a percentage.

3 - Grading on a curve

MAX() version:

Orbit 1 - 95%+ of the highest member's Love
Orbit 2 - 75-95% of the highest member's Love
Orbit 3 - 25-75% of the highest member's Love
Orbit 4 - 0-25% of the highest member's Love

MEAN() version:

Orbit 1 - 200%+ of the mean member's Love
Orbit 2 - 100-200% of the mean member's Love
Orbit 3 - 50-100% of the mean member's Love
Orbit 4 - 0-50% of the mean member's Love

This approach decides Orbit Levels relative to the Love score for a member in the community at a certain percentile. It doesn't limit how many people can be in each level, unlike option 2. Percentages and which function to use would be customized per community. The values here are just illustrative and do not represent recommendations (true for all numbers in this comment).

See this comment on #29 for why MEAN might be a better approach than MAX here. The MEAN() approach above chooses numbers that puts the "mean member" right at the border of Orbits 2 and 3.

Specific number and type of activities

There are other approaches to Orbit Level calculation that could not use the Love score. One simple example is to set threshholds on just the number of activities done in a time period, of various types. For example, doing 10 pull requests and attending 3 events in the last 12 months would place a member into Orbit 1.

I'm not a huge fan of this approach since it doesn't use Love and feels like it puts too much onus on the model user to figure out exactly what this needs to look like. But I'm listing here anyway in the spirit of laying out all the options.

[zmarkan]

I like 1 the most, with some percentage aspects of 2 and 3.

With 1 I like the flexibility it offers, and with 2 and 3 I like the automated aspect - almost fire and forget kind of approach. Then again, we're talking communities here, and fire and forget isn't really a tactic to use (I hope).
Finally since most answers to community related questions tend to start with "Well, it depends, ..." - I'm more leaning towards 1.

Maybe you could the run percentages (either 1 or 2) as a side calculation - in order to help inform the community managers of what are some suggested Orbit levels and thresholds for their community -> which would then feed into the concrete settings for 1.

There's also the outliers (especially with newer Orbit communities) that you might want to take into account. Last thing you want to have is confuse everything in a new adopter because of lack of data which makes orbits levels jump around on a daily basis.

Slightly off topic - What about the idea of manually setting the Orbit level for a particular user?
I mean as an exception, not as a rule. Maybe someone provided a really great piece of feedback, or done something truly remarkable without doing a constant stream of activities - in that case you might want to override whatever the required score is - without having to create a special highly scored activity just for that single action.

[joshed-io]

Thanks for the thoughts here @zmarkan.

I really like the idea of showing folks the distribution of the Love scores, in general, and for helping decide Orbit Level thresholds. There are some things that only become obvious when looking at the distribution, like any clustering or big gaps.

Re: manual Orbit Level overrides, I think a community could incorporate that, but I'd probably look closely at the motivation since it adds complexity on the implementation side. If the goal is to "boost" someone in the rankings, or catapult them into another Orbit, I'd be curious about why & what comes after that. In some cases, applying a tag or putting the member in a certain group could be better. In other cases, it might mean a change to the Love formula or adjusting the Orbit Level configuration is needed. Creating a special activity with a big weight is an interesting idea, the benefit of that is that it would decay according to the same rules as other activities.

[erlend-sh]

I also prefer (1). It’s what I’m most used to already from Discourse’s trust levels.

For my purposes I don’t really want any mean or average calculation determining what’s high or low. I want to set fixed numbers for each orbit tier on my own (if the defaults need calibration). This is the best way for me to get a sense for what these scores actually mean.

I also want the last tier, Ambassador/Inner, to be a manual step. It would work like this (I’d still want activity regularity to be taken into account, just simplifying scores here for demonstration purposes):

Outer: 5p
Far: 20p
Near: 50p
Inner: >100p

Any member that goes past 100p would get automatically nominated for the Inner tier, but they would require manual approval to get added. This adds to the honor of reaching the top tier, because you’ll be personally initiated by the person who vouched for you.

[joshed-io]

@erlend-sh Thanks for the insights here! Great pointer to the trust levels from Discourse, it's really impressive how each trust level is awarded based on a plentitude of different activities and targets associated.

The +1 for the step function is registered, all signs are pointing toward that being the best choice for now. It wins on stability, predictably detecting when orbit levels change (no floating targets based on other members) and (as you mention) developing a sense of their meaning over time.

As for the nomination-style approach for Orbit 1, there's multiple voices saying that's a good idea, so I think we should include that in the new documentation. I'd like to make it opt-in though, so communities can do it either way.

[simskij]

I also prefer (1). It’s what I’m most used to already from Discourse’s trust levels.

For my purposes I don’t really want any mean or average calculation determining what’s high or low. I want to set fixed numbers for each orbit tier on my own (if the defaults need calibration). This is the best way for me to get a sense for what these scores actually mean.

I also want the last tier, Ambassador/Inner, to be a manual step. It would work like this (I’d still want activity regularity to be taken into account, just simplifying scores here for demonstration purposes):

Outer: 5p
Far: 20p
Near: 50p
Inner: >100p

Any member that goes past 100p would get automatically nominated for the Inner tier, but they would require manual approval to get added. This adds to the honor of reaching the top tier, because you’ll be personally initiated by the person who vouched for you.

I really like this!

I would be interested in the opposite as well. People in the top tier who's love has decayed over time could be suggested to be decreased to another tier (maybe even on a fixed schedule, like once a year). This would make the top tier a good alternative for managing actual ambassador programs.

If using this nomination-style however, I'd really prefer if it was an additional level rather than the default 4, or we'd lose a lot of the granularity of the automated "tiering", which I really like.