Gutenberg progressive rollout

[maxme]

Fixes #10747 - Implement a client only way to make a progressive rollout for the Gutenberg editor. I currently set an arbitrary number of 5%, but since we already have ~30% of users posting with Gutenberg, this should only move the needle to 3.5% ((100%-30%) x %5 = 3.5%) ie. when adding someone to the rollout group, there is a 30% chance they're already using Gutenberg. This number is even more skewed that this code exclude most self hosted users. It's arbitrary anyway and we only want to make this progressive (5% now, then 10%, 20%, 50%, 100% later).

I targeted 13.7, our target release for v2.

Test 1:

1. Make sure your the remote mobile editor setting is unset for the user/site you're testing (ping me if you need help for this).
2. Optional step: Clear your app data or uninstall the app (only required if you updated the remote mobile editor setting).
3. Set GB_ROLLOUT_PERCENTAGE to 0
4. Run the app.
5. Create a new post, you will get Aztec.

Test 2:

1. Make sure your the remote mobile editor setting is unset for the user/site you're testing (ping me if you need help for this).
2. Set GB_ROLLOUT_PERCENTAGE to 100
3. Run the app (on startup, your test user should be enrolled).
4. Go to the post list, tap the Create a new post button, you will get Gutenberg and you will see the info dialog.
5. (Note: if you try to create a post right after the switch on the main screen, you'll get Aztec. The site is not updated on the main screen, and it wasn't updated on purpose. I'm not sure why it was done that way, but to be safe I won't update it. It's not a big deal when the switch actually happen).

Notes:

• I added a new analytic parameter to the EDITOR_GUTENBERG_ENABLED event: ON_PROGRESSIVE_ROLLOUT.
• I was tempted to move this logic out of the EditPostActivity but we'll remove all of this at some point, so I'm not touching it.
• I preferred a modulo logic on the user id over random number generation as the user might use the app on different devices and eventually uninstall/reinstall the app. In these cases, they should have the same default editors.
• I added a new pref AppPrefs.isUserInGutenbergRolloutGroup to avoid re-doing the same logic. This might also be reused later if we want to change the rollout strategy.

Checks:

• I have considered adding unit tests where possible.
• I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

[maxme]

I'll work on the iOS patch when the logic is validated and this is merged.

[peril-wordpress-mobile]

You can test the changes on this Pull Request by downloading the APK here.

[daniloercoli]

The code in enableBlockEditor does make a network call to the backend. If you have many sites in the app there is a spike in network calls, and some of those can even fail...(We've experienced this problem when migrated the Editor pref from app-wide to site settings).
We should re-use the action SiteActionBuilder.newDesignateMobileEditorForAllSitesAction, like here: https://github.com/wordpress-mobile/WordPress-Android/pull/10752/files#diff-cb360ad2878e66ffe65d8c54967921d6R83

[maxme]

Will this action override all sites configuration? Even sites that have been switched back to Aztec? (that's the reason why I used enableBlockEditor instead).

[daniloercoli]

Yes, it does override all sites configuration. I think there is a way to by-pass this behavior, we need to add a new action to FluxC that does set the parameter set_only_if_empty to true on the underlying network call.
Unfortunately making sequential network calls, even if you delay it from the client, may run into race condition on the PHP backend, and there is no guarantee that the call succeed as expected.

Make sure to test via Developer console first, to check if the parameter is working fine or not.

[maxme]

I'll try the set_only_if_empty parameter. If it doesn't work there is actually another thing we could do: check if there is one aztec setting for any of the user's sites:

• If there is none, then enable gutenberg for all sites.
• If there is at least one, then don't put this user in the rollout cohort.

[daniloercoli]

I like your idea above, it should help on making the logic simpler, and dones't required FluxC changes.

We should also make sure the device has network connectivity, since the action is "fired" but we're not listening for errors I guess. So better to try when the device is online only.

[maxme]

Logic updated in 2ff8e8d - users who have at least one aztec enabled site will be excluded from the cohort

[hypest]

I preferred a modulo logic on the user id over random number generation as the user might use the app on different devices and eventually uninstall/reinstall the app. In these cases, they should have the same default editors.

Just to make sure I understand: the modulo logic will only take part in deciding whether the user is in the rollout cohort, so, on a different device or after uninstall/reinstall the same user will get prompted again... we are not going to keep a flag remotely about whether they got prompted already or not, right?

[maxme]

Just to make sure I understand: the modulo logic will only take part in deciding whether the user is in the rollout cohort,

Yes.

so, on a different device or after uninstall/reinstall the same user will get prompted again...

No they won't get prompted again.

we are not going to keep a flag remotely about whether they got prompted already or not, right?

We're not keeping a remote flag for this, but we're updating the remote mobile editor setting. We only show the dialog when the mobile editor setting is changed by the app (ie. not a user initiated action) from [empty string] (unset) to gutenberg.

[maxme]

This is ready for another pass.

[daniloercoli]

I've tested this again with detailed testing steps, and it's working as expected.

I've only found a small edge case, where the device is with spotty connectivity (or airplane mode) and the code that does the progressive rollout here https://github.com/wordpress-mobile/WordPress-Android/pull/10752/files#diff-cb360ad2878e66ffe65d8c54967921d6R65 does set the local flag, but unable to set the editor preferences on the remote.

• Clear editor settings on the web
• Login to the app
• Put the device in airplane mode and close the app
• Start the app again. This does execute the progressive rollout code
• If you start a new post it does show the prompt and start GB Mobile as expected.
• Close the app
• Make the device online
• Start the app again
• Wait a couple of secs, until the prefs are not synched from the server
• Now the editor prefs should be empty (both local and on the RC card on the web)
• Make a new post ---> Aztec.

Additional testing steps

• Switch to another site and quickly press the new post button
• You see the dialog and then GB Mobile
• Close the Editor
• Tap again on the new post button
• Boom --> Aztec

I think would be safer for us to check if the device is online before executing the rollout, without going crazy on checking other conditions in the code, since it's already quite complex to follow the flows.

[maxme]

I think would be safer for us to check if the device is online before executing the rollout, without going crazy on checking other conditions in the code, since it's already quite complex to follow the flows.

Agreed. We'll ignore the isUserInGutenbergRolloutGroup() flag in the next rollout phase anyway.

I added a network test in eec0ca7

[daniloercoli]

LGTM!