Quality Assurance Team

[loriopatrick]

There are quite a few DApps that depend on a MetaMask being reliable for our users. Is there a community team of QA testers to verify a build before release? If not, I think it would be hugely beneficial and I would be happy to dedicate my time. We're building merklex.io and currently our only solution to the issue is, keep trying or use an old version.

[marsrobertson]

EDIT: me troll 😇

[loriopatrick]

Lol, I think it's more of a release procedure issue than funding. Tho more funding probably won't hurt.

What I would like to see is a request for feedback period of ~48 hours before non critical updates. During this time different dependent application can test the proposed release against their application to make sure everything is acting properly. If all looks good then the release can be published to the extension repositories.

Would be great to get some feedback on this. Does something like this already exist? @danfinlay @kumavis

[bdresser]

@loriopatrick we'd be open to trying something like this. Our current release process is a slow rollout via the Chrome Web Store - usually a day at < 10% to catch any spikes in errors or early anecdotal reports, then gradual increase to 100%.

Today's error was tricky – 4byte went down because our traffic overwhelmed them, which didn't happen during the earlier partial rollout. We're making a note in our internal process to make sure we've communicated with any new external services we're adding.

Anyways - we'd welcome QA help from the community. Any suggestions on the best way to notify folks of a new prod-ready build?

[loriopatrick]

Great to hear. Yeah this bug was a tricky one. In terms of notifying the community, I think a development twitter or mailing list would do well.

[bdresser]

@loriopatrick & others - we're thinking of adding a nightly build of develop to the top of MetaMask's README. We can put a small standing bounty up for any bugs found on the build. Rather than having to post or notify folks every time we want to release, a continuous process might be easier for everyone.

I'd also be happy to create a Github group for those who find & report bugs reliably, and can ping that group when we're headed towards a release. Sound good?

cc @whymarrh @danfinlay

[danfinlay]

We could also build up this process into our release guide. For example, maybe we add:

• For non-urgent releases, a branch should be made for the next version, and a Release Candidate commit should be made, and Pull requested against develop, which should stand for at least two business days for public scrutiny before being pushed to production.
• Release candidates should be given the special Release Candidate label, so the public can more easily track these events.
• Additionally, release candidate events should be published to a public feed: Either a mailing list, or a special twitter account, or a Github group, TBD.

For that last item (the notification process), I'd prefer one that devs can easily subscribe to without our participation or permission, so making it a GH group sounds like an excessive administrative hoop to me. A mailing list or twitter feed are my two current favorites. Ideally something that IFTTT could easily hook into :)

[loriopatrick]

These options sounds great! It would be really helpful for me to have the two business day period for review that @danfinlay suggests. Would the pull request be structured so that the changes from the last release would be outlined in the diff? I imagine that would be ideal but possible difficult depending on how things are currently setup. As for the notification, all those options would work for me, tho I do imagine using an IFTTT integration would be easiest to maintain.

[danfinlay]

Currently develop represents our staging branch, and at the time we are proposing a version bump, we:

• branch off develop
• increment the version and changelog
• PR the version branch to develop
• After merge, we PR develop into master, which represents production.

We would probably have to revise that pattern a little bit to make it more clear which is staging/UAT. @whymarrh is going to take a pass at a specific proposal, and we'll continue from there.

[loriopatrick]

Hi @whymarrh, checking in. Any progress?

[whymarrh]

@loriopatrick not quite yet, we'll be sure to share updates in this thread

[tmashuang]

@loriopatrick Don't really have a system down yet, but here's a build to go through. #6596

[loriopatrick]

Great, thank you. Will give it a test.

[loriopatrick]

@tmashuang

Gave v6.5.0 a run through with our application. Currently failing on the Ropsten Network:

To recreate

• go to https://ropsten.etherscan.io/address/0x603bc175a26c8f74eaf9bc6ef5dd288afb6fc4b8#writeContract
• run approve with _sender = 0xfd2288b96643fe5f75917df6d846998d197bbcdb, _value = 100000000

Gives error: ALERT: Trying to call a function on a non-contract address.

---

Also first contract call after loading extension suggested 0 gas for transaction, after reject, second attempt picks appropriate value.

[loriopatrick]

Opened issue #6602. Currently experiencing error with released version on Chrome Store.

[tmashuang]

@loriopatrick we have setup a github group for the community-qa initiative, I sent you an invite. Also a new release, #6679.

[bdresser]

Hey @loriopatrick & others, thanks for your patience with this

We just launched https://twitter.com/metamask_bot, which will post all release candidates 48-72 hours before we begin an incremental production rollout. It includes a link to download a build for each browser, so you don't have to do any branch management - just download the build, load it in your browser, and play around.

Any bugs found on those builds that are introduced by new code in that release are eligible for a bug bounty at our discretion. We're intentionally leaving it open-ended for now given the way folks have reacted to past bug bounties, but trust that we very much appreciate the help and will incentivize accordingly.

Thanks for pushing us on this. Let me know if you have questions or other suggestions!