[Spike] Check which release tests are already automated

[sauloperez]

Description

- As a: core team member
- I want to be able to do: gain efficiency and confidence on the release process so that I can focus my efforts on bringing more value to our users.

Acceptance Criteria

We need to get a list of the tests that are performed as a release testing (https://docs.google.com/document/d/15EdXKb4B3BS8gekD6_9PDdPTZsjUCAVS2011zRf55lM/edit#heading=h.alpdtdl2tc74) that are already included in our automated test suite and how well covered they are.

Then, we can decide which ones we stop testing manually.

[daniellemoorhead]

I was talking to @mkllnk about this last week. We thought we could make a list and then each release we do we check off one of them. If it's not automated, we add that to be automated. If it is then we wait till the next release to check the next one.

@mkllnk did I remember correctly?

[RachL]

Yep! But this issue is only about the list :) We would need separate issue for the automated tests.

[sauloperez]

yep, not jumping into implementations yet. But yes, that iterative process you mention seems really good @daniellemoorhead

[sauloperez]

I copy here the contents of the Testing Handbook v2 for reference

How to do a sanity check (release testing)?

First steps

• Check signing up for a customer or a producer: https://github.com/openfoodfoundation/openfoodnetwork/blob/master/spec/features/consumer/authentication_spec.rb#L47-L86
• Check that the producer account is correctly displayed on the map
• Complete your enterprise setting and check that both payment method and shipping method works for stripe and paypal
• Products : create, edit (add an image for example), duplicate (check that image is also duplicated), delete
• Can you create on OC? Does it show in front office? Can you duplicate / edit / delete it? If you delete products do they still appear in the shopfront?

Checkout

• Select an open shop and add item to the cart
• Press continue
• See your cart, but go back “continue shopping” in order to add another product
• Press continue
• Delete one of the first product you’ve added on the cart and change the quantity on another one
• Proceed to checkout, and login
• Select delivery and cash payment option (repeat for stripe, paypal and cash)
• Check you get a confirmation and a confirmation email
• Check your order is appearing correctly in /admin/orders
• Order for producer: can you see it? Can you filter on the page? Can you ship it? Can you capture payments? Can you create an order as an admin?
• Customers : add a customer and edit its credentials afterwards

Bulk Order Management

• Go to /admin/orders/bulk_management
• Are you able to change weight and was the order total recalculated accordingly?
• Are you able to change quantity and was the order total recalculated accordingly ?
• Are you able to cancel an order
• Are you able to add columns?
• Are you able to search for a product?
• Are you able to change the dates and see results accordingly?

Subscriptions

• Can enable from superadmin?
• Can create schedule?
• Can trigger subs?
• Get emails?
• Out of stock messaging works?
• Editing sub, pausing sub?

Inventory

• Override price
• Override stock levels
• Check that override don't have an effect on the producer stock level
• Reset stock levels

Reports

• Orders and fulfillment reports
• Packing reports
• Customers
• Xero
• Sales tax

[sauloperez]

Minutes of the call we just had with @RachL about the Release testing handover:

In the discussion we did spot few issues that need to be tackled and discussed separately

• Testing in iOS and Android devices
• What browsers and versions do we actually support?
• We need to ensure the mobile UI also works

having seen that, we agreed that our first focus is automating this release testing for desktop for now. This will already be a huge improvement.

Finally, we identified the need to investigate exploratory testing. As we transition towards not having any manual release testing, the always necessary bit of manual testing we'll not depend on releases anymore and we'll become something else. Also, POs will eventually test acceptance criteria manually to consider things done.

[mkllnk]

You remembered that correctly, @daniellemoorhead. This spike is going in the same direction. :-)

[sauloperez]

@lin-d-hop @daniellemoorhead @RachL I checked the checkboxes with the ones I think we're properly covered. I created the Release testing automation wiki page to report the findings and create the necessary issues later on. My goal is to evolve this page until it says. "100% automated, move on." 🎉

You'll see some items need clarification like the item "Complete your enterprise setting and check that both payment method and shipping method work for stripe and paypal".

[luisramos0]

I guess we can convert this to an epic that includes automating each of the tests?

[sauloperez]

I would do that in a separate issue as this is only for the spike and so we close this when done.

[lin-d-hop]

I'd like to add some smoke testing on Tagging. This is a powerful feature that we have let slide. Let's bring it back to a healthy state!

It is a pain to test manually, which is I think part of the problem. But automated testing will be a godsend!

I'll do some work on that when I have a moment to breathe :-)

[lin-d-hop]

More good things to have on this list:

• VAT
• Product Import
• Invoicing
• Order cycle notification emails to producers

[sauloperez]

Ok, so the spike is done. As mentioned above the outcome of my investigation is listed in https://github.com/openfoodfoundation/openfoodnetwork/wiki/Release-testing-automation. From that, in #4127 (comment) I checked each of the tests that IMO are not worth testing manually since they are properly covered by our automated test suite by the tests that interact with a browser (they live in https://github.com/openfoodfoundation/openfoodnetwork/tree/master/spec/features).

I haven't considered your suggestions yet @lin-d-hop . IMO these fall outside of this scope and deserve other issues.

I think the next step should be having a call and discuss whether we:

• Stop doing release testing altogether
• Only test the unchecked items above as part of the release testing

But also:

• What we think the next steps should be

I summon you @RachL, tester master but perhaps @lin-d-hop or someone else would like to attend?

[sauloperez]

@RachL and I met yesterday and come up with a proposal for you all in https://community.openfoodnetwork.org/t/release-testing-next-steps/1741

[RachL]

For this last release I change the perimeter of the release test. If we all agree on this new perimeter maybe we can close this issue and start opening an epic with what we need to automate?

https://docs.google.com/document/d/1lsewB9hohQTsEDrSA2mF_f0Ni1eIH58dIdAFZcb58j4/edit#

[RachL]

OK as @sauloperez created #4333 let's keep on discussing there. I'm closing this issue 🎉