Fix e2e tests

[mxstbr]

Status

• WIP
• Ready for review
• Needs testing

@brianlovin we need to run resetdb both before() and beforeEach(),
because otherwise the cy.visit('/whatever') runs before the beforeEach()
so it'd 404.

[mxstbr]

Welp, didn't work. Digging in.

[mxstbr]

That last run failed because

  3) Community settings billing tab should reset the administration email should reset email addres:
     CypressError: cy.visit() failed trying to load:

http://localhost:3001/api/email/validate/test-payments/reset

The response we received from your web server was:

  > 503: Service Unavailable

This was considered a failure because the status code was not '2xx'.

Why would our API respond with 503? Dafuq?

[brianlovin]

3. Community settings billing tab should reset the administration email should reset email addres:
   CypressError: cy.visit() failed trying to load:

This should be fixed if you pull latest alpha - that url is invalid on our api, but I should have fixed it in #2664

[mxstbr]

It seems like the toobusy handling was causing Iris to err out on CI, since that machine probably doesn't have a ton of power. I've disabled that while testing, so let's see what happens now...

[mxstbr]

Good catch, merged in. Let's hope this does the trick!

[brianlovin]

It seems like the toobusy handling was causing Iris to err out on CI, since that machine probably doesn't have a ton of power. I've disabled that while testing, so let's see what happens now...

Interesting; one problem I've been bumping into when doing extensive testing is the api bogging down to the point that the e2e tests fail. Quick api restart fixes it.

[mxstbr]

Interesting; one problem I've been bumping into when doing extensive testing is the api bogging down to the point that the e2e tests fail. Quick api restart fixes it.

That... should never happen 😅

[brianlovin]

That... should never happen 😅

For sure. Things just get super bogged down over time if I run multiple 'run all tests' with cypress in a row.

[mxstbr]

Some random ones are still failing locally, ugh. @brianlovin mind taking this branch and running with it today? I'm not sure how to fix the rest.

[brianlovin]

Sure

[brianlovin]

I'm perpetually getting reconnect issues while running test locally :/

[brianlovin]

Restarting the api fixes it - it just gets bogged down about halfway through a 'run all tests' cycle

[mxstbr]

it just gets bogged down about halfway through a 'run all tests' cycle

That doesn't make any sense. Let me profile it locally and see what the bottleneck is.

[brianlovin]

So all of the tests that are failing still run perfectly in isolation, but when I do a 'run all' iris essentially gets bogged down mid way and starts failing to serve pages (thread views, community settings views, etc).

Not sure how to address this @mxstbr

[brianlovin]

Want to pair on this @mxstbr ?

[mxstbr]

1. Start iris with inspector: TEST_DB=true yarn run dev:api --inspect
2. Then open the Node tab in the chrome dev tools (see https://www.nearform.com/blog/self-detect-memory-leak-node/#nodes-inspect-flag)
3. Run a memory profile and/or a general JavaScript perf profile
4. Run e2e test until it crashes
5. Stop profiling and try and figure out why it crashed

[brianlovin]

Here's an allocation timeline screenshot - seems healthy enough except for one spike early one. But around minute 10 everything in the tests just started crashing.

[brianlovin]

Turns out that profiling an 11min snapshot takes a long time as well. But can confirm that a quick api restart fixes tests - even individual, tiny tests like the pricing page.

[brianlovin]

There are a lot of errors getting thrown in the api though throughout these tests. Here are a few:

Not sure if these are building up over time, if they are related to the test suite speed, if they are genuine errors, or something else :P

[brianlovin]

I tried to make a couple fixes including clearing localstorage along with each db reset - helps with some individual tests, but still same issue collectively.

[mxstbr]

Just merged my CircleCI refactor try from #2742 which seems to have passed on CI—if it doesn't work here I'll remove that change again.

[spectrum-bot]

	Warnings
⚠️	These modified files do not have Flow enabled: shared/testing/setup.js shared/testing/teardown.js

Generated by 🚫 dangerJS

[mxstbr]

Only a single test failed in that last run oh my god. 🤞

[mxstbr]

So close, and that one failure is an actual failure. (fixed in the last commit) Let's hope that this is it! 🤞

[mxstbr]

Argh now they're failing again 😢 I thought for just a second I'd fixed it.

[brianlovin]

Unable to figure out the proper way to patch in a check for the existence of the testing db, but I think #2810 is probably a better place for that patch to come in. Since this PR fixes a bunch of tests and frontend code, let's ship it.