Tests are not running

[benmccann]

Describe the bug

It looks like our browser tests are not running

This would explain recent regressions (e.g. #2041)

I'm hesitant to merge any PRs until this gets fixed. I've added a note to the PR template informing PR submitters of this

Help looking into this issue would be appreciated

Reproduction

If I make the first line throw new Error('this should fail'); the test suite still passes

kit/packages/kit/test/apps/basics/src/routes/routing/_tests.js

Line 205 in 459efa2

test('falls through', '/routing/fallthrough/borax', async ({ page, clicknav }) => {

Logs

No response

System Info

`master`

Severity

blocking an upgrade

Additional Information

No response

[harvey-k]

There seems to be a large number of tests being skipped.

This appears to be the last build of master where most tests were not skipped:
https://github.com/sveltejs/kit/runs/3057873558?check_suite_focus=true#step:11:276

And this looks like the first master build where many tests are skipped, and is still occurring:
https://github.com/sveltejs/kit/runs/3060779795?check_suite_focus=true#step:11:276

[harvey-k]

Test 'successfully' fails again when reverting packages/kit/tset/apps/basics/package.json back to the original code here:
c9f31dd#diff-cd5fb14d34909f44d04d296204d55dc16ccd7540c5a03f56106856bfcde2a0be

If I submit a pull request, should I attempt fixing the other tests?

[Mlocik97]

actually I can see ../ 3x = actual folder is kit/packages/kit/test/,.. you maybe want actual folder to be kit/packages/kit/, so it means we need ../ 4x

[benmccann]

I just changed it locally back to:

		"dev": "svelte-kit dev",
		"build": "svelte-kit build",
		"preview": "svelte-kit preview"

It doesn't look like it changes anything. I changed a test so that it should fail and I'm still not seeing any test failures

The reason for changing was so that I could run the test projects myself with pnpm run dev and see what the tests were seeing. I added a line to the readme describing this:

You can run the test server with cd packages/kit/test/apps/basics; pnpm run dev to hit it with your browser.

I think the path is correct as is since the following works:

cd packages/kit/test/apps/basics/
ls ../../../svelte-kit.js

[benmccann]

This appears to be the last build of master where most tests were not skipped:
https://github.com/sveltejs/kit/runs/3057873558?check_suite_focus=true#step:11:276

And this looks like the first master build where many tests are skipped, and is still occurring:
https://github.com/sveltejs/kit/runs/3060779795?check_suite_focus=true#step:11:276****

Aha!! Thank you! That helped me figure it out: e8b391b

[Mlocik97]

".only()" 😆

[Mlocik97]

now we see, there is tons of errors... yikes

[benmccann]

yeah 😝 that was before I found FILTER in the code I guess. the custom test runner meant I couldn't figure out how to do even basic things since standard stuff like mocha's --grep didn't work and there were no docs. I tried to document some stuff, so hopefully it's a bit better now

but yeah, now time to fix all the errors 😢

[Mlocik97]

I also have idea, we should probably set in workflow, that skipped tests must be = 0.

[Mlocik97]

So test were not running from Jul 13, 2021... Yikes,... how a lot of code changes in half month?

[sebastianrothe]

I also have idea, we should probably set in workflow, that skipped tests must be = 0.

It would be a good idea to use eslint-plugin-jest and enforce the linting in the CI. The plugin does not allow skipping or restricting tests.