Test suite does not run WebDriver tests on Windows

[rmorshea]

Summary

At the moment WebDriver tests are not run on Windows because they are failing for some unknown reason.

How To Work On This Bug

At the moment WebDriver tests are being skipped. To unskip them when you run the test suite comment out these lines:

https://github.com/idom-team/idom/blob/2804ae2620d4a2b719684460fd56b97deb65b4ba/tests/conftest.py#L151-L161

Current Findings

I don't have a Windows machine myself so it's quite difficult to try and debug. Further the tracebacks and logs that are produced don't really help much in figuring out what the problem is. The only thing I've been able to identify is that not all the files are being loaded from the server when loading up the index.html page.

On Linux (expected behavior):

[2021-01-29 16:40:55 -0800] - (sanic.access)[INFO][127.0.0.1:56132]: GET http://localhost:5000/  302 0
[2021-01-29 16:40:55 -0800] - (sanic.access)[INFO][127.0.0.1:56132]: GET http://localhost:5000/client/index.html  200 387
[2021-01-29 16:40:55 -0800] - (sanic.access)[INFO][127.0.0.1:56132]: GET http://localhost:5000/client/index.js  200 982
[2021-01-29 16:40:55 -0800] - (sanic.access)[INFO][127.0.0.1:56132]: GET http://localhost:5000/client/web_modules/idom-client-react.js  200 39291
[2021-01-29 16:40:55 -0800] - (sanic.access)[INFO][127.0.0.1:56132]: GET http://localhost:5000/client/web_modules/htm.js  200 1220
[2021-01-29 16:40:55 -0800] - (sanic.access)[INFO][127.0.0.1:56136]: GET http://localhost:5000/client/web_modules/common/index-6ed86a98.js  200 9836

On Windows (missing GET requests):

[2021-01-30 00:32:17 -0000] - (sanic.access)[INFO][127.0.0.1:50486]: GET http://127.0.0.1:50482/  302 0
[2021-01-30 00:32:17 -0000] - (sanic.access)[INFO][127.0.0.1:50486]: GET http://127.0.0.1:50482/client/index.html  200 403
[2021-01-30 00:32:17 -0000] - (sanic.access)[INFO][127.0.0.1:50486]: GET http://127.0.0.1:50482/client/index.js  200 1016

This is odd because index.js should import idom-client-react.js which itself imports other JS files, however the expected requests are missing when running the test suite in CI.

Get Support

I'm happy to help debug this issue with anyone who's interested in investigating.

[rmorshea]

@acivitillo I noticed that you're working with IDOM on Windows. Is there any chance you might be able to help out with this issue?

[acivitillo]

I will take a look to the tests. I am not too familiar with the internals of idom so this might take a while and I might have a few questions for you.

On Windows, I was able to make idom work by doing pip install idom==0.34.0. I get the below files when visiting an idom page, so far the app is pulling the right files.

[rmorshea]

Actually, now that I think about it, this issue is pretty old. It would be good for me to try and reproduce it now and update any logs before you give this a go.

[Archmonger]

On upstream/main@b036f03 (latest to date)

nox -s test -- --headless

• Docker build failure: snowpack could not load fast-json-patch.
• Test Docs failure: Unknown error. No error codes or console output.
• Test Python Suite failure: Codecov did not reach 100% coverage
• Test JavaScript warning: NPM Audit 5 vulnerabilities (4 moderate, 1 high)
• Test JavaScript Style failure: Prettier detected formatting is required within client\packages\idom-app-react

nox > * test: success
nox > * test_python: success
nox > * test_docs: failed
nox > * test_javascript: success
nox > * test_python_suite: failed
nox > * test_python_types: success
nox > * test_python_style: success
nox > * test_python_build: success
nox > * test_javascript_suite: success
nox > * test_javascript_build: success
nox > * test_javascript_style: failed

[rmorshea]

There's probably more problems that are currently being hidden. In order to get the suite to pass on Windows in CI, I've been skipping all tests that require the browser (hence the missing coverage).

[Archmonger]

I'll edit out the conditional and see what else breaks. Will report back in a few minutes.

[Archmonger]

Yep, everything webdriver related is broken. Probably because Selenium on Windows is broken in general.

[rmorshea]

I've been wanting to switch to an async web driver. If one of the available options uses a different web driver solution, that might solve these issues too.

[acivitillo]

I know we want to make Windows working directly with idom, but given the fact idom probably needs to be deployed as a container anyway, could we maybe fix this with a tutorial "how to setup your dev environment with idom and docker".

[Archmonger]

IDOM works perfectly on Windows. I've been using it for a couple months now with no Windows specific bugs. Notably, I've been doing my development on bare-metal with no containers.

It's just the IDOM tests that are broken, due to reliance on the selenium webserver.

[rmorshea]

Ultimately we should try and get this working on Windows, but for now that's a great option. I can write up an issue to capture that tomorrow or you should feel free to use the "Plan a Task" issue template.

[acivitillo]

I confirm idom works great on my Windows too.

[rmorshea]

Yeah, I think the main thing I want to fix here is the contributor experience. I've been developing this on my own for the most part, but at some point I'm gonna need help, so making sure its easy to make contributions is going to be important.