Frontend testing using puppeteer

[juliusknorr]

TODO:

• failing tests when running in drone
• Use custom docker image
• compare with DRONE_REPO_BRANCH
• Add static frontend for viewing the test results
• Maybe use https://github.com/nextcloud/docker-ci/pull/84/files for running servers
• Add script to run tests locally
• deploy screenshots of failing tests to some location
• Add qhd, uhd, uwqhd resolutions
• Add docker image Add Dockerfile for ui-regression tests docker-ci#88

Results are pushed to s3 right now: https://s3.eu-central-1.amazonaws.com/nextcloud-ui-regression/nextcloud/server/8357/index.html

Running tests locally

cd tests/ui-regression
# adjust settings in config.js

# run all tests
node runTests.js
# run just one test suite
node runTests.js settings

Tests

installation:

• installation page
• auto login after installing

login:

• log in page
• forgot password
• files page after login

files:

• file list popover
• files sidebar (comments, sharing, versions)

settings:

• personal settings page
• admin settings - sharing, security, theming, encryption, additional, tips-tricks
• apps management
• user management

files_sharing:

• public share link
• public share link not found

[skjnldsv]

qhd & uhd? :)

[juliusknorr]

I'll have a look when I the tests are running fine in drone. 👍

[skjnldsv]

Also uwqhd! It starts to be more and more common nowadays (21/9 monitors) :)

[juliusknorr]

Some input about how we should handle failing screenshot comparison would be nice. Ideally we would have the following:

• Drone tests do not fail if screenshots are not the same
• Upload screenshots and diff to some storage (like s3) http://plugins.drone.io/drone-plugins/drone-s3/
• Leave a comment in the PR about what tests are failing with a link to the screenshots

@MorrisJobke If i remember this correctly, we can use secrets in drone securely and use them within pull requests, if the drone job doesn't run any script from the git repo. So I thought we can create a drone plugin that is separate from the main test run for publishing the screenshot files and commenting on github. Is there anything else that I need to be aware of here?

[MorrisJobke]

@MorrisJobke If i remember this correctly, we can use secrets in drone securely and use them within pull requests, if the drone job doesn't run any script from the git repo. So I thought we can create a drone plugin that is separate from the main test run for publishing the screenshot files and commenting on github. Is there anything else that I need to be aware of here?

Sounds good. Yes - should be fine. I would still like to have a new GitHub account for this and not use the Nextcloud-bot, because this one has push permissions to the repos. On the other hand the API token should also have quite limited scope if set up correctly.

[MorrisJobke]

Just drop the .weasel.rocks, as it is under the correct domain now.

[juliusknorr]

As discussed with @rullzer Lets get this in and fix possible risky tests after. Failing UI regression tests will not cause blocking of pr merge as of right now.

Tests that sometimes fail/existing issues:

• install/advanced: Those will succeed once this is merged to master. See 4bd7cb7 Sadly I cannot reproduce locally so it is pretty hard to debug

• failure caused by app menu: This is most likely an issue with the app menu right now, that is caused by moving it to flexbox (Header issues with small screens #10452 probably)
  

[juliusknorr]

Failing acceptance-app-files unrelated: https://drone.nextcloud.com/nextcloud/server/9324/234 Fails on master as well