Run master commits on Taskcluster #9226

jgraham · 2018-01-29T12:02:25Z

This PR actually has two parts:

A docker image for running tests. I assume this will be relatively uncontroversial, although there are certainly ways that we could improve the details.
The required configuration to run each push on master on TaskCluster. This won't do anything unless the relevant GitHub integration is enabled. If it is, however, it will typically provide the full results of the testsuite within about an hour of a merge to master. You can see an example run from my fork. Each job uploads an artifact containing the results from that chunk in wptreport.json format; these could be combined to give the full set of results for each push.

This change is

ghost · 2018-01-29T12:19:40Z

Build PASSED

Started: 2018-03-16 17:35:33
Finished: 2018-03-16 17:53:33

View more information about this build on:

foolip · 2018-04-13T09:51:40Z

A docker image for running tests

Interesting! @jugglinmike, very relevant to web-platform-tests/results-collection#535 (comment)

will typically provide the full results of the testsuite within about an hour of a merge to master

Wow. That'd achieve our wpt.fyi results are available within 1 hour for at least Firefox, and way sooner than I thought was possible.

@lukebjerring @boaz @mariestaver @jugglinmike, this could change our roadmap drastically, so let's discuss at the earliest opportunity.

foolip · 2018-04-13T09:53:41Z

Each job uploads an artifact containing the results from that chunk in wptreport.json format; these could be combined to give the full set of results for each push.

@Hexcles, can you take a look at the bits around this, to see if it'll be a trivial integration with a wpt.fyi results POST endpoint? Seems like the token management and such would have to be very similar to what we've speculated about for PRs on Travis.

foolip · 2018-04-13T09:55:05Z

.taskcluster.yml

+        \ &&\n            git config advice.detachedHead false &&\n            git\
+        \ checkout {{event.head.sha}} &&\n            ./tools/ci/ci_taskcluster.sh\
+        \ chrome reftest 1 6"]
+    image: harjgam/web-platform-tests:0.5


Should we have a "web-platform-tests" organization on Docker Hub? Is that a thing?

It is and we can; I just created an org with the namespace webplatformtests; neither wpt or web-platform-tests worked, but I can't tell if they were taken or if there are some character/length restrictions.

Where does the code live for creating the docker images?

foolip · 2018-04-13T09:56:28Z

.taskcluster.yml

@@ -0,0 +1,687 @@
+allowPullRequests: collaborators


This configuration file is huge and fairly repetitive. I assume it has to be checked in to the repo like .travis.yml, but could parts of it be generated?

Ha, there is a script :) Suggestion is then to add a comment at the top pointing to the script.

foolip · 2018-04-13T09:57:24Z

tools/ci/ci_taskcluster.sh

+if [ $1 == "firefox" ]; then
+    ./wpt run firefox --log-tbpl=- --log-tbpl-level=debug --log-wptreport=../artifacts/wpt_report.json --this-chunk=$3 --total-chunks=$4 --test-type=$2 -y --install-browser --no-pause --no-restart-on-unexpected
+elif [ $1 == "chrome" ]; then
+    ./wpt run chrome --log-tbpl=- --log-tbpl-level=debug --log-wptreport=../artifacts/wpt_report.json --this-chunk=$3 --total-chunks=$4 --test-type=$2 -y  --no-pause --no-restart-on-unexpected


Which version of chrome will this be?

Based on https://tools.taskcluster.net/groups/bLpNdT49RGCR7nEhELQE1A I'm guessing chrome dev, but I can't tell why.

It gets installed using apt in tools/docker/start.sh, pretty much like we do for travis. This probably isn't the optimal way to do this; the current setup is more of a proof of concept than anything, so I'm sure there are plenty of improvements we can make.

Ah, well as long as it's not just accidentally chrome dev, that's just fine!

foolip · 2018-04-13T12:36:19Z

tools/docker/start.sh

+cd web-platform-tests
+git pull --depth=1
+
+# Install Chome unstable


In #8838 I tried to banish "unstable" everywhere except package name, can you call it "Chrome dev"?

foolip · 2018-04-13T12:36:49Z

tools/docker/start.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+sudo sh -c 'echo "


@gsnedders wrote something to generate this output, can you use that here?

./wpt make-hosts-file does this

Hexcles · 2018-04-17T01:28:50Z

tools/docker/Dockerfile

+
+RUN apt-get -qqy install \
+    firefox \
+    chromium-browser \


IIUC, the extra dependencies below are dependencies of Chrome (that you happen to know) but not of chromium-browser.

IMHO, since Chrome is already distributed in a deb package, we can perhaps skip this (i.e. only install firefox). See my next comment in start.sh for dependency problems.

Hexcles · 2018-04-17T01:30:54Z

tools/docker/start.sh

+if ! sudo dpkg --install $deb_archive; then
+    sudo apt-get -y install --fix-broken
+    sudo dpkg --install $deb_archive
+fi


I think this is sufficient for solving Chrome's dependencies, so there's no need for installing chromium-browser in the Docker image.

Besides, you might want to try gdebi, which installs a local deb package along with its dependencies (from apt). I think it's available by default on Ubuntu.

@Hexcles This does the same, FWIW. (Also, BTW, you're not on IRC.)

Yeah I know it's the same, but it'd save a couple lines :P

foolip · 2018-04-17T08:39:59Z

I'll step away from the detailed code review now as @Hexcles will do that, makes the most sense since it needs to be integrated together with the results upload endpoint, somehow.

foolip · 2018-04-17T08:46:49Z

I have a question about browser version management. If each shard individually downloads https://dl.google.com/linux/direct/google-chrome-unstable_current_amd64.deb then once every few days it's possible they will not get the same version. At minimum it would be good if the browser version is recorded in the wptreport JSON so that we can detect this. Even better would be if the browser version is somehow managed together with the container.

The very simplest would be to have the browsers installed in the container itself, which would mean updating the container daily, but that can be automated pretty easily. https://github.com/whatwg/html-build/blob/master/ci-deploy/outside-container.sh does this.

How we then ensure that the same Docker image is used for each Taskcluster shard I don't know. But it'd be very good if it's also recorded in the wptreport somehow. (A bit backwards, I know, but a variable could be passed in.)

Hexcles · 2018-04-23T18:05:12Z

Sorry I was busy (travelling) last week and not available on IRC.

I've glanced through the PR. Apart from the nit comment I made earlier, I have one high-level question: is there no way to configure the number of chunks other than using a script to dump each chunk as a task? I'm a bit surprised if Taskcluster doesn't some sharding this natively.

jgraham · 2018-04-24T21:40:02Z

So. Tthe good way to support that is to write a decision task that schedules all the other tasks dynamically. We should do that, but it's not totally necessary for the first iteration.

It's possible that there's some syntax for parameterising the yaml file; I can look for documentation. I'm not very pleased with the current setup either, but it seemed like the fastest path to something that worked.

jgraham · 2018-04-24T21:46:27Z

https://bugzilla.mozilla.org/show_bug.cgi?id=1335915 seems to be the bug for enabling the fancy templating stuff in .taskcluster.yml; since that's still open I assume it's not possible yet, but we should look to switch to it once that bug is fixed.

This is a docker image based on the Ubuntu1604 base, with the runtime deps of Firefox and Chrome installed. There is also a shallow clone of wpt to act as a placeholder. The start.sh script will update this clone and start Xvfb, so it's possible to run tests.

TaskCluster is Mozilla's CI system, which also provides a GitHub integration. With this configuration enabling the TaskCluster integration on GitHub will cause each pust to master to create a TaskCluster run consisting of the full testsuite in latest Firefox Nightly and latest Chrome dev. The run will be chunked into 12 pieces for testharness.js tests and 6 for reftests, plus one for wdspec tests. With this chunking the end to end time for completing the run is around an hour (the task timeout is set to 1.5 hours). The resulting tasks upload wptreport.json files that can be used to reconstruct the results of the full run. There are also tbpl log files that could be used with e.g. Mozilla's reftest analyzer to inpect reftest failures.

This isn't very useful unless there is expectation data avaiable, which is not the common case

Also add a comment to indicate that the tasks file is generated.

Hexcles

LGTM except for a nit.

(I agree with using a script to generate .taskcluster.yml for now. We can explore simplifications later.)

Hexcles · 2018-04-25T17:09:00Z

tools/docker/Dockerfile

+# to know that chrome requires
+
+RUN apt-get -qqy install \
+    firefox \


Sorry I meant everything below could be removed, i.e. just apt-get -qqy install firefox should be enough.

The other packages below are "some extra deps we happen to know that chrome requires" according to the comment above. And since gdebi chrome.deb will take care of them, we don't need to guess and install these dependencies any more.

The comment above should also be updated to reflect that we are only installing firefox now.

Installing more stuff at build time does have the advantage that we don't do so much work at runtime, so generally it makes more sense to move stuff into the docker build rather than the other way around.

In this case we agreed on irc that it doesn't make much difference either way and there are plenty of performance optimisations and potential cleanup left on the table, so it's OK to just merging what we have now.

jugglinmike · 2018-04-25T17:49:42Z

tools/docker/start.sh

+deb_archive=google-chrome-unstable_current_amd64.deb
+wget https://dl.google.com/linux/direct/$deb_archive
+
+sudo gdebi -n $deb_archive


Note: Installing Google Chrome will add the Google repository so your system
will automatically keep Google Chrome up to date. If you don’t want Google's
repository, do sudo touch /etc/default/google-chrome before installing the
package.

Source: https://www.google.com/intl/en/chrome/browser/

Is this relevant given the expected lifespan of the container?

No; any changes only persist for the lifetime of the container, which is just for a single test run.

foolip · 2018-04-25T19:03:51Z

Awesome! Looks like the very first run on master is this one, still ongoing:
https://tools.taskcluster.net/groups/Bv2TZUgeSyGhXblM8UU0aA

foolip · 2018-04-25T19:05:53Z

And here are actual results from shards from a shard that has finished:
https://taskcluster-artifacts.net/N59MId9GQoir2KBBxEH_FA/0/public/results/wpt_report.json

This will be great :)

wpt-pr-bot added the infra label Jan 29, 2018

wpt-pr-bot requested a review from gsnedders January 29, 2018 12:02

jgraham force-pushed the jgraham/taskcluster branch from 5bb339f to 696c6c3 Compare January 30, 2018 10:59

jgraham requested a review from mjzffr February 5, 2018 14:14

jgraham force-pushed the jgraham/taskcluster branch from 696c6c3 to ab20739 Compare February 6, 2018 13:28

jugglinmike mentioned this pull request Apr 12, 2018

Synchronize browsers used by WPT CI and results collector web-platform-tests/results-collection#535

Open

foolip reviewed Apr 13, 2018

View reviewed changes

Hexcles reviewed Apr 17, 2018

View reviewed changes

This was referenced Apr 17, 2018

Expose results for experimental browsers in a way that is easily linked web-platform-tests/wpt.fyi#60

Closed

Run PRs on Taskcluster #10503

Closed

jgraham added 6 commits April 25, 2018 14:53

Don't return failure for wpt run if not all tests pass.

5be6bb5

This isn't very useful unless there is expectation data avaiable, which is not the common case

Rename wpt generate-tasks to wpt make-tasks

f3672de

Also add a comment to indicate that the tasks file is generated.

Switch to gdebi for installing Chrome and use make-hosts-file

aed9615

Update to the latest version of the docker image

e7bae7a

jgraham force-pushed the jgraham/taskcluster branch from 4f87b52 to e7bae7a Compare April 25, 2018 16:30

Hexcles approved these changes Apr 25, 2018

View reviewed changes

jugglinmike reviewed Apr 25, 2018

View reviewed changes

jgraham merged commit 875c65e into master Apr 25, 2018

foolip deleted the jgraham/taskcluster branch April 25, 2018 18:36

foolip mentioned this pull request Apr 26, 2018

Taskcluster master runs fail very often #10653

Closed

foolip mentioned this pull request May 8, 2018

Run stable Chrome and Firefox on Taskcluster daily or weekly #10908

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run master commits on Taskcluster #9226

Run master commits on Taskcluster #9226

jgraham commented Jan 29, 2018 •

edited by wpt-reviewable-bot

Loading

ghost commented Jan 29, 2018 •

edited by ghost

Loading

foolip commented Apr 13, 2018

foolip commented Apr 13, 2018

foolip Apr 13, 2018 •

edited

Loading

jgraham Apr 13, 2018

foolip Apr 13, 2018

foolip Apr 13, 2018

foolip Apr 13, 2018

foolip Apr 13, 2018

foolip Apr 13, 2018

jgraham Apr 13, 2018

foolip Apr 13, 2018

foolip Apr 13, 2018

foolip Apr 13, 2018

gsnedders Apr 17, 2018

Hexcles Apr 17, 2018

Hexcles Apr 17, 2018

gsnedders Apr 17, 2018

Hexcles Apr 23, 2018

foolip commented Apr 17, 2018

foolip commented Apr 17, 2018

Hexcles commented Apr 23, 2018

jgraham commented Apr 24, 2018

jgraham commented Apr 24, 2018

Hexcles left a comment

Hexcles Apr 25, 2018 •

edited

Loading

jgraham Apr 25, 2018

jugglinmike Apr 25, 2018

jgraham Apr 25, 2018

foolip commented Apr 25, 2018 •

edited

Loading

foolip commented Apr 25, 2018

Run master commits on Taskcluster #9226

Run master commits on Taskcluster #9226

Conversation

jgraham commented Jan 29, 2018 • edited by wpt-reviewable-bot Loading

ghost commented Jan 29, 2018 • edited by ghost Loading

Build PASSED

foolip commented Apr 13, 2018

foolip commented Apr 13, 2018

foolip Apr 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foolip commented Apr 17, 2018

foolip commented Apr 17, 2018

Hexcles commented Apr 23, 2018

jgraham commented Apr 24, 2018

jgraham commented Apr 24, 2018

Hexcles left a comment

Choose a reason for hiding this comment

Hexcles Apr 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foolip commented Apr 25, 2018 • edited Loading

foolip commented Apr 25, 2018

jgraham commented Jan 29, 2018 •

edited by wpt-reviewable-bot

Loading

ghost commented Jan 29, 2018 •

edited by ghost

Loading

foolip Apr 13, 2018 •

edited

Loading

Hexcles Apr 25, 2018 •

edited

Loading

foolip commented Apr 25, 2018 •

edited

Loading