w3c-test.org/tools/runner/index.html blocked by manifest update

[tkent-google]

Steps

1. Open http://w3c-test.org/tools/runner/index.html
2. Uncheck "Reftests" and "Manual tests"
3. Change "Run tests under path" field to "/dom/"
4. Press [Start] button

I expect tests starts in a few minutes, however "Updating and loading test manifest; this may take several minutes" won't finish for several hours.

@sideshowbarker

[jdsmith3000]

I see an internal server error 500 that points at the "Manifest generation failed" line below. The XHR for the manifest is completing, but status is apparently failed.

generate: function(loaded_callback) {
    var xhr = new XMLHttpRequest();
    xhr.onreadystatechange = function() {
        if (xhr.readyState !== 4) {
            return;
        }
        if (!(xhr.status === 200 || xhr.status === 0)) {
            throw new Error("Manifest generation failed");
        }
        this.do_load(loaded_callback);
    }.bind(this);
    xhr.open("POST", "update_manifest.py");
    xhr.send(null);
},

[sideshowbarker]

OK I “fixed” this—just by manually regenerating the manifest file from the command line on the server. So I think if you try again it should work.

That said, whatever was causing the problem may (probably) happen again.

When/if it does, I can respond more quickly if you ping me on #testing or #whatwg or #chromium

But raising an issue here is fine too, as long as you’re not in hurry and it’s not blocking you

[aleygues]

@sideshowbarker thank you it was working fine this afternoon (for /presentation-api/controlling-ua), but yet it seems to be broken again.

runner.js:47 POST http://w3c-test.org/tools/runner/update_manifest.py 500 (Internal Server Error)
runner.js:42 Uncaught Error: Manifest generation failed at Manifest.<anonymous> (runner.js:42)

[sideshowbarker]

@aleygues thanks for the heads-up—it should be working again now (though as before I have no idea why it failed nor if/when it will fail again)

[tkent-google]

Is this an issue of wptserve/manifest software logic, or an issue specific to the w3c-test.org machine?

[sideshowbarker]

Is this an issue of wptserve/manifest software logic, or an issue specific to the w3c-test.org machine?

I believe it’s a problem with the wptserve/manifest software logic but I can’t prove it. I think it gets exposed on w3c-test.org just because wptserve is running continuously there and that puts it under more load than it seems to be capable of dealing with—wptserve does not do concurrency well, as far as I can see. We have plenty of CPU and RAM resources on w3c-test.org —the server environment is 4 fast cores and 8GB of RAM with at least 5GB of it free always. So wptserve is not stressing the environment at all.

I guess it could be some filesystem I/O limitations in wptserve that cause but I dunno.

Anyway the gist of it is, I think we would be having the same problems with wptserve when we run it persistently (the way it is on w3c-test.org)—regardless of what environment we run it in.

[mgiuca]

There was a discussion about this in #6287 but I think it's more appropriate to continue here. Quoting @sideshowbarker below:

Yeah that sounds like what may be happening on w3c-test.org but I don’t have the free cycles to try to troubleshoot it.

Instead what I did a day or two ago to try to work around it is, I set up a cron job that runs once an hour at :15 after that deletes the MANIFEST.json file and then runs ./manifest to regenerate it.

(Was that in response to this issue?)

I thought that’d prevent the /tools/runner/index.html problem, but based on your comments it sounds like it might’ve instead made things worse. If so, let me know and I’ll remove the cron job.

I think so. Maybe try removing it and we'll see if it gets better.

But the thing is, we don’t have anybody interested in maintaining the /tools/runner/index.html source, so it seems like it’s really not something we should be continuing to encourage anybody to rely on unless/until we have somebody actively maintaining it.

What we have instead is wptrun, as a command-line alternative intended to be easy to use and to solve the same user/information need that people are wanting to use /tools/runner/index.html for.

I had some trouble using wptrun (as with the browser-based runner, I couldn't get the SSL stuff to work locally). At least I can use localhost to hack around that with the tools/runner. But it did at least load Chrome and run the automated tests.

I also couldn't find a way to pass command-line flags to the browser using wptrun (I need to pass --enable-experimental-web-platform-features to Chrome to test my feature). Is there a way?

However, I don't think wptrun satisfies a lot of the use cases for tools/runner:

• Manual testing: does wptrun attempt to run manual tests at all? My tests are almost completely manual.
• Easy testing across many PCs: It's trivial to test tools/runner once the test is landed in the WPT repo. I just point any browser to the URL from any PC. I don't have to set up git, Python, virtualenv, and everything else required to set up wptrun. (Imagine I'm testing across Linux, Windows and Mac... I don't want to have to figure out the platform-specific quirks to set up the development environment on every platform.)
• Testing on mobile: I don't think it's possible to use wptrun at all on Android or iOS is it? Whereas it's again trivial to use tools/runner on mobile platforms.

So I'd really like to see tools/runner continue to work in the future. Apart from the manifest loading issue, it seems to be working great. I can't commit to maintaining it but I can look into the manifest loading as a once-off, if you need the extra hands.

[sideshowbarker]

Instead what I did a day or two ago to try to work around it is, I set up a cron job that runs once an hour at :15 after that deletes the MANIFEST.json file and then runs ./manifest to regenerate it.

(Was that in response to this issue?)

yeah

I thought that’d prevent the /tools/runner/index.html problem, but based on your comments it sounds like it might’ve instead made things worse. If so, let me know and I’ll remove the cron job.

I think so. Maybe try removing it and we'll see if it gets better.

OK I’ve turned it off for now.

So I'd really like to see tools/runner continue to work in the future. Apart from the manifest loading issue, it seems to be working great. I can't commit to maintaining it but I can look into the manifest loading as a once-off, if you need the extra hands.

yeah we could really use help with troubleshooting and fixing the current manifest-loading issue, and we have nobody at this point who’s stepped forward to put time into investigating it

[mgiuca]

OK I’ve turned it off for now.

I just tried https://w3c-test.org/tools/runner/index.html and typing "/payment-request". Ran for about half an hour. I don't think this helped.

[sideshowbarker]

I have un-borked the immediate problem in the w3c-test.org environment for now, and was able to run https://w3c-test.org/tools/runner/index.html on some tests successfully. So it’s working again for the time being

[scottlow]

@sideshowbarker I'm seeing this behavior again today. Any chance it could get un-borked again? :)

[sideshowbarker]

@scottlow should be working as expected again now

[jdsmith3000]

@sideshowbarker I believe the test harness manifest is borked again. Please "unbork"...

[sideshowbarker]

@jdsmith3000 should be working again now

[jdsmith3000]

@sideshowbarker I'm blocked again. Is there anything that can be done at the client end to get the updated manifest? I really only lasted a few days after you kindly unblocked me 6 days ago...

[sideshowbarker]

@sideshowbarker I'm blocked again.

I seem to have got it working again for the time being

Is there anything that can be done at the client end to get the updated manifest? I really only lasted a few days after you kindly unblocked me 6 days ago...

The http://w3c-test.org/tools/runner/index.html instance is never going to get any more reliable, and there’s nothing you can do from the user side to get it working again after it’s wedged

So your alternatives are to either run the command ./wpt serve from your web-platform-tests working directory, then browse to http://web-platform.test:8000/tools/runner/index.html and use that instead of the http://w3c-test.org instance, or else just use ./wpt run instead of the web runner.

[jdsmith3000]

@sideshowbarker Thanks for the suggestion. I've run that way before successfully, but can't seem to do so today. I'm getting errors on the manifest update. From the console:

• update_manifest.py Failed to load resource: the server responded with a status of 500 (Internal Server Error)
• runner.js:42 Uncaught Error: Manifest generation failed at Manifest. (runner.js:42)
• :8000/favicon.ico Failed to load resource: the server responded with a status of 404 (Not Found)

This was using .\wpt serve and http://web-platform.test:8000/tools/runner/index.html.

I was guessing no manifest update would be needed since I'm using a fresh clone of the repository.

[bobholt]

This is something we need to either fix or completely remove support for. Marking it as roadmap so we look at it sooner than later.

[foolip]

All, is anyone using this as part of their daily workflow and having trouble because of the slowness?

Right now it actually works for me, without any waiting at all. @jgraham, was this impacted by the manifest-download command perhaps?

[foolip]

On http://w3c-test.org/tools/runner/index.html I still see "Manifest generation failed" in the console. But given that nobody said this was part of their workflow, I'll downgrade the priority of this.