Unable to take a screenshot of a fully loaded page

[gsnedders]

At the moment, with WebDriver, you can trivially wait until after the load event has fired before taking a screenshot. However, not everything delays the load event (see HTML's notion of "critical subresources" and "delays the load event"). Anything you can detect whether or not it has loaded using the DOM can equally be dealt with through the script APIs.

However, CSS is especially awkward because of w3c/csswg-drafts#1088. At the moment, Edge doesn't block the load event on background-image and list-style-image and these are undetectable through the DOM.

We need some proper solution to this problem in a general sense. As a web developer, to do any sort of visual testing, you want to be able to take a screenshot of a page after everything has loaded (of course, this can get complicated by long polling XHR, WebSockets, etc.).

I think there's at least a couple of options options:

• Convince everyone (WGs, implementers) that the page should be in a stable state after the load event has fired
• Add some custom notion to WebDriver of loadedness (OperaDriver has something like this with opera.idle)

Notably, this is causing problems for wptrunner running reftests, as we need some way to take a screenshot after everything has loaded. (web-platform-tests/wpt#5412)

[shs96c]

I realise that your first solution was probably flippant, and your second the desired outcome, but I feel the first is the correct solution.

The problem is that for anyone to actually implement the second suggestion, the first must be true. Worse, as new standards emerge, someone would have to be responsible for making sure that the load event actually meant loaded. This would either mean endless minor updates to this spec (through the w3c approval process to get to Recommendation) or the relevant specs should include language to indicate how they influence the load event.

From the linked WPT bug, it seems like at least some browser vendors have considered this and are treating all "remote loads" as impacting the load event (which seems sensible to me), and from w3c/csswg-drafts#1088 it sounds like the CSS folks are considering the matter too, if I read @fantasai's comment correctly.

Closing this here, but please feel free to reopen if you feel that more discussion is necessary or more reasoning is lax.

[gsnedders]

@shs96c I'd on the whole agree the first solution is probably the correct solution!

Realistically, I filed this bug primarily because this is something that needs to be doable through WebDriver and currently isn't, and a few of us decided a bug on the WebDriver spec was probably an appropriate thing for it, even though the correct solution lies elsewhere, especially when this has been an open issue in CSS for over five years. (It is, prima-facie, a limitation of WebDriver today, after all.)

[shs96c]

Technically, we're waiting for the underlying page to reach a document readiness state of complete. That's dependent on the other specs correctly interacting with how this value is changed. There are issues in other specs that webdriver is exposing --- this is a Good Thing, especially since we always claimed that one of the audiences for this spec are spec authors :)

[jgraham]

I don't really understand the discussion on this bug.

There's no technical reason that the load event has to be the only relevant thing. In fact I don't think there's any browser where that's entirely true; @gsnedders has claimed that it's true for Firefox, but in implementing the reftest harness for Firefox I have to explicitly flush a layout and then check if there are any MozPaint events pending before taking the screenshots. I think this might work for WebDriver just because the IPC overhead is large enough that we can't observe the pre-paint state, but there's certainly no spec I'm aware of that claims that the load event also means that a layout and paint occured, even before we consider whether web fonts must be loaded after load fires &c.

It seems entirely reasonable for WebDriver spec to either state that taking screenshots in particular must wait until a paint has happened and webfonts have loaded and whatever else, or to have another load strategy that waits for these things.

[gsnedders]

So I think there's a few things here, thinking about this some more and talking with @jgraham:

• There needs to be some way to take a screenshot after the load event has fired.

  Notably, the current approach of:

  response = driver.execute_async_script("""
    const done = arguments[0];
    document.onload = function(){
      done();
    });
  """)
  driver.save_screenshot("foo.png")
  

  doesn't suffice because I don't think it's currently defined that the screenshot command will only be executed after the load event has completed firing (i.e., as opposed to merely after the listener that called done was invoked). We probably therefore want to tie screenshots (maybe all WebDriver commands?) to the browsing context event loop.

• As @jgraham pointed out in the previous comment, we need some way to guarantee a style/layout/paint has occurred before the screenshot is taken. For the load case, we can't just guarantee that scripts have completed executing, we need to guarantee everything that follows that (style/etc.) has completed. It's probably reasonable to say that we want to guarantee that style/etc. has happened before all screenshots, given wanting to take a screenshot after some DOM event is probably the common use of screenshots.

[gsnedders]

Oh, and:

• In the hypothetical future, we probably want it to be possible to take screenshots before page load. e.g., one can imagine as part of performance work wanting to take a screenshot of first paint and another after page load to guarantee above the fold content is painted identically initially

[andreastt]

Waiting only for the document load event does indeed not guarantee that the document has completed reflowing (due to layout or web fonts) or finished painting. I think it’s achievable to put in some prose that loosely describes that layout, font loading, and painting, must occur before taking a screenshot. Most modern browsers also have a concept of ‘wait until the main thread is idle’ which we use in Marionette at the moment to provide synchronisation on top of IPC. Whilst there isn’t a spec to exactly describe this, all engines I know of have equivalents of these concepts. Also sprach Geoffrey Sneddon:

In the hypothetical future, we probably want it to be possible to take screenshots *before* page load. e.g., one can imagine as part of performance work wanting to take a screenshot of first paint and another after page load to guarantee above the fold content is painted identically initially

So WebDriver does have page loading strategies, which means the call to navigation will return when the document’s readyState reaches an earlier state than "complete". However, there is still an IPC overhead due to the lack of command pipelining, that may delay a “screenshot of first paint” sufficiently that it’s not actually a screenshot taken exactly on first paint.

[shs96c]

Perhaps we should modify the behaviour of the normal page loading strategy to indicate that not only should we wait for the load event, but also at least one paint, and the "main thread" is idle. Not sure waiting for web fonts to load is a Good Idea --- I've never had a great experience with those loading, and it'll ruin test performance.

Attempting time-sensitive operations in an environment without real-time guarantees seems inherently racey and prone to failure. At the moment, WebDriver implementations can effectively be observers of page loading. With the change proposed in #893 (comment) we'd need to be able to pause loading long enough to take that screenshot.

[jgraham]

I agree that the "screenshot on first paint" case doesn't match the current WebDriver design.

Fonts are arguably supposed to block the load event anyway, although I think browser behaviour differs here. In any case it should certainly be possible to wait for them to load since it's needed for some use cases. This is incidentially why I didn't suggest modifying the existing load strategies.

[andreastt]

Also sprach Simon Stewart:

Perhaps we should modify the behaviour of the `normal` page loading strategy to indicate that not only should we wait for the load event, but also at least one paint, and the "main thread" is idle.

I’d be in favour of this. Marionette already waits for the main thread to become idle under a number of circumstances, most notably waiting for the window to resize or reposition itself. Waiting requestAnimationFrame after the document readyState has turned complete I think makes a lot of sense, and I don’t think this would impact performance all that much. Waiting for the main thread to become idle has a higher performance impact, but might under some circumstances make sense. I would, however, like a better idea of under what circumstances that is the right thing to do.

[whimboo]

Does this only affect screenshots or also other webdriver commands?

I'm asking because there can also be other ongoing requests eg. triggered via AJAX which cause some kind of page load activity. If we only care about page load status here, we would miss those changes and content which gets dynamically added would not be present in the screenshot.

So if this problem is screenshot only maybe the command itself should ensure that the main thread is idle and such? That would guarantee that screenshots would always be accurate (as best as possible).

[shs96c]

@whimboo, if there's AJAX calls going on, then the test author should make use of something like Selenium's Wait helpers to ensure that those calls have finished.

[whimboo]

What about other dynamic changes which do not involve AJAX but only JS, and which results in some items added to the main thread? Do we want to force users to always use Wait in those cases?

[shs96c]

Yes. If the user wants to have a screenshot after a series of JS operations, they should wait for those to complete. The mechanism to get stable screenshots depends on the site or app being visited — there's no reliable way for the driver to determine this.

…

Sent from my iPhone

On 3 Aug 2017, at 14:57, Henrik Skupin ***@***.***> wrote: What about other dynamic changes which do not involve AJAX but only JS, and which results in some items added to the main thread? Do we want to force users to always use Wait in those cases? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[shs96c]

This is waiting on the "conservative" page loading strategy, so punted to Level 2.

[SetTrend]

IMHO, this is something that cannot be computed, because there is no "moment of final load". Think of rotating ads or carussels. With these, the page will never be finally loaded. See this example:

[jgraham]

Sure, it's widely understood that pages may never reach a static render state.

The point of this issue is that these exist some subresources that are loaded as part of the normal document loading pipeline, but don't block the load event. These don't neccessarily expose an API that allows one to determine when they have been loaded. It is perfectly reasonable to want to take a screenshot after fonts and other CSS subresources have loaded, without wanting to solve the problem that pages may never reach a stable state.

[SetTrend]

Understood. Pardon me for interfering.