Docker BackstopJS sometimes produces white or cut-off screenshots

[jonathanloske]

Update: This now occurred with both 4.4.2 Docker image and 4.5.0 Docker image as indicated in my comment below

Our builds currently break with the newest Docker BackstopJS image, 4.5.0. It breaks with both npm version 4.4.2 and 4.5.0.

Out of our ~1100 tests, 395 fail. Judging from how the tests fail, it seems like BackstopJS doesn't wait long enough until the content is rendered. Most screenshots are completely white and a few show content which is not completely rendered. Most of the tests that are failing are simple HTML/CSS pages. They load some JavaScript but it does not change the appearance. We also have parts of our pages built with React. With the React components, the "white screenshot" issue happens with a much smaller percentage but it is still there.

The original of the screenshot I'm showing here can be seen here: https://www.static-immobilienscout24.de/fro/core/4.2.3/documentation/html/campaign-box.html#usage

If need be, I could also provide a zip file with the HTML pages and our backstop.json.

[garris]

That doesn't sound good. This happened when you bumped your version?

[garris]

Does the smoke test run for you? https://github.com/garris/BackstopJS#smoke--feature-tests

[jonathanloske]

We didn't bump it but we had image: backstopjs/backstopjs (without a version number) in our docker-compose.yml.

[jonathanloske]

I wouldn't know how to run the smoke test in a local installation with a custom docker-compose.yml. I tried pointing the script to the correct config file (node_modules/backstopjs/test/configs/backstop) but then lots of files are missing.

I also just cloned the BackstopJS repo and ran the smoke tests from there. There, it looked fine.

[jonathanloske]

Also of note: We're using a custom docker-compose.yml with the following content:

version: "2.1"
services:
    testrunner:
        image: backstopjs/backstopjs:4.5.0
        volumes_from:
            - container:${HOSTNAME}:rw
        working_dir: ${WORKSPACE}
        shm_size: 2048m
        command: "test  --config ./src/test/regression/.generated.backstop.json"

We're running it with the following command: docker-compose -f docker-compose.yml run --rm testrunner

[engelfrost]

I have a similar issue, I'm also getting some incomplete screenshots. Mostly on larger screenshots, not sure if it's a pattern or a coincidence:

I run the smoke tests (cs node_modules/backstopjs && npm run smoke-test-docker) and get 4 errors, I'm not sure if they're of any help or not:

filename: puppet_backstop_features_expanded_0_getItBlock_0_phone.png

filename: puppet_backstop_features_expanded_1_getItBlock__n1_0_phone.png

filename: puppet_backstop_features_hover_0_moneyshot_0_phone.png

filename: puppet_backstop_features_hover_0_moneyshot_1_tablet.png

[jonathanloske]

I had this error happen to me recently locally with an older BackstopJS docker image as well, so it might not be related to the image update. I was able to fix the error by restarting Docker. Could it be that the cleanup in Docker after the testing is somewhat lacking?

[rengare]

This happens also for me

Config:

{
  "id": "FS",
  "viewports": [
    {
      "width": 1920,
      "height": 1080
    }
  ],
  "scenarios": [    {
    "urlPrefix": "http://localhost:3030",
    "delay": 100,
    "postInteractionWait": 0,
    "selectorExpansion": true,
    "expect": 0,
    "misMatchThreshold": 0.1,
    "requireSameDimensions": true,
    "label": "list",
    "urlSuffix": "/list",
    "selectors": [
        ".backstop"
    ],
    "url": "http://localhost:3030/list"
},],
  "paths": {
    "bitmaps_reference": "test/visual-regression-tests/reference_images",
    "bitmaps_test": "test/visual-regression-tests/backstop_data/bitmaps_test",
    "html_report": "test/visual-regression-tests/backstop_data/html_report",
    "ci_report": "test/visual-regression-tests/backstop_data/ci_report"
  },
  "report": ["CI", "browser"],
  "engine": "chromy",
  "engineOptions": {
    "waitTimeout": 320000,
    "chromeFlags": [
      "--disable-dev-shm-usage"
    ]
  },
  "asyncCompareLimit": 1,
  "asyncCaptureLimit": 1,
  "debug": false,
  "debugWindow": false
}

I have run tests with and without docker with same results. I think that maybe it is related to the newest chrome

[kaipahl]

Strange. I got a similar problem but with slightly different circumstances:

• Using BackstopJS only local, not via docker (I'm on a Mac)
• Screenshots works till BackstopJS 4.4.2
• Starting with BackstopJS 4.5.0 (Puppeteer 2.0.0) screenshots are white or cut off like in the example from march 2. The problems stays with 4.5.0, 4.5.1, 5.0.0 and 5.0.1.

It concerns a class selector with 14 appearances on the test page (enables via selectorExpansion flag)

I installed Puppeteer 3.0.1 --> still broken screenshots with BackstopJS 5.0.1
I then installed Puppeteer 1.20.0 and it worked with BackstopJS 5.0.1

It somewhat smells like Puppeteer 2.0.0 introduced some problems for BackstopJS.

[garris]

@kaipahl This is great troubleshooting data -- thank you very much for these clear observations.

Is this intermittent? Or consistent? Does it happen only with selectorExpansion -- or it doesn't matter? Does it happen with large screen captures or does size not seem to matter?

[garris]

@kaipahl and everyone else -- can you try adding this flag to your config?

Please see the help doc here... https://github.com/garris/BackstopJS/blob/f360992b3d4ef4b7dac7928ae4f6fe1cbd5ec629/README.md#alternate-way-of-taking-a-full-page-screenshot

Would very much like to see if this resolves the issue for anyone!

[jonathanloske]

I left the project this was happening in last week and don't have access to the code anymore, so I won't be able to test, unfortunately :(

[pkra]

I encounter what looks like the same issue reliably on a project, usually on extremely large pages. I always assumed the problem is puppeteer/puppeteer#477 and/or https://bugs.chromium.org/p/chromium/issues/detail?id=760596.

can you try adding this flag to your config?

I didn't try this but IIUC that setting only helps with full-page screenshots which would be insufficient for my use case.

[kaipahl]

So I created a test for the live homepage of a client and ran the test through different backstop versions.

The whole page is quite long: between 7171px (mobile viewport) and 5889px (large desktop). The largest 24bit-PNG screenshot is 7,5MB big (2560 × 5889px).

The tests ran through five viewports: mobile (375), tablet (768) and three desktop sizes (1024, 1440 and 2560)

By default I left the puppeteerOffscreenCaptureFix switch to true

I included four scenarios:

• every teaser by class (.o-teaser-default) (with selectorExpansion set to true) = 14 teaser
• eight wrapper and teaser by IDs
• the whole document via document selector
• the whole document via the outer wrapper class (.page)

So each run produced 120 screenshots

I tested BackstopJS 4.3.2, 4.4.2, 4.5.1 and 5.0.1

All results, with only two exceptions, were reproducible.

In nearly all cases puppeteerOffscreenCaptureFix: false produced much worse results than set to true.

mergeImgHack:true created mostly more failed screenshots than improving the results. But the focus of my test runs was more on the screenshotting of selectors instead of whole documents. So it didn't bothered me too much.

As expected, the four BackstopJS versions split into two groups with nearly identical results inside each group: 4.3.2 and 4.4.2 (Puppeteer 1.20/Chrome 78) on the one side and 4.5.1 and 5.0.1 (Puppeteer 2.0/Chrome 80) on the other side with the former group producing in my case much better results.

4.3.2/4.4.2 got only problems with three of the 120 screenshots
4.5.1/5.0.1 failed in 115 of the 120 screenshots!

What were the problems of 4.3.2/4.4.2?

Three of five screenshots for the whole document via the outer wrapper class failed. Like in the campaign box screenshot example that opened the issue, all screenshots hat the correct height, but the "rendering" is cut-off in-midst of the screenshot.

• Mobile: after 4320px of 7171px overall height
• Desktop-1440: after 5080px of 5889px
• Desktop-2560: after 5366px of 5889px
  These cut-offs were, as all other cut-offs that happened during the runs, reproducible!

What's strange: you could be thinking, that it's some sort of memory problem of Puppeteer/Chrome. But why then, is the rendering part of the larger 2560 viewport bigger that of mobile or Desktop-1440?

• Setting puppeteerOffscreenCaptureFix to false had 74–93 of 120 screenshots failed (number differed from run to run)

What were the problems of 4.5.1/5.0.1?

The numbers were pretty bad (115/120 failed). The only screenshots that worked, were the screenshots made via the document selector.

• 13 screenhots were of the cut-off variant -- again: reproducible, always at the same height, without any obvious reason (overlapping stuff or so)
• 5 of these 13 were again the outer wrapper screenshot on .page
• The other 8 cut off screenshots were simple teaser (4x via class selector, 4x via ID selector) with simple, dark background color, white text, webfont (on the server) and icon font (on the server)

• All other 102 screenshots were simply blanks. Correct heights, but completely white.

Funny enough: setting puppeteerOffscreenCaptureFix to false improved the result slightly. Only 109 instead of 115 failures. While the number stayed the same from run to run, the specific teasers that were screenshot correctly differed! It were always teaser with ID selectors, but always different teasers across all viewports. So I ignored that freak result.

Reducing asyncCaptureLimit or asyncCompareLimit didn't improve anything.

To be honest: i fail to see any pattern here.

Addentum: running BackstopJS on the Mac via NodeJS/NPM script, no Docker involved.

[garris]

@kaipahl thanks for this. What is your node version? And what is the hardware profile? The thing that is not adding up for me is that no one is complaining about this at work -- however page lengths are not very long there and we only test one viewport.

[kasparsz]

I had the same issue, I was taking screenhots of multiple individual elements on a page, I didn't have issues with whole document screenshots.

This is the solution which helped, but only after updating to the latest puppeteer (3.1.0)
Edit: puppeteer 3.1.0 is not needed, error in testing, reverted to 2.0.0 and it works
https://petertran.com.au/2018/07/12/blank-images-puppeteer-screenshots-solved/

Made a change and pull request
kasparsz@60361ce
sanity-test and smoke-test passes apart from text rendering differences between my OS and references OS

OS: Windows 10
Node: v10.20.1

[kasparsz]

5.0.1 version, this issue happens with docker and non-docker versions and puppeteerOffscreenCaptureFix doesn't solve it.

Running smoke-test this issue happens in 1, 2 or 3 tests, just like on my website it's inconsistent between runs. Issue exists in docker and non-docker smoke-tests.

On my website:

• Changing puppeteer to 3.1.0 didn't helped
• Same happens with just plain HTML without any CSS or JS
• I disabled all backstopjs from injecting or running any scripts on the page, still the same issue
• Checked el.boundingBox() and it returns correct boundingBox offset and size for the elements, but screenshot is still clipped or empty

[ghost]

We are having the same issue also with cut off and white screenshots using individual selectors

Issue occurred from 4.5.0 onwards to 5.0.1
Last working version for us was 4.4.2

Running on Docker and Non-Docker Versions on Mac OS

Node Version 10.15.0

[r-oldenburg]

Hi there. Same here. Updating to higher than 4.4.x makes a big part of our tests fail due to cropped images.
Even with 5.0.6 and "puppeteerOffscreenCaptureFix": true/false and "useBoundingBoxViewportForSelectors": true/false there is no difference.

[zahorulko]

@kaipahl and everyone else -- can you try adding this flag to your config?

Please see the help doc here... https://github.com/garris/BackstopJS/blob/f360992b3d4ef4b7dac7928ae4f6fe1cbd5ec629/README.md#alternate-way-of-taking-a-full-page-screenshot

Would very much like to see if this resolves the issue for anyone!

Hi, it didn't help me, i also have the same problem

[npazo]

Is there a surefire workaround for this? Is it to go back to a specific version of BackstopJS? Docker or no-Docker? I have tried rolling back to Docker 4.4.0 version and am still seeing certain images cut off.

[engelfrost]

I'm not sure, but for me it seems to be related to webfonts and how they are loaded.

[digitaldonkey]

Also running into this using 5.3.2.
We have server loaded webfonts. But currently I can't investigate deeper into it.

[r-oldenburg]

Somehow this seems to be gone with 5.3.4... Can anybody else confirm?

[pkra]

https://bugs.chromium.org/p/chromium/issues/detail?id=760596 is not fixed so the problem likely persists. However, I also don't have the problem anymore after upgrading to a new machine recently (which would fit with the idea that this problem is due to a race condition in chromium).

[r-oldenburg]

Interestingly it is said there that it may be "something that might already be addressed"... Maybe we are lucky..?!

[r-oldenburg]

Nope, that wasn't a help. Errors came back.

But I found out that in fact we had two issues. And I found the solution! See #1344