test: reduce visual regression tests flakiness

[web-padawan]

Description

It turns out the openSubMenus helper was not using promises and the test was apparently passing or failing randomly, most likely depending on the network latency (in some cases, the screenshot was taken before last menu was opened).

Fixes #238

Type of change

• Internal change

[sissbruecker]

Code changes look good, needs checking why the visual tests fail

[web-padawan]

It turns out that vaadin-menu-bar tests have failed because of font rendering issues. Example:

This is a known problem that happened in the past with individual repositories.
So there we actually had to fix browser version instead of latest.

It's unclear why the threshold in the diffOptions does not help, as in theory it should prevent this.

[sissbruecker]

It's unclear why the threshold in the diffOptions does not help, as in theory it should prevent this.

Hmm, looking at node_modules/@web/test-runner-visual-regression/src/visualDiffCommand.ts it seems to not respect the diffOptions at all:

  const passed = diffPercentage === 0;

  if (!passed) {
    await saveDiff();
  }

  if (!passed || options.buildCache) {
    await saveFailed();
  }

  return {
    errorMessage: !passed
      ? `Visual diff failed. New screenshot is ${diffPercentage.toFixed(2)} % different.`
      : undefined,
    diffPercentage: -1,
    passed,
  };

[sonarcloud]

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
30.0% Duplication