Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INFRA] Some e2e tests fail on Chrome and Edge browsers when run with GitHub Actions #1933

Closed
tbouffard opened this issue Apr 29, 2022 · 4 comments · Fixed by #1977
Closed

[INFRA] Some e2e tests fail on Chrome and Edge browsers when run with GitHub Actions #1933

tbouffard opened this issue Apr 29, 2022 · 4 comments · Fixed by #1977
Assignees
@tbouffard
Labels
chore Build, CI/CD or repository tasks (issues/PR maintenance, environments, ...)
Milestone
0.24.0

Comments

@tbouffard
Copy link
Member

tbouffard commented Apr 29, 2022 •
edited
Loading

Starting from 2022-04-27T14:54:28Z (detected in #1929), some visual tests on Chrome involving labels are not passing anymore. All OS are affected.
Starting from 2022-04-29T13:21:03Z (detected in #1934), the same occurs with the Edge browser!

The failure may be related to a GitHub Actions environment change.

Environment information for the Chrome issue

Information taken from #1929 runs.

macos-11 OK - run logs
On 2022-04-26T16:51:57Z

Current runner version: '2.290.1'
Operating System
  macOS
  11.6.5
  20G527
Virtual Environment
  Environment: macos-11
  Version: 20220419.3
  Included Software: https://github.com/actions/virtual-environments/blob/macOS-11/20220419.3/images/macos/macos-11-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/macOS-11%2F20220419.3
Virtual Environment Provisioner
  1.0.0.0-main-20220406-1

Browsers listed in the image Readme

Google Chrome 100.0.4896.127
Microsoft Edge 100.0.1185.44
MSEdgeDriver 100.0.1185.44

macos-11 KO - run logs
On 2022-04-27T14:54:28Z

Current runner version: '2.290.1'
Operating System
  macOS
  11.6.5
  20G527
Virtual Environment
  Environment: macos-11
  Version: 20220424.1
  Included Software: https://github.com/actions/virtual-environments/blob/macOS-11/20220424.1/images/macos/macos-11-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/macOS-11%2F20220424.1
Virtual Environment Provisioner
  1.0.0.0-main-20220421-1

Browsers listed in the image Readme

No Chrome change
Microsoft Edge 100.0.1185.50
MSEdgeDriver 100.0.1185.50

The virtual environment image changed from 20220419.3 to 20220424.1.
The provisioner changed, from 1.0.0.0-main-20220406-1 to 1.0.0.0-main-20220421-1

ubuntu-20.04 OK - run logs
On 2022-04-26T16:51:57

Current runner version: '2.290.1'
Operating System
  Ubuntu
  20.04.4
  LTS
Virtual Environment
  Environment: ubuntu-20.04
  Version: 20220410.2
  Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220410.2/images/linux/Ubuntu2004-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220410.2
Virtual Environment Provisioner
  1.0.0.0-main-20220406-1

Browsers listed in the image Readme

Google Chrome 100.0.4896.75
ChromeDriver 100.0.4896.60

ubuntu-20.04 KO - run logs
On 2022-04-27T14:54:56Z

Current runner version: '2.290.1'
Operating System
  Ubuntu
  20.04.4
  LTS
Virtual Environment
  Environment: ubuntu-20.04
  Version: 20220410.2
  Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220410.2/images/linux/Ubuntu2004-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220410.2
Virtual Environment Provisioner
  1.0.0.0-main-20220421-1

This is the exact same virtual environment image. Only the provisioner changed, from 1.0.0.0-main-20220406-1 to 1.0.0.0-main-20220421-1

windows-2019 OK - run logs
On 2022-04-26T16:52Z

Current runner version: '2.290.1'
Operating System
  Microsoft Windows Server 2019
  10.0.17763
  Datacenter
Virtual Environment
  Environment: windows-2019
  Version: 20220410.1
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20220410.1/images/win/Windows2019-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win19%2F20220410.1
Virtual Environment Provisioner
  1.0.0.0-main-20220406-1

Browsers listed in the image Readme

Google Chrome 100.0.4896.75
Chrome Driver 100.0.4896.60

windows-2019 KO - run logs
On `2022-04-27T15:03:57Z

Current runner version: '2.290.1'
Operating System
  Microsoft Windows Server 2019
  10.0.17763
  Datacenter
Virtual Environment
  Environment: windows-2019
  Version: 20220410.1
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20220410.1/images/win/Windows2019-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win19%2F20220410.1
Virtual Environment Provisioner
  1.0.0.0-main-20220421-1

This is the exact same virtual environment image. Only the provisioner changed, from 1.0.0.0-main-20220406-1 to 1.0.0.0-main-20220421-1

Environment information for the Edge issue

Working, in #1929, on 2022-04-27T14:55:21Z

Current runner version: '2.290.1'
Operating System
  Microsoft Windows Server 2019
  10.0.17763
  Datacenter
Virtual Environment
  Environment: windows-2019
  Version: 20220410.1
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20220410.1/images/win/Windows2019-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win19%2F20220410.1
Virtual Environment Provisioner
  1.0.0.0-main-20220421-1

Browsers listed in the image Readme

Microsoft Edge 100.0.1185.36
Microsoft Edge Driver 100.0.1185.36

not working, in #1934, on 2022-04-29T13:34:54Z

Current runner version: '2.290.1'
Operating System
  Microsoft Windows Server 2019
  10.0.17763
  Datacenter
Virtual Environment
  Environment: windows-2019
  Version: 20220410.1
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20220410.1/images/win/Windows2019-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win19%2F20220410.1
Virtual Environment Provisioner
  1.0.0.0-main-20220421-1

This is the exact same virtual environment image and the exact same environment provisioner.
@tbouffard tbouffard added the chore Build, CI/CD or repository tasks (issues/PR maintenance, environments, ...) label Apr 29, 2022
@tbouffard tbouffard added this to the 0.24.0 milestone Apr 29, 2022
@tbouffard tbouffard self-assigned this Apr 29, 2022
@tbouffard tbouffard mentioned this issue Apr 29, 2022
Merged
@tbouffard tbouffard changed the title [INFRA] some e2e tests fail on Chrome when run with GitHub Actions [INFRA] Some e2e tests fail on Chrome when run with GitHub Actions Apr 29, 2022
@tbouffard tbouffard changed the title [INFRA] Some e2e tests fail on Chrome when run with GitHub Actions [INFRA] Some e2e tests fail on Chrome and Edge browsers when run with GitHub Actions Apr 29, 2022
This was referenced Apr 29, 2022
Closed
Closed
@tbouffard
Copy link
Member Author

tbouffard commented Apr 29, 2022 •
edited
Loading

ℹ️ I have just reproduced the issue locally with Chrome 101.0.4951.41 (Official build) 64 bits on Ubuntu 20.04.4 LTS by running a test with the labels.01.general diagram. The other failing tests involve the fit feature when using diagram with labels.

local error:

Expected image to match or be a close match to snapshot but was 1.7988124362196545% different from snapshot (8634.299693854342 differing pixels). Failure threshold was set to 0.0005%.

GitHub Actions error (according to the image release notes, with Chrome 100):

Expected image to match or be a close match to snapshot but was 1.8007200003331452% different from snapshot (8643.456001599096 differing pixels). Failure threshold was set to 0.0005%.

@tbouffard tbouffard mentioned this issue May 2, 2022
Merged
@tbouffard
Copy link
Member Author

tbouffard commented May 2, 2022 •
edited
Loading

Thanks to process-analytics/github-actions-playground#78, I have been able to get the actual versions of chrome and edge

macOS: not implemented for now
Ubuntu: https://github.com/process-analytics/github-actions-playground/runs/6253668243?check_suite_focus=true#step:2:7
Windows: https://github.com/process-analytics/github-actions-playground/runs/6253668364?check_suite_focus=true#step:3:1

Here are information on 2022-05-02. Still apply on 2022-05-05
Image versions are the ones listed in the issue description.

runner browser expected version current version comments
ubuntu 20.04 google-chrome-stable 100.0.4896.75 100.0.4896.127-1 ⚠️ change
windows 2019 Google Chrome 100.0.4896.75 100.0.4896.75 🤔 no change, should work 🤔
windows 2019 Microsoft Edge 100.0.1185.36 101.0.1210.32 ⚠️ change

@tbouffard
Copy link
Member Author

ℹ️ New environment detected today

ubuntu
https://github.com/process-analytics/github-actions-playground/runs/6316731583?check_suite_focus=true#step:1:6

Current runner version: '2.291.1'
Operating System
  Ubuntu
  20.04.4
  LTS
Virtual Environment
  Environment: ubuntu-20.04
  Version: 20220503.1
  Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220503.1/images/linux/Ubuntu2004-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220503.1
Virtual Environment Provisioner
  1.0.0.0-main-20220421-1

google-chrome-stable Version: 101.0.4951.54-1
--> consistent with the image release notes

windows
https://github.com/process-analytics/github-actions-playground/runs/6316731656?check_suite_focus=true#step:1:7

Current runner version: '2.291.1'
Operating System
  Microsoft Windows Server 2019
  10.0.17763
  Datacenter
Virtual Environment
  Environment: windows-2019
  Version: 20220503.1
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20220503.1/images/win/Windows2019-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win19%2F20220503.1
Virtual Environment Provisioner
  1.0.0.0-main-20220421-1

Google Chrome 101.0.4951.54 Google LLC 20220504
Microsoft Edge 101.0.1210.32 Microsoft Corporation 20220504
--> consistent with the image release notes

@tbouffard tbouffard mentioned this issue May 17, 2022
Merged
@tbouffard tbouffard mentioned this issue May 18, 2022
Merged
@tbouffard
Copy link
Member Author

tbouffard commented Nov 16, 2022 •
edited
Loading

At that time, we weren't using the browsers installed in the GitHub runners. Chrome and Edge was installed prior running tests.
So the investigation described is wrong because it only focused on what is installed in the GH runners.
At that time, I haven't realized that.

This should change with #2355

@tbouffard tbouffard mentioned this issue Nov 16, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tbouffard tbouffard

Labels
chore Build, CI/CD or repository tasks (issues/PR maintenance, environments, ...)
Projects
None yet
Milestone
0.24.0
Development

Successfully merging a pull request may close this issue.

[INFRA] Restore Chrome and Edge e2e tests on CI process-analytics/bpmn-visualization-js
1 participant
@tbouffard