Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate E2E tests to Playwright #38851

Closed
JustinyAhin opened this issue Feb 16, 2022 · 18 comments
Closed

Migrate E2E tests to Playwright #38851

JustinyAhin opened this issue Feb 16, 2022 · 18 comments
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Overview Comprehensive, high level view of an area of focus often with multiple tracking issues [Type] Tracking Issue Tactical breakdown of efforts across the codebase and/or tied to Overview issues.

Comments

@JustinyAhin
Copy link
Member

JustinyAhin commented Feb 16, 2022 •
edited by Mamaduka
Loading

This issue tracks progress on migrating Gutenberg's E2E tests from Puppeteer (packages/e2e-tests) to Playwright (test/e2e).

Resources:

Tests to migrate

Flaky tests

These tests are ranked by their estimated flaky rates. Experience developers should prioritize migrating them in this order.

Other tests

These tests don't have corresponding flaky reports. Newcomers can start migrating these since they are more stable and probably easier to start with.

Performance tests
@JustinyAhin JustinyAhin added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Feb 16, 2022
@JustinyAhin
Copy link
Member Author

cc @kevin940726

@Mamaduka Mamaduka added the [Type] Tracking Issue Tactical breakdown of efforts across the codebase and/or tied to Overview issues. label Feb 17, 2022
@JustinyAhin JustinyAhin mentioned this issue Feb 21, 2022
Closed
@juhi123 juhi123 mentioned this issue Mar 7, 2022
Closed
This was referenced Mar 14, 2022
Merged
Closed
This was referenced Mar 18, 2022
Merged
Merged
@noisysocks noisysocks changed the title Overview of Playwright tests to be written Migrate E2E tests to Playwright Mar 22, 2022
@WordPress WordPress deleted a comment from JustinyAhin Mar 22, 2022
@WordPress WordPress deleted a comment from JustinyAhin Mar 22, 2022
@noisysocks noisysocks added the [Type] Overview Comprehensive, high level view of an area of focus often with multiple tracking issues label Mar 22, 2022
@noisysocks
Copy link
Member

I reformatted this issue a little bit, added some links, and un-checked all the checkboxes since, to my knowledge, we haven't merged anything that migrates any tests yet. (#38570 is just the infrastructure.)

@noisysocks
Copy link
Member

It would be nice to flag 🚩 which tests are flaky so that we can prioritise them. That will show us if Playwright offers any improvements there.

@JustinyAhin
Copy link
Member Author

I removed the e2e-utils checkboxes from the issue description, since they will be added if necessary when we are migrating tests.

@juhi123
Copy link
Contributor

juhi123 commented Mar 23, 2022

@JustinyAhin Can you please update the README.md and MIGRATION.md links in the description

https://github.com/WordPress/gutenberg/blob/trunk/test/e2e/MIGRATION.md
https://github.com/WordPress/gutenberg/blob/trunk/test/e2e/README.md

@kevin940726
Copy link
Member

@juhi123 Updated! Thanks!

@juhi123 juhi123 mentioned this issue Mar 24, 2022
Closed
This was referenced Mar 25, 2022
Closed
Closed
Closed
@kevin940726 kevin940726 mentioned this issue Mar 28, 2022
Merged
@ockham ockham mentioned this issue Mar 28, 2022
Merged
3 tasks
@shanjidah shanjidah mentioned this issue Mar 30, 2022
Merged
@carolinan carolinan mentioned this issue Apr 1, 2022
Open
@kevin940726 kevin940726 mentioned this issue Apr 8, 2022
Merged
@talldan talldan mentioned this issue Apr 12, 2022
Merged
@DevinWalker DevinWalker mentioned this issue Apr 18, 2022
Open
@talldan
Copy link
Contributor

talldan commented Apr 19, 2022 •
edited
Loading

I started looking at the second most flakey test (site-editor/template-part.test.js).

I hit a roadblock that I think is worth trying to solve early on while there's a chance to put the right structure in place.

The site editor is using an iframe for the editor canvas. Because of this, none of the existing test utils for interacting with the editor canvas can be used for site editor tests. That's because they all use page.locator and the like, while for an iframe frameLocator needs to be used to find the iframe first, and then frame.locator can be used to select elements inside the iframe.

It's clear this was also a problem in Puppeteer, most of the puppeteer site editor tests have duplicate utility functions in the test file.

I'm not all that familiar with playwright, but I wonder if it could be an option to move the canvas utils into a class and do something like the following:

test.use( {
	editorCanvasUtils: async (
		{ page, pageUtils },
		use
	) => {
		await use(
			new EditorCanvasUtils( { page, pageUtils, isSiteEditor: true } )
		);
	},
} );

EditorCanvasUtils could have a private canvas property that uses either the iframe or the page depending on the isSiteEditor param. All utilities defined in the class then use canvas.locator.

This was referenced Dec 27, 2023
Merged
Merged
Merged
Merged
@pooja-muchandikar pooja-muchandikar mentioned this issue Dec 29, 2023
Closed
This was referenced Dec 29, 2023
Merged
Merged
@kevin940726 kevin940726 mentioned this issue Jan 2, 2024
Merged
This was referenced Jan 3, 2024
Merged
Merged
Merged
Merged
Merged
Merged
@fai-sal fai-sal mentioned this issue Jan 5, 2024
Merged
This was referenced Jan 9, 2024
Merged
Merged
Merged
Merged
This was referenced Jan 18, 2024
Merged
Merged
Merged
@Mamaduka Mamaduka mentioned this issue Jan 31, 2024
Merged
@kevin940726 kevin940726 mentioned this issue Feb 7, 2024
Merged
@Mamaduka
Copy link
Member

Every end-to-end test has been migrated to the Playwright. Thank you everyone for contributing!

@kevin940726
Copy link
Member

Thank you all! ❤️ 💯

There is still some work left to do to clean up. We'll do them in follow-up PRs!

@Mamaduka Mamaduka mentioned this issue Feb 23, 2024
Merged
@Mamaduka Mamaduka mentioned this issue Apr 2, 2024
Open
@carstingaxion carstingaxion mentioned this issue Aug 5, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Overview Comprehensive, high level view of an area of focus often with multiple tracking issues [Type] Tracking Issue Tactical breakdown of efforts across the codebase and/or tied to Overview issues.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@Mamaduka @noisysocks @talldan @ndiego @kevin940726 @juhi123 @JustinyAhin