Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate performance tests to Playwright #51084

Merged
merged 53 commits into from
Jun 27, 2023

Conversation

WunderBart
Copy link
Member

@WunderBart WunderBart commented May 30, 2023

Note
We're migrating but NOT switching to Playwright yet - it will happen in a dedicated PR.

What?

This PR migrates existing performance tests from Puppeteer to Playwright. The aim is to stay as close to the original tests as possible to prevent skewing the numbers but, at the same time, improve the overall stability, which Playwright has proven to be good at. Any inconsistencies or improvements (e.g., #51379 #51383 #51376) should be addressed in a follow-up PR.

Metrics

The following compares before and after-migration performance metrics (average values, as calculated in the reporter script). Both jobs were run locally so that the milliseconds might differ from the ones in CI. The timings are not so important, though - the % change is.

edit: See the long-running comparison in #51084 (comment)

Classic Theme Spec

Metric Puppeteer Playwright % Change
Server Response (ms) 250.00 235.50 -5.80%
First Paint (ms) 720.00 689.00 -4.31%
DOM Content Loaded (ms) 750.00 715.50 -4.60%
Loaded (ms) 1250.00 1200.00 -4.00%
First Contentful Paint (ms) 790.00 762.00 -3.54%
First Block (ms) 950.00 925.00 -2.63%
Type (ms) 45.00 42.50 -5.56%

Block Theme Spec

Metric Puppeteer Playwright % Change
Server Response (ms) 243.13 239.16 -1.63%
First Paint (ms) 746.10 732.54 -1.82%
DOM Content Loaded (ms) 769.25 758.33 -1.42%
Loaded (ms) 1269.38 1256.25 -1.04%
First Contentful Paint (ms) 806.26 794.20 -1.50%
First Block (ms) 974.20 963.10 -1.14%
Type (ms) 44.00 43.20 -1.82%

Post Editor Spec

Metric Puppeteer Playwright % Change
Server Response (ms) 202.33 200.85 -0.73%
First Paint (ms) 157.70 160.38 +1.70%
DOM Content Loaded (ms) 567.33 571.07 +0.66%
Loaded (ms) 569.70 573.44 +0.66%
First Contentful Paint (ms) 669.67 675.44 +0.86%
First Block (ms) 8520.00 8556.67 +0.43%
Type (ms) 36.67 36.86 +0.52%

Site Editor Spec

Metric Puppeteer Playwright % Change
Server Response (ms) 206.03 228.3 +10.81%
First Paint (ms) 177.20 250.13 +41.20%
DOM Content Loaded (ms) 593.10 757.13 +27.64%
Loaded (ms) 595.40 757.5 +27.23%
First Contentful Paint (ms) 696.50 792.37 +13.76%
First Block (ms) 8806.67 6474.03 -26.46%
Type (ms) 38.00 31.06 -18.26%

Reporters

There is a slight difference in reporting, where Playwright lists each test case vs. only the top-level suite.

Warning
Our custom PerformanceReporter doesn't report on the Theme* tests, as no section for theme metrics exists has been created yet.

Puppeteer Playwright
Screenshot 2023-06-14 at 13 22 03 Screenshot 2023-06-14 at 13 12 25

Execution Speed

Testing locally, Playwright's execution is, on average, 20% faster than Puppeteer's (~4 vs. ~5 minutes per job). The following factors most likely cause this:

  • RequestUtils that perform backend requests to, e.g., switch theme instead of clicking through the UI, and
  • Auto-waiting, which automatically releases the element as actionable as soon as possible.

I've also noticed, while testing headfully, that Playwright is generally executing faster than Puppeteer. I'm unsure if we're missing some throttling config, but the metrics seem to stay consistent after the migration, so I wouldn't worry about that much, I guess.

Browsers

The frameworks provided the following browsers (running on M1 Pro, Ventura 13.1):

Browser Puppeteer Playwright
Chromium Version 99.0.4844.0 (Developer Build) (x86_64 translated) Version 112.0.5615.29 (Developer Build) (arm64)

@github-actions
Copy link

github-actions bot commented May 30, 2023

Size Change: +40.4 kB (+3%)

Total Size: 1.44 MB

Filename Size Change
build/api-fetch/index.min.js 2.28 kB -7 B (0%)
build/block-editor/index.min.js 208 kB +13.2 kB (+7%) 🔍
build/block-editor/style-rtl.css 14.7 kB -216 B (-1%)
build/block-editor/style.css 14.7 kB -210 B (-1%)
build/block-library/blocks/image/interactivity.min.js 1.52 kB +178 B (+13%) ⚠️
build/block-library/blocks/image/style-rtl.css 1.34 kB +5 B (0%)
build/block-library/blocks/image/style.css 1.35 kB +5 B (0%)
build/block-library/blocks/navigation/editor-rtl.css 2.26 kB -96 B (-4%)
build/block-library/blocks/navigation/editor.css 2.26 kB -95 B (-4%)
build/block-library/editor-rtl.css 12.1 kB -77 B (-1%)
build/block-library/editor.css 12.1 kB -79 B (-1%)
build/block-library/index.min.js 201 kB -284 B (0%)
build/block-library/style-rtl.css 13.6 kB +105 B (+1%)
build/block-library/style.css 13.6 kB +104 B (+1%)
build/blocks/index.min.js 50.9 kB +73 B (0%)
build/commands/index.min.js 14.9 kB +6 B (0%)
build/components/index.min.js 240 kB +9.61 kB (+4%)
build/components/style-rtl.css 11.8 kB -24 B (0%)
build/components/style.css 11.8 kB -20 B (0%)
build/core-commands/index.min.js 2.26 kB +146 B (+7%) 🔍
build/core-data/index.min.js 16.1 kB +436 B (+3%)
build/data/index.min.js 8.27 kB +19 B (0%)
build/edit-post/index.min.js 34.8 kB +764 B (+2%)
build/edit-post/style-rtl.css 7.58 kB -1 B (0%)
build/edit-post/style.css 7.57 kB -1 B (0%)
build/edit-site/index.min.js 83.9 kB +13.2 kB (+19%) ⚠️
build/edit-site/style-rtl.css 12.5 kB +936 B (+8%) 🔍
build/edit-site/style.css 12.5 kB +939 B (+8%) 🔍
build/edit-widgets/index.min.js 16.8 kB +17 B (0%)
build/editor/index.min.js 45.5 kB +829 B (+2%)
build/editor/style-rtl.css 3.58 kB +49 B (+1%)
build/editor/style.css 3.58 kB +49 B (+1%)
build/format-library/index.min.js 7.62 kB +2 B (0%)
build/private-apis/index.min.js 943 B +4 B (0%)
build/reusable-blocks/index.min.js 2.38 kB +174 B (+8%) 🔍
build/rich-text/index.min.js 10.9 kB +234 B (+2%)
build/style-engine/index.min.js 1.81 kB +397 B (+28%) 🚨
ℹ️ View Unchanged
Filename Size
build/a11y/index.min.js 955 B
build/annotations/index.min.js 2.69 kB
build/autop/index.min.js 2.1 kB
build/blob/index.min.js 451 B
build/block-directory/index.min.js 6.99 kB
build/block-directory/style-rtl.css 1.02 kB
build/block-directory/style.css 1.02 kB
build/block-editor/content-rtl.css 4.22 kB
build/block-editor/content.css 4.22 kB
build/block-editor/default-editor-styles-rtl.css 381 B
build/block-editor/default-editor-styles.css 381 B
build/block-library/blocks/archives/editor-rtl.css 61 B
build/block-library/blocks/archives/editor.css 60 B
build/block-library/blocks/archives/style-rtl.css 90 B
build/block-library/blocks/archives/style.css 90 B
build/block-library/blocks/audio/editor-rtl.css 150 B
build/block-library/blocks/audio/editor.css 150 B
build/block-library/blocks/audio/style-rtl.css 122 B
build/block-library/blocks/audio/style.css 122 B
build/block-library/blocks/audio/theme-rtl.css 126 B
build/block-library/blocks/audio/theme.css 126 B
build/block-library/blocks/avatar/editor-rtl.css 116 B
build/block-library/blocks/avatar/editor.css 116 B
build/block-library/blocks/avatar/style-rtl.css 104 B
build/block-library/blocks/avatar/style.css 104 B
build/block-library/blocks/block/editor-rtl.css 305 B
build/block-library/blocks/block/editor.css 305 B
build/block-library/blocks/button/editor-rtl.css 584 B
build/block-library/blocks/button/editor.css 582 B
build/block-library/blocks/button/style-rtl.css 624 B
build/block-library/blocks/button/style.css 623 B
build/block-library/blocks/buttons/editor-rtl.css 337 B
build/block-library/blocks/buttons/editor.css 337 B
build/block-library/blocks/buttons/style-rtl.css 332 B
build/block-library/blocks/buttons/style.css 332 B
build/block-library/blocks/calendar/style-rtl.css 239 B
build/block-library/blocks/calendar/style.css 239 B
build/block-library/blocks/categories/editor-rtl.css 113 B
build/block-library/blocks/categories/editor.css 112 B
build/block-library/blocks/categories/style-rtl.css 124 B
build/block-library/blocks/categories/style.css 124 B
build/block-library/blocks/code/editor-rtl.css 53 B
build/block-library/blocks/code/editor.css 53 B
build/block-library/blocks/code/style-rtl.css 121 B
build/block-library/blocks/code/style.css 121 B
build/block-library/blocks/code/theme-rtl.css 124 B
build/block-library/blocks/code/theme.css 124 B
build/block-library/blocks/columns/editor-rtl.css 108 B
build/block-library/blocks/columns/editor.css 108 B
build/block-library/blocks/columns/style-rtl.css 409 B
build/block-library/blocks/columns/style.css 409 B
build/block-library/blocks/comment-author-avatar/editor-rtl.css 125 B
build/block-library/blocks/comment-author-avatar/editor.css 125 B
build/block-library/blocks/comment-content/style-rtl.css 92 B
build/block-library/blocks/comment-content/style.css 92 B
build/block-library/blocks/comment-template/style-rtl.css 199 B
build/block-library/blocks/comment-template/style.css 198 B
build/block-library/blocks/comments-pagination-numbers/editor-rtl.css 123 B
build/block-library/blocks/comments-pagination-numbers/editor.css 121 B
build/block-library/blocks/comments-pagination/editor-rtl.css 222 B
build/block-library/blocks/comments-pagination/editor.css 209 B
build/block-library/blocks/comments-pagination/style-rtl.css 235 B
build/block-library/blocks/comments-pagination/style.css 231 B
build/block-library/blocks/comments-title/editor-rtl.css 75 B
build/block-library/blocks/comments-title/editor.css 75 B
build/block-library/blocks/comments/editor-rtl.css 840 B
build/block-library/blocks/comments/editor.css 839 B
build/block-library/blocks/comments/style-rtl.css 637 B
build/block-library/blocks/comments/style.css 636 B
build/block-library/blocks/cover/editor-rtl.css 647 B
build/block-library/blocks/cover/editor.css 650 B
build/block-library/blocks/cover/style-rtl.css 1.61 kB
build/block-library/blocks/cover/style.css 1.6 kB
build/block-library/blocks/details/editor-rtl.css 65 B
build/block-library/blocks/details/editor.css 65 B
build/block-library/blocks/details/style-rtl.css 159 B
build/block-library/blocks/details/style.css 159 B
build/block-library/blocks/embed/editor-rtl.css 293 B
build/block-library/blocks/embed/editor.css 293 B
build/block-library/blocks/embed/style-rtl.css 410 B
build/block-library/blocks/embed/style.css 410 B
build/block-library/blocks/embed/theme-rtl.css 126 B
build/block-library/blocks/embed/theme.css 126 B
build/block-library/blocks/file/editor-rtl.css 316 B
build/block-library/blocks/file/editor.css 316 B
build/block-library/blocks/file/interactivity.min.js 395 B
build/block-library/blocks/file/style-rtl.css 269 B
build/block-library/blocks/file/style.css 270 B
build/block-library/blocks/file/view.min.js 375 B
build/block-library/blocks/footnotes/style-rtl.css 183 B
build/block-library/blocks/footnotes/style.css 182 B
build/block-library/blocks/freeform/editor-rtl.css 2.58 kB
build/block-library/blocks/freeform/editor.css 2.58 kB
build/block-library/blocks/gallery/editor-rtl.css 947 B
build/block-library/blocks/gallery/editor.css 952 B
build/block-library/blocks/gallery/style-rtl.css 1.53 kB
build/block-library/blocks/gallery/style.css 1.53 kB
build/block-library/blocks/gallery/theme-rtl.css 108 B
build/block-library/blocks/gallery/theme.css 108 B
build/block-library/blocks/group/editor-rtl.css 654 B
build/block-library/blocks/group/editor.css 654 B
build/block-library/blocks/group/style-rtl.css 57 B
build/block-library/blocks/group/style.css 57 B
build/block-library/blocks/group/theme-rtl.css 78 B
build/block-library/blocks/group/theme.css 78 B
build/block-library/blocks/heading/style-rtl.css 76 B
build/block-library/blocks/heading/style.css 76 B
build/block-library/blocks/html/editor-rtl.css 336 B
build/block-library/blocks/html/editor.css 337 B
build/block-library/blocks/image/editor-rtl.css 834 B
build/block-library/blocks/image/editor.css 833 B
build/block-library/blocks/image/theme-rtl.css 126 B
build/block-library/blocks/image/theme.css 126 B
build/block-library/blocks/latest-comments/style-rtl.css 357 B
build/block-library/blocks/latest-comments/style.css 357 B
build/block-library/blocks/latest-posts/editor-rtl.css 213 B
build/block-library/blocks/latest-posts/editor.css 212 B
build/block-library/blocks/latest-posts/style-rtl.css 478 B
build/block-library/blocks/latest-posts/style.css 478 B
build/block-library/blocks/list/style-rtl.css 88 B
build/block-library/blocks/list/style.css 88 B
build/block-library/blocks/media-text/editor-rtl.css 266 B
build/block-library/blocks/media-text/editor.css 263 B
build/block-library/blocks/media-text/style-rtl.css 507 B
build/block-library/blocks/media-text/style.css 505 B
build/block-library/blocks/more/editor-rtl.css 431 B
build/block-library/blocks/more/editor.css 431 B
build/block-library/blocks/navigation-link/editor-rtl.css 712 B
build/block-library/blocks/navigation-link/editor.css 711 B
build/block-library/blocks/navigation-link/style-rtl.css 115 B
build/block-library/blocks/navigation-link/style.css 115 B
build/block-library/blocks/navigation-submenu/editor-rtl.css 296 B
build/block-library/blocks/navigation-submenu/editor.css 295 B
build/block-library/blocks/navigation/interactivity.min.js 978 B
build/block-library/blocks/navigation/style-rtl.css 2.21 kB
build/block-library/blocks/navigation/style.css 2.2 kB
build/block-library/blocks/navigation/view-modal.min.js 2.78 kB
build/block-library/blocks/navigation/view.min.js 438 B
build/block-library/blocks/nextpage/editor-rtl.css 395 B
build/block-library/blocks/nextpage/editor.css 395 B
build/block-library/blocks/page-list/editor-rtl.css 401 B
build/block-library/blocks/page-list/editor.css 401 B
build/block-library/blocks/page-list/style-rtl.css 175 B
build/block-library/blocks/page-list/style.css 175 B
build/block-library/blocks/paragraph/editor-rtl.css 174 B
build/block-library/blocks/paragraph/editor.css 174 B
build/block-library/blocks/paragraph/style-rtl.css 279 B
build/block-library/blocks/paragraph/style.css 281 B
build/block-library/blocks/post-author/style-rtl.css 175 B
build/block-library/blocks/post-author/style.css 176 B
build/block-library/blocks/post-comments-form/editor-rtl.css 96 B
build/block-library/blocks/post-comments-form/editor.css 96 B
build/block-library/blocks/post-comments-form/style-rtl.css 508 B
build/block-library/blocks/post-comments-form/style.css 508 B
build/block-library/blocks/post-date/style-rtl.css 61 B
build/block-library/blocks/post-date/style.css 61 B
build/block-library/blocks/post-excerpt/editor-rtl.css 71 B
build/block-library/blocks/post-excerpt/editor.css 71 B
build/block-library/blocks/post-excerpt/style-rtl.css 141 B
build/block-library/blocks/post-excerpt/style.css 141 B
build/block-library/blocks/post-featured-image/editor-rtl.css 588 B
build/block-library/blocks/post-featured-image/editor.css 586 B
build/block-library/blocks/post-featured-image/style-rtl.css 319 B
build/block-library/blocks/post-featured-image/style.css 319 B
build/block-library/blocks/post-navigation-link/style-rtl.css 153 B
build/block-library/blocks/post-navigation-link/style.css 153 B
build/block-library/blocks/post-template/editor-rtl.css 99 B
build/block-library/blocks/post-template/editor.css 98 B
build/block-library/blocks/post-template/style-rtl.css 314 B
build/block-library/blocks/post-template/style.css 314 B
build/block-library/blocks/post-terms/style-rtl.css 96 B
build/block-library/blocks/post-terms/style.css 96 B
build/block-library/blocks/post-time-to-read/style-rtl.css 69 B
build/block-library/blocks/post-time-to-read/style.css 69 B
build/block-library/blocks/post-title/style-rtl.css 100 B
build/block-library/blocks/post-title/style.css 100 B
build/block-library/blocks/preformatted/style-rtl.css 103 B
build/block-library/blocks/preformatted/style.css 103 B
build/block-library/blocks/pullquote/editor-rtl.css 135 B
build/block-library/blocks/pullquote/editor.css 135 B
build/block-library/blocks/pullquote/style-rtl.css 335 B
build/block-library/blocks/pullquote/style.css 335 B
build/block-library/blocks/pullquote/theme-rtl.css 167 B
build/block-library/blocks/pullquote/theme.css 167 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B
build/block-library/blocks/query-pagination/editor-rtl.css 221 B
build/block-library/blocks/query-pagination/editor.css 211 B
build/block-library/blocks/query-pagination/style-rtl.css 288 B
build/block-library/blocks/query-pagination/style.css 284 B
build/block-library/blocks/query-title/style-rtl.css 63 B
build/block-library/blocks/query-title/style.css 63 B
build/block-library/blocks/query/editor-rtl.css 450 B
build/block-library/blocks/query/editor.css 449 B
build/block-library/blocks/quote/style-rtl.css 222 B
build/block-library/blocks/quote/style.css 222 B
build/block-library/blocks/quote/theme-rtl.css 223 B
build/block-library/blocks/quote/theme.css 226 B
build/block-library/blocks/read-more/style-rtl.css 132 B
build/block-library/blocks/read-more/style.css 132 B
build/block-library/blocks/rss/editor-rtl.css 149 B
build/block-library/blocks/rss/editor.css 149 B
build/block-library/blocks/rss/style-rtl.css 289 B
build/block-library/blocks/rss/style.css 288 B
build/block-library/blocks/search/editor-rtl.css 178 B
build/block-library/blocks/search/editor.css 178 B
build/block-library/blocks/search/style-rtl.css 587 B
build/block-library/blocks/search/style.css 584 B
build/block-library/blocks/search/theme-rtl.css 114 B
build/block-library/blocks/search/theme.css 114 B
build/block-library/blocks/search/view.min.js 531 B
build/block-library/blocks/separator/editor-rtl.css 146 B
build/block-library/blocks/separator/editor.css 146 B
build/block-library/blocks/separator/style-rtl.css 234 B
build/block-library/blocks/separator/style.css 234 B
build/block-library/blocks/separator/theme-rtl.css 194 B
build/block-library/blocks/separator/theme.css 194 B
build/block-library/blocks/shortcode/editor-rtl.css 323 B
build/block-library/blocks/shortcode/editor.css 323 B
build/block-library/blocks/site-logo/editor-rtl.css 754 B
build/block-library/blocks/site-logo/editor.css 754 B
build/block-library/blocks/site-logo/style-rtl.css 203 B
build/block-library/blocks/site-logo/style.css 203 B
build/block-library/blocks/site-tagline/editor-rtl.css 86 B
build/block-library/blocks/site-tagline/editor.css 86 B
build/block-library/blocks/site-title/editor-rtl.css 116 B
build/block-library/blocks/site-title/editor.css 116 B
build/block-library/blocks/site-title/style-rtl.css 57 B
build/block-library/blocks/site-title/style.css 57 B
build/block-library/blocks/social-link/editor-rtl.css 184 B
build/block-library/blocks/social-link/editor.css 184 B
build/block-library/blocks/social-links/editor-rtl.css 674 B
build/block-library/blocks/social-links/editor.css 673 B
build/block-library/blocks/social-links/style-rtl.css 1.43 kB
build/block-library/blocks/social-links/style.css 1.42 kB
build/block-library/blocks/spacer/editor-rtl.css 348 B
build/block-library/blocks/spacer/editor.css 348 B
build/block-library/blocks/spacer/style-rtl.css 48 B
build/block-library/blocks/spacer/style.css 48 B
build/block-library/blocks/table/editor-rtl.css 433 B
build/block-library/blocks/table/editor.css 433 B
build/block-library/blocks/table/style-rtl.css 645 B
build/block-library/blocks/table/style.css 644 B
build/block-library/blocks/table/theme-rtl.css 146 B
build/block-library/blocks/table/theme.css 146 B
build/block-library/blocks/tag-cloud/style-rtl.css 251 B
build/block-library/blocks/tag-cloud/style.css 253 B
build/block-library/blocks/template-part/editor-rtl.css 403 B
build/block-library/blocks/template-part/editor.css 403 B
build/block-library/blocks/template-part/theme-rtl.css 101 B
build/block-library/blocks/template-part/theme.css 101 B
build/block-library/blocks/term-description/style-rtl.css 111 B
build/block-library/blocks/term-description/style.css 111 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B
build/block-library/blocks/text-columns/editor.css 95 B
build/block-library/blocks/text-columns/style-rtl.css 166 B
build/block-library/blocks/text-columns/style.css 166 B
build/block-library/blocks/verse/style-rtl.css 99 B
build/block-library/blocks/verse/style.css 99 B
build/block-library/blocks/video/editor-rtl.css 552 B
build/block-library/blocks/video/editor.css 555 B
build/block-library/blocks/video/style-rtl.css 174 B
build/block-library/blocks/video/style.css 174 B
build/block-library/blocks/video/theme-rtl.css 126 B
build/block-library/blocks/video/theme.css 126 B
build/block-library/classic-rtl.css 179 B
build/block-library/classic.css 179 B
build/block-library/common-rtl.css 1.1 kB
build/block-library/common.css 1.1 kB
build/block-library/editor-elements-rtl.css 75 B
build/block-library/editor-elements.css 75 B
build/block-library/elements-rtl.css 54 B
build/block-library/elements.css 54 B
build/block-library/interactivity/runtime.min.js 2.69 kB
build/block-library/interactivity/vendors.min.js 8.2 kB
build/block-library/reset-rtl.css 478 B
build/block-library/reset.css 478 B
build/block-library/theme-rtl.css 686 B
build/block-library/theme.css 691 B
build/block-serialization-default-parser/index.min.js 1.12 kB
build/block-serialization-spec-parser/index.min.js 2.87 kB
build/commands/style-rtl.css 827 B
build/commands/style.css 827 B
build/compose/index.min.js 12 kB
build/customize-widgets/index.min.js 11.9 kB
build/customize-widgets/style-rtl.css 1.46 kB
build/customize-widgets/style.css 1.45 kB
build/data-controls/index.min.js 640 B
build/date/index.min.js 40.4 kB
build/deprecated/index.min.js 451 B
build/dom-ready/index.min.js 324 B
build/dom/index.min.js 4.63 kB
build/edit-post/classic-rtl.css 544 B
build/edit-post/classic.css 545 B
build/edit-widgets/style-rtl.css 4.53 kB
build/edit-widgets/style.css 4.53 kB
build/element/index.min.js 4.8 kB
build/escape-html/index.min.js 537 B
build/format-library/style-rtl.css 554 B
build/format-library/style.css 553 B
build/hooks/index.min.js 1.55 kB
build/html-entities/index.min.js 448 B
build/i18n/index.min.js 3.58 kB
build/is-shallow-equal/index.min.js 527 B
build/keyboard-shortcuts/index.min.js 1.64 kB
build/keycodes/index.min.js 1.84 kB
build/list-reusable-blocks/index.min.js 2.13 kB
build/list-reusable-blocks/style-rtl.css 836 B
build/list-reusable-blocks/style.css 836 B
build/media-utils/index.min.js 2.9 kB
build/notices/index.min.js 948 B
build/plugins/index.min.js 1.77 kB
build/preferences-persistence/index.min.js 1.84 kB
build/preferences/index.min.js 1.24 kB
build/primitives/index.min.js 943 B
build/priority-queue/index.min.js 1.52 kB
build/react-i18n/index.min.js 615 B
build/react-refresh-entry/index.min.js 8.44 kB
build/react-refresh-runtime/index.min.js 7.31 kB
build/redux-routine/index.min.js 2.7 kB
build/reusable-blocks/style-rtl.css 243 B
build/reusable-blocks/style.css 243 B
build/router/index.min.js 1.77 kB
build/server-side-render/index.min.js 1.94 kB
build/shortcode/index.min.js 1.39 kB
build/token-list/index.min.js 582 B
build/url/index.min.js 3.57 kB
build/vendors/inert-polyfill.min.js 2.48 kB
build/vendors/react-dom.min.js 41.8 kB
build/vendors/react.min.js 4.02 kB
build/viewport/index.min.js 958 B
build/warning/index.min.js 268 B
build/widgets/index.min.js 7.16 kB
build/widgets/style-rtl.css 1.15 kB
build/widgets/style.css 1.16 kB
build/wordcount/index.min.js 1.02 kB

compressed-size-action

@github-actions
Copy link

github-actions bot commented May 30, 2023

Flaky tests detected in e77d0b7.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/5345949860
📝 Reported issues:

@WunderBart WunderBart force-pushed the refactor/playwright-performance-tests branch from e9e0177 to 2f163a3 Compare June 7, 2023 18:04
@WunderBart WunderBart force-pushed the refactor/playwright-performance-tests branch from 3e61312 to 4bf1c77 Compare June 9, 2023 15:26
@WunderBart WunderBart self-assigned this Jun 14, 2023
@WunderBart WunderBart force-pushed the refactor/playwright-performance-tests branch from 28e5037 to d9f2672 Compare June 16, 2023 15:09
@WunderBart
Copy link
Member Author

WunderBart commented Jun 19, 2023

I've prepared a Playwright vs. Puppeteer metrics comparison by running all the specs against both frameworks in a single CI job. Here are the results:

post-editor

metric trunk-puppeteer trunk-playwright % Change
serverResponse 238.94 ms 226.72 ms -5.39%
firstPaint 296.66 ms 455.46 ms 53.53%
domContentLoaded 704.82 ms 1122.16 ms 59.23%
loaded 705.48 ms 1124.4 ms 59.38%
firstContentfulPaint 1247.14 ms 1698.1 ms 36.11%
firstBlock 11093.48 ms 11332.06 ms 2.15%
type 73.42 ms 69.79 ms -5.21%
minType 71.72 ms 66.85 ms -7.28%
maxType 78.16 ms 78.58 ms 0.54%
typeContainer 17.21 ms 18.33 ms 6.50%
minTypeContainer 13.64 ms 14.78 ms 8.33%
maxTypeContainer 33.95 ms 24.8 ms -36.91%
focus 58.17 ms 69.85 ms 20.01%
minFocus 46.69 ms 58.15 ms 24.71%
maxFocus 91.38 ms 93.23 ms 2.02%
inserterOpen 33.34 ms 33.15 ms -0.57%
minInserterOpen 22.55 ms 22.32 ms -1.03%
maxInserterOpen 106.63 ms 119.19 ms 11.77%
inserterSearch 13.5 ms 6.1 ms -121.31%
minInserterSearch 4.08 ms 4.73 ms 15.96%
maxInserterSearch 34.78 ms 8.21 ms -323.51%
inserterHover 39.37 ms 38.88 ms -1.26%
minInserterHover 33.12 ms 35.71 ms 7.73%
maxInserterHover 86.2 ms 44.35 ms -94.36%
listViewOpen 275.32 ms 237.77 ms -15.80%
minListViewOpen 253.91 ms 213.88 ms -18.73%
maxListViewOpen 376.75 ms 324.29 ms -16.17%

site-editor

metric trunk-puppeteer trunk-playwright % Change
serverResponse 102.43 ms 122.4 ms 19.50%
firstPaint 289.07 ms 643.07 ms 122.47%
domContentLoaded 512.87 ms 1132.63 ms 120.77%
loaded 513.43 ms 1133.43 ms 120.68%
firstContentfulPaint 576.83 ms 1231.8 ms 113.42%
firstBlock 10697.07 ms 9241.53 ms -15.43%
type 53.77 ms 51.43 ms -4.35%
minType 51.03 ms 48.86 ms -4.25%
maxType 119.53 ms 66.51 ms -79.71%

front-end-classic-theme

metric trunk-puppeteer trunk-playwright % Change
timeToFirstByte 28.15 ms 34.75 ms 23.41%
largestContentfulPaint 64 ms 102.25 ms 59.77%
lcpMinusTtfb 36.15 ms 66.65 ms 84.27%

front-end-block-theme

metric trunk-puppeteer trunk-playwright % Change
timeToFirstByte 52 ms 53.65 ms 3.17%
largestContentfulPaint 82.15 ms 106.8 ms 30.00%
lcpMinusTtfb 29.15 ms 51.9 ms 78.06%

The biggest difference seems to be around the initial painting area (FP, FCP, LCP), which might be caused by the initialization of the custom e2e utils for each page, as mentioned by @kevin940726 in #51084 (comment). I'd like to try going vanilla Playwright to see if this is true, but I'd leave it for a follow-up PR. For this PR, I'd leave Puppeteer running perf specs as they are and only merge the migrated ones. Once we agree that the numbers are 👌, we do the switch. How does that sound, @youknowriad @oandregal @dmsnell?

@dmsnell
Copy link
Member

dmsnell commented Jun 19, 2023

@WunderBart thanks again for all the work you've put into this. I'm afraid I don't have much of value to offer, other than I don't see any reason this shouldn't be done. it's obvious the metrics will change; which I figure is the tradeoff you see as valuable enough to accept in order to eliminate Puppeteer from the repo. That's something you have more context about than I do, so it sounds fine enough.

I'll add my approval which is just that: me voicing my thoughts and nothing more. Please lean on whatever other review you see fitting, as well as your own vetting and testing.

@dmsnell dmsnell self-requested a review June 20, 2023 08:54
@dmsnell
Copy link
Member

dmsnell commented Jun 20, 2023

There is some confusion at the moment with existing discrepancies in testing metrics. After speaking with @youknowriad it seems like this could be a poor time to change them even more dramatically until we all know what happened and how to fix consistency. For background, it appears like changing the base version against which performance numbers are compared has skewed the measurements, or led to reporting slower performance when there's suspicion that it isn't slower.

All this is to say I pulled my approval back because we need to figure these things out first.

@WunderBart have you considered the impact on release performance reporting? Have you thought about running the tests against old versions to establish new baselines for comparing performance?

@WunderBart
Copy link
Member Author

There is some confusion at the moment with existing discrepancies in testing metrics. After speaking with @youknowriad it seems like this could be a poor time to change them even more dramatically until we all know what happened and how to fix consistency.

Possibly this was the reason for the skewed metrics? #51407 (comment)

For background, it appears like changing the base version against which performance numbers are compared has skewed the measurements, or led to reporting slower performance when there's suspicion that it isn't slower.

All this is to say I pulled my approval back because we need to figure these things out first.

@WunderBart have you considered the impact on release performance reporting? Have you thought about running the tests against old versions to establish new baselines for comparing performance?

Yes, although for this PR, I only wanted to do the 1:1 migration (no improvements, do with Playwright the same thing we do with Puppeteer) and merge without switching to Playwright. In the follow-up PRs, I wanted to address the aforementioned inconsistencies and improvements and collect all the measurements necessary for the switch.

Right now, the 1:1 migration is done, and the code is working. Merging this PR would not affect the current performance job at all.

@youknowriad
Copy link
Contributor

Thanks for all the work on this PR. Just want to make sure that we're still producing the same files as output so we continue to log things properly. Other than that I'm fine merging this.

@youknowriad
Copy link
Contributor

I think after a dozen commits, we'll get a clear idea about whether this is more stable or less stable than before.

@WunderBart
Copy link
Member Author

WunderBart commented Jun 26, 2023

edit: I accidentally deleted the comment instead of just editing it, so reposting now. Sorry for the noise!


Below are the results of 100 rounds of the Puppeteer vs. Playwright perf comparison job for the post editor suite. A hundred rounds were way too much for CI (which is how I learned about the 6h job timeout), so I decided to run it locally overnight. Click here for the source sheet.

Regarding the results, one thing that can be noticed immediately is that Puppeteer shows more stability in the load aspect, while Playwright is definitely better with the UI. Some metrics also show significant yet regular bumps (inserterOpen, listViewOpen, focus) for both frameworks which we might want to even out (e.g., by dropping the warmup sample).

I'd like to move forward with this PR (uninvasively), so please let me know what you think, folks! 🙌

/cc @dmsnell @youknowriad @oandregal @kevin940726

Loading performance

Metric Puppeteer Playwright
serverResponse Puppeteer_serverResponse Playwright_serverResponse
domContentLoaded Puppeteer_domContentLoaded Playwright_domContentLoaded
loaded Puppeteer_loaded Playwright_loaded
firstPaint Puppeteer_firstPaint Playwright_firstPaint
firstContentfulPaint Puppeteer_firstContentfulPaint Playwright_firstContentfulPaint
firstBlock Puppeteer_firstBlock Playwright_firstBlock

UI performance

Metric Puppeteer Playwright
type Puppeteer_type Playwright_type
typeContainer Puppeteer_typeContainer Playwright_typeContainer
inserterOpen Puppeteer_inserterOpen Playwright_inserterOpen
inserterHover Puppeteer_inserterHover Playwright_inserterHover
inserterSearch Puppeteer_inserterSearch Playwright_inserterSearch
listViewOpen Puppeteer_listViewOpen Playwright_listViewOpen
focus Puppeteer_focus Playwright_focus

@dmsnell
Copy link
Member

dmsnell commented Jun 26, 2023

I defer to @youknowriad and the aforementioned issue with the base version (which I think might already be resolved) - as far as my review is concerned this seems fine to merge.

@WunderBart
Copy link
Member Author

Thanks for all the work on this PR. Just want to make sure that we're still producing the same files as output so we continue to log things properly. Other than that I'm fine merging this.

Cool! This PR's code is totally passive - it doesn't change anything in the performance metrics area, just adds the migrated tests so that we have a starting point.

@WunderBart
Copy link
Member Author

As per #51084 (comment) and #51084 (comment), this one's safe to merge as it doesn't yet switch to Playwright. The remaining suggestions will be addressed in a follow-up PR so there's a clear distinction between the migration and enhancements. /cc @kevin940726

@WunderBart WunderBart merged commit 72b7d89 into trunk Jun 27, 2023
@WunderBart WunderBart deleted the refactor/playwright-performance-tests branch June 27, 2023 15:09
@github-actions github-actions bot added this to the Gutenberg 16.2 milestone Jun 27, 2023
sethrubenstein pushed a commit to pewresearch/gutenberg that referenced this pull request Jul 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Performance Related to performance efforts
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants