-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix flakey chromatic visual regression tests #10038
Comments
A couple hypotheses:
|
I was able to replicate a layout shift in storybook:
Interestingly, subsequent refreshes don't cause a layout shift, only the first after an empty cache replicate-shift.mov |
seems to be flakey calculations in with the same window size and process followed from above
As you can see, the |
It looks like the |
I've determined this is a font loading issue. Since the width is being incorrectly calculated, I put a
I tried pulling #9993 but it didn't fix the issue |
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Fixes #10038 Use system font in chromatic snapshots (use Inter still in storybook) so that we can use the regression tests without flakiness during our se23 consolidation work. I made an [issue here](#10104) to figure out how to get stories to not load until the font has loaded so we can use the proper font. ### WHAT is this pull request doing? Using the [`isChromatic`](https://www.chromatic.com/docs/ischromatic) function to only control behaviour in snapshots, this removes the font loading link from the document preview head. ### How to 🎩 * Check the [UI Tests CI](https://shopify.chromatic.com/build?appId=5d559397bae39100201eedc1&number=19503) to see the system font updated in all snapshots * Check that the published [storybook in CI](https://5d559397bae39100201eedc1-tyfbdvqiif.chromatic.com/) is still using Inter
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Fixes #10038 Use system font in chromatic snapshots (use Inter still in storybook) so that we can use the regression tests without flakiness during our se23 consolidation work. I made an [issue here](#10104) to figure out how to get stories to not load until the font has loaded so we can use the proper font. ### WHAT is this pull request doing? Using the [`isChromatic`](https://www.chromatic.com/docs/ischromatic) function to only control behaviour in snapshots, this removes the font loading link from the document preview head. ### How to 🎩 * Check the [UI Tests CI](https://shopify.chromatic.com/build?appId=5d559397bae39100201eedc1&number=19503) to see the system font updated in all snapshots * Check that the published [storybook in CI](https://5d559397bae39100201eedc1-tyfbdvqiif.chromatic.com/) is still using Inter
### WHY are these changes introduced? Resolves #10038 ### WHAT is this pull request doing? * Preloads the `inter` font to avoid flakey width/postion calculations in chromatic snapshots (more context in the issue) * Run chromatic in CI when `polaris-react/.storybook` changes * Accept chromatic baseline UI snapshots using inter > Note: I think inter now shows when the beta flag is off as well but this is ok as we are on the `next` branch and in the process of removing the beta flag ### How to 🎩 #### To duplicate the original bug: 1. Go to [this story](https://storybook.polaris.shopify.com/?path=/story/all-components-legacycard--with-separate-header&globals=polarisSummerEditions2023:true) 2. Make sure the cache is empty (empty cache and hard reload) 3. Refresh (or rerender) to see popover overlay shift <details> <summary>Replicate Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/53ed5e19-7c8d-4ee4-ab95-81a2c08e279d </details> #### To tophat fix: 1. Go to [this story on this PR's chromatic](https://5d559397bae39100201eedc1-qhkfhdyjuo.chromatic.com/?path=/story/all-components-legacycard--with-separate-header) 2. Repeat steps 2-3 above <details> <summary>Fixed Bug Demo</summary> https://github.com/Shopify/polaris/assets/20652326/de28ff59-84ab-46bd-b28a-31dbede8f001 </details>
### WHY are these changes introduced? Fixes #10038 Use system font in chromatic snapshots (use Inter still in storybook) so that we can use the regression tests without flakiness during our se23 consolidation work. I made an [issue here](#10104) to figure out how to get stories to not load until the font has loaded so we can use the proper font. ### WHAT is this pull request doing? Using the [`isChromatic`](https://www.chromatic.com/docs/ischromatic) function to only control behaviour in snapshots, this removes the font loading link from the document preview head. ### How to 🎩 * Check the [UI Tests CI](https://shopify.chromatic.com/build?appId=5d559397bae39100201eedc1&number=19503) to see the system font updated in all snapshots * Check that the published [storybook in CI](https://5d559397bae39100201eedc1-tyfbdvqiif.chromatic.com/) is still using Inter
For some reason, stories with
Popover
and/orTooltip
are indeterminately changing every time chromatic regression tests are run. Examples:We should figure out why this is happening and fix it
The text was updated successfully, but these errors were encountered: