Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate screenshots during tests #266

Merged
merged 2 commits into from
Jan 28, 2019
Merged

Conversation

SebastienGllmt
Copy link
Contributor

Overview

This PR makes that all cucumber-js steps that contain "I should see" take a screenshot AFTER the step is done.

The problem with screenshot-based testing is that you may get very small pixel differences between baseline.
This is caused by UI elements like a spinner indicating a loading (the difference could just be the spinner at a different rotation)

The rationale for only taking screenshots on "I should see" tests is that these tests should BLOcK on UI elements and "see" a stable view of the UI after the blocking is done
This is exactly what we want to take a screenshot of to compare with.

Changes made to cucumber library

Unfortunately Cucumber doesn't let you add a hook for every step (which is what we need)

To fix this, I did the following:

  1. Upgraded cucumber-js to 5.1.0 (changelog here)
  2. Forked cucumber-js into my own repo
  3. Copied this PR to my personal repo
  4. Made changes to my branch to match NPM releae format of cucumber-js (notably, lib folder was required (otherwise in .gitignore)

Future work

You can see for example in my PR, one of our tests said "I should see" but in fact it had no blocking UI call at all (it only checked the value of some variable)

Therefore, to effectively do screenshot-based testing, one would have to

  1. Run the E2E-test a few times and see if any of the screenshot differ by small amounts (and then figure out how we can avoid it)
  2. Go over all our tests to see which steps should be renamed to use "I should see" (for maximum screenshot coverage)

This PR doesn't make screenshot testing as part of our CI but that is the ultimate goal.

Ideally, our CI would cache the screenshot tests for specific commits. Then, when you make a PR, it checks at which commit you forked from develop and compare your PR to that (Note: you CANNOT compare to current develop as too many things may have changed. You have to compare to where the branch forked)

Save Location

  • Scenarios in their own folders
  • Each step is its own file (step # as first part of key)
  • Some tests have multiple "examples" that mean we run the same scenario multiple times. Therefore we also add the line number to the unique key of the file

Note: Screenshots folder is deleted every time you run a new test

Example

Test definition

  @it-8
  Scenario Outline: Wallet renaming (IT-8)
    When I am testing "Wallet Settings Screen"
    And There is a wallet stored named Test
    And I navigate to the general settings screen
    And I click on secondary menu "wallet" item
    And I click on "name" input field
    And I enter new wallet name:
    | name         |
    | <walletName> |
    And I click outside "name" input field
    And I navigate to wallet transactions screen
    Then I should see new wallet name "<walletName>"
    Examples:
    | walletName                               |                    |
    | first Edited                             |2 words name        | 
    |ウォレットの追加                             |Japanese            |
    |지갑 추가                                 | Korean             |
    |НАСТРОЙКИ                                 | Russian            |
    | a                                        |1-characters length |
    | asdfghjklpoiuytrewqazxcvbnmlkjhgfdsaqwer |40 characters length|

Run test

npm run test-by-tag-chrome @it-8

Created files

yoroi-frontend/screenshots/Wallet renaming IT8/10_96-I should see new wallet name .png
yoroi-frontend/screenshots/Wallet renaming IT8/10_97-I should see new wallet name .png
yoroi-frontend/screenshots/Wallet renaming IT8/10_98-I should see new wallet name .png
yoroi-frontend/screenshots/Wallet renaming IT8/10_99-I should see new wallet name .png
yoroi-frontend/screenshots/Wallet renaming IT8/10_100-I should see new wallet name .png
yoroi-frontend/screenshots/Wallet renaming IT8/10_101-I should see new wallet name .png

Screenshot generated

image

@shinsuny shinsuny self-requested a review January 28, 2019 06:38
@shinsuny
Copy link
Contributor

LGTM, cool !!

@SebastienGllmt SebastienGllmt merged commit d7f83dd into develop Jan 28, 2019
@SebastienGllmt SebastienGllmt deleted the feature/screenshots branch January 28, 2019 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants