Run e2e performance regression tests on merge into `main` via AWS Device farm #12320

AndrewGable · 2022-10-31T22:28:39Z

Details

This PR adds e2e performance tests that is testing for regressions on PRs merged into main. It compares the last production release to the commit that was merged into main, if there is any performance regression found it will leave the DeployBlockerCash label and an App Deployer will look into the regression and decide wether to revert or move forward.

Fixed Issues

$ #11711

Tests

We need to merge this PR to fully test this
On merge, I would expect the test to run and a comment looking like this will be added:

Offline tests

N/A - No source code changes, just adding a test

QA Steps

N/A - No source code changes, just adding a test

PR Author Checklist

PR Reviewer Checklist

The reviewer will copy/paste it into a new comment and complete it after the author checklist is completed

Screenshots/Videos

roryabraham · 2022-11-29T09:21:37Z

@AndrewGable is this ready for review?

mananjadhav · 2022-11-29T09:22:45Z

does this require C+ review?

AndrewGable · 2022-11-29T14:30:20Z

Yes ready for review, yes a c+ review would be great.

mananjadhav · 2022-11-30T17:27:54Z

Still reviewing. Will need a day.

AndrewGable · 2022-12-01T23:04:06Z

Bump @roryabraham @mananjadhav @PauloGasparSv - If you cannot review, please let me know and I can find new reviewers, thanks!

roryabraham

Few comments. Overall looks okay but is a challenging PR to review

.github/workflows/preDeploy.yml

hannojg · 2022-12-05T09:42:35Z

@AndrewGable are we still sure that 30 test runs are necessary to get accurate results? Maybe something lower like 24 or so does also work? Because as we will add more tests that will greatly have an impact on the test suite's duration 😅

.github/workflows/preDeploy.yml

PauloGasparSv

Based on the comments here and my understanding changes LGTM.

PauloGasparSv · 2022-12-05T19:59:18Z

Reviewer Checklist

I have verified the author checklist is complete (all boxes are checked off).
I verified the correct issue is linked in the ### Fixed Issues section above
I verified testing steps are clear and they cover the changes made in this PR
- I verified the steps for local testing are in the Tests section
- I verified the steps for Staging and/or Production testing are in the QA steps section
- I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
- I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
I checked that screenshots or videos are included for tests on all platforms
I included screenshots or videos for tests on all platforms
I verified tests pass on all platforms & I tested again on:
- iOS / native
- Android / native
- iOS / Safari
- Android / Chrome
- MacOS / Chrome
- MacOS / Desktop
If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
I verified proper code patterns were followed (see Reviewing the code)
- I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
- I verified that comments were added to code that is not self explanatory
- I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
- I verified any copy / text shown in the product was added in all src/languages/* files
- I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
- I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
- I verified the JSDocs style guidelines (in STYLE.md) were followed
If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
I verified that this PR follows the guidelines as stated in the Review Guidelines
I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
If a new component is created I verified that:
- A similar component doesn't exist in the codebase
- All props are defined accurately and each prop has a /** comment above it */
- The file is named correctly
- The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
- The only data being stored in the state is data necessary for rendering and nothing else
- For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
- Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
- All JSX used for rendering exists in the render method
- The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
If a new CSS style is added I verified that:
- A similar style doesn't already exist
- The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)
If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

Mobile Web - Chrome

Mobile Web - Safari

Desktop

iOS

Android

roryabraham · 2022-12-05T20:01:48Z

.github/actions/composite/configureAwsCredentials/action.yml

@@ -18,4 +22,4 @@ runs:
      with:
        aws-access-key-id: ${{ inputs.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ inputs.AWS_SECRET_ACCESS_KEY }}
-        aws-region: us-east-1
+        aws-region: ${{ inputs.AWS_REGION }}


I see what you mean about this action being pointless now ... annoying that local composite actions from the same repo don't have access to secrets.

AndrewGable · 2022-12-05T21:08:18Z

@hannojg - Let's reduce the number of test runs when we add more tests.

AndrewGable · 2022-12-05T21:11:48Z

I will monitor and test now that this is merged 👍

OSBotify · 2022-12-05T21:14:16Z

✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

AndrewGable · 2022-12-05T21:24:25Z

Looks like testing will be held on this internal task: https://github.com/Expensify/Expensify/issues/232949#issuecomment-1338183844

I forgot to enable large runners on the new workflow.

mananjadhav · 2022-12-06T10:27:27Z

I could only review the code here, and I got stuck at the fastlane installation to run the e2e tests locally. By the time I am now able to work on this, this is already merged.

OSBotify · 2022-12-13T22:40:56Z

🚀 Deployed to production by @chiragsalian in version: 1.2.38-6 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

AndrewGable added 4 commits October 31, 2022 12:41

Apply some logging tweaks

b857f73

Add TestSpec

106a88b

Tweak TestSpec to a working solution

9ce17a6

Tweak artifacts

b06342f

AndrewGable self-assigned this Oct 31, 2022

AndrewGable added 25 commits November 2, 2022 14:54

Move e2e to tests folder and remove unused code

7bec64d

Try making run from GitHub actions

6c255fb

Use correct version

1631bea

Use workflow_dispatch as well

cb30060

Use macos for Bundle

30d1c33

Delete unused workflows to clean up my inbox

579141e

Run on push

478ee01

Set up ruby and include device pool

48c9f8c

Tweak android build settings

574b8fb

Add debug lines

84b94e4

Correct env file location

90f41cc

Tweak two configs to pass tests

57dc2d7

Remove debug

1d3a54b

Tweak install path for adb

0633815

Show all files

a57301e

Add find

6b53c5d

Try to fix file paths

278350d

Try to grab artifacts from device farm

1a95f0a

Try to see if this improves perf of build

3e9deb0

Test cache

4ad6c44

Try to use straight npm ci

4bd55e1

Try different path for artifacts

1ffc8d0

Try one last path

665296a

Clean up TestSpec

540f36b

Try to save to WORKING_DIRECTORY

2048bd7

This comment was marked as outdated.

Sign in to view

Remove test file

372aa2e

Move checkout back

e679eef

hannojg mentioned this pull request Nov 30, 2022

dev: perf regression test configuration #13194

Merged

51 tasks

roryabraham requested changes Dec 2, 2022

View reviewed changes

Merge branch 'main' into andrew-aws-df

e11332d

AndrewGable added the DeployBlockerCash This issue or pull request should block deployment label Dec 2, 2022

OSBotify mentioned this pull request Dec 2, 2022

Deploy Checklist: New Expensify 2022-12-02 #13283

Closed

38 tasks

AndrewGable removed the DeployBlockerCash This issue or pull request should block deployment label Dec 2, 2022

Apply peer review

df5905f

AndrewGable requested a review from roryabraham December 2, 2022 20:07

PauloGasparSv reviewed Dec 5, 2022

View reviewed changes

.github/workflows/preDeploy.yml Show resolved Hide resolved

PauloGasparSv approved these changes Dec 5, 2022

View reviewed changes

roryabraham approved these changes Dec 5, 2022

View reviewed changes

AndrewGable merged commit 6380fe8 into main Dec 5, 2022

AndrewGable deleted the andrew-aws-df branch December 5, 2022 21:08

francoisl mentioned this pull request Dec 8, 2022

Deploy Checklist: New Expensify 2022-12-07 #13431

Closed

72 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run e2e performance regression tests on merge into `main` via AWS Device farm #12320

Run e2e performance regression tests on merge into `main` via AWS Device farm #12320

AndrewGable commented Oct 31, 2022 •

edited

Loading

This comment was marked as outdated.

roryabraham commented Nov 29, 2022

mananjadhav commented Nov 29, 2022

AndrewGable commented Nov 29, 2022

mananjadhav commented Nov 30, 2022

AndrewGable commented Dec 1, 2022

roryabraham left a comment

hannojg commented Dec 5, 2022

PauloGasparSv left a comment

PauloGasparSv commented Dec 5, 2022 •

edited

Loading

roryabraham Dec 5, 2022

AndrewGable commented Dec 5, 2022

AndrewGable commented Dec 5, 2022

OSBotify commented Dec 5, 2022

AndrewGable commented Dec 5, 2022

mananjadhav commented Dec 6, 2022

OSBotify commented Dec 13, 2022

Run e2e performance regression tests on merge into main via AWS Device farm #12320

Run e2e performance regression tests on merge into main via AWS Device farm #12320

Conversation

AndrewGable commented Oct 31, 2022 • edited Loading

Details

Fixed Issues

Tests

Offline tests

QA Steps

PR Author Checklist

PR Reviewer Checklist

Screenshots/Videos

This comment was marked as outdated.

roryabraham commented Nov 29, 2022

mananjadhav commented Nov 29, 2022

AndrewGable commented Nov 29, 2022

mananjadhav commented Nov 30, 2022

AndrewGable commented Dec 1, 2022

roryabraham left a comment

Choose a reason for hiding this comment

hannojg commented Dec 5, 2022

PauloGasparSv left a comment

Choose a reason for hiding this comment

PauloGasparSv commented Dec 5, 2022 • edited Loading

Reviewer Checklist

Screenshots/Videos

roryabraham Dec 5, 2022

Choose a reason for hiding this comment

AndrewGable commented Dec 5, 2022

AndrewGable commented Dec 5, 2022

OSBotify commented Dec 5, 2022

AndrewGable commented Dec 5, 2022

mananjadhav commented Dec 6, 2022

OSBotify commented Dec 13, 2022

Run e2e performance regression tests on merge into `main` via AWS Device farm #12320

Run e2e performance regression tests on merge into `main` via AWS Device farm #12320

AndrewGable commented Oct 31, 2022 •

edited

Loading

PauloGasparSv commented Dec 5, 2022 •

edited

Loading