Investigate workflow job failing on main: e2ePerformanceTests / Run E2E tests in AWS device farm

[github-actions]

🚨 Failure Summary 🚨:

• 📋 Job Name: e2ePerformanceTests / Run E2E tests in AWS device farm
• 🔧 Failure in Workflow: Process new code merged to main
• 🔗 Triggered by PR: PR Link
• 👤 PR Author: @struc
• 🤝 Merged by: @roryabraham
• 🐛 Error Message:
  warning: The following actions use a deprecated Node.js version and will be forced to run on node20: realm/aws-devicefarm@7b9a912. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
  failure: Process completed with exit code 1.
  warning: No matching artifact of type LOG found for debug.log, skipping
  failure: Test run failed after 2450 seconds with: undefined. Timeout is set to 5400

⚠️ Action Required ⚠️:

🛠️ A recent merge appears to have caused a failure in the job named e2ePerformanceTests / Run E2E tests in AWS device farm.
This issue has been automatically created and labeled with Workflow Failure for investigation.

👀 Please look into the following:

1. Why the PR caused the job to fail?
2. Address any underlying issues.

🐛 We appreciate your help in squashing this bug!

[roryabraham]

weird, all we did here is remove effects and animations, which I would expect to make things simpler overall.

Looking at the logs, it seems unrelated.

[roryabraham]

asked for help in slack: https://expensify.slack.com/archives/C035J5C9FAP/p1725321344063379

[struc]

I think the issue is mainly here:

failed with error: Command failed: adb shell pm grant com.expensify.chat.e2edelta android.permission.POST_NOTIFICATIONS

Exception occurred while executing 'grant':
java.lang.IllegalArgumentException: Unknown permission: android.permission.POST_NOTIFICATIONS

As far as I remember this permission is only available on Android 33+, so older versions will not recognize the permission.

[kirillzyusko]

@struc we know that and we handle it. Few lines below we print a message:

[Mon Sep 02 2024 22:57:58 GMT+0000 (Coordinated Universal Time)]  ⚠️  Failed to grant push notifications permissions. It might be due to the fact that push-notifications permission type is not supported on this OS version yet. Continue tests execution...

The problem was the fact that upgrade to RN 0.75 made "typing test" flaky and almost all runs were not successful. When we merged #48191 we fixed flakiness in delta builds, and thus we got a miracle chance that tests may pass (baseline was still flaky, but still there could be a chance). And that's exactly what we've got:

And our CI saw previous successful runs and associated the failure with a merged PR and created the issue. But changes from these PR don't affect tests. Tests were flaky and because of that we end up in this situation.

After a new release I believe all tests will become stable again and we will not have random failures 🤞

[kirillzyusko]

I think the issue can be closed because e2e pipeline seems to be stable now!