Polaris: As a developer I want an automated way to create and run e2e tests

[sergi]

Create e2e test automation using https://github.com/wix/detox

[andreaforni]

I've done some researches to see how to integrate Detox with Expo.

Short answer: it's not possible to use Detox with an Expo project.

Details

Detox does not officially support Expo, as written in their documentation:

Note: Expo support is entirely a community driven effort. We have no specific support in Detox for Expo applications (ejected or otherwise).

I've searched online for tutorials or examples, but they are all outdated. All the examples I've found use a couple of Expo helpers:

• expo/detox-tools: Tools for the detox testing framework for expo apps
• expo/detox-expo-helpers

But, expo/detox-tools:

• Last commit: November 2018
• 1 issue open: Not working with detox > 12.3.0

And expo/detox-expo-helpers:

• Last commit: November 2018
• 7 issues open:
  • app not starts on iOS 13.4 simulator
  • Add android example / support

This is a list of tutorial/example I tried, but they didn't work:

1. Testing Expo Apps with Detox and react-native-testing-library:
   • It's dated back to January 2019
   • It uses expo/detox-tools and expo/detox-expo-helpers
   • The command detox test crashes
   • There is no support for Android
2. Sample Expo app with e2e tests using Detox, Jest and TypeScripts: GitHub - yaron1m/expo-detox-typescript-example
   • Last commit: August 2019
     • It uses expo/detox-tools and expo/detox-expo-helpers
   • Issues:
     1. This example doesn’t work any more · Issue #5 (Issues with detox >=12)
        • The README, In the FAQ section, states the project does not work with detox > 12.3.0
     2. Android support? · Issue #4 (Android support missing)
3. Tutorial on ReactNativeTesting.io: Setting Up Detox
   • The configuration proposed does not work, the binaryPath to set in .detoxrc.json (path: iOS/build/Build/Products/Debug-iphonesimulator/...) does not exist in a non-ejected expo project

Conclusion

To make Detox work with Polaris, we should:

• Try ejecting it. In a previous expo-ejected project, I was able to configure Detox. To be honest, I had other issues related to the synchronisation between Detox and the tested app (see Dealing With Synchronization Issues in Tests
  ), but I was able to run simple e2e tests.
• Do not use Expo.

[andreaforni]

The official Expo documentation does not mention any solution to provide e2e tests. The documentation only describes:

• how to do unit test, snapshot test and code coverage reports with Jest.
• how to test manually the app on a real device or emulator.

[andreaforni]

Another thing to note is that Detox is a:

Gray box end-to-end testing and automation library for mobile apps.

It does not support e2e testing for web apps. So we should configure an alternative tool to run e2e tests on web.

In addition to that, even though Detox uses Jest to run the tests, I'm not so sure we can use the same test and run it on both mobile and web environment. Detox, in fact, provides its own matchers, actions and expectations. So we could end up writing the same test two times: one version for mobile and the other for the web.

[andreaforni]

We decided to provide separate support to e2e web testing using Cypress.

For the mobile side, an alternative to Detox could be Appium. I'll take some time to test it.

[andreaforni]

Mobile e2e tests with Appium

In this post, I want to summarise my experience with Appium, what I've done and what still needs to be done.

All the code I developed so far is in the draft PR: #59

These are the two steps required to run the mobile e2e test with Appium in Polaris:

1. Build the iOS and Android app.
2. Run the tests.

Build the apps

Appium installs the app inside the emulator/simulator before running the tests, so it needs an Android .apk and an iOS .app files.

To build the apps locally, you have to:

1. Publish the resources.
2. Build apps.

Publish the resources

Expo officially provides support to build apps locally, please check the official documentation: Building Standalone Apps on Your CI.

The first step to build an app locally is to publish the static resources. This step requires an Expo account, so, if you don't have one, go to https://expo.io/signup. Then, you can run:

npm run publish

This command publishes the resources in a private URL (something like: https://exp.host/@your_expo_username/polaris), thanks to the config property:

// app.json
"privacy": "unlisted",

In this way, only who knows the URL can load the app using the QRCode and the Expo client on his phone.

Please note: there is even a safer option, as described in Hosting Updates on Your Servers, but requires to set-up a private server where publish the updated resources.

Build the apps

If it's the first time building the app, follow the installation steps in Building Standalone Apps on Your CI. Please, remember to set-up the two environment variables EXPO_USERNAME and EXPO_PASSWORD with your Expo's credentials (as described in "Start the build") before running the following scripts.

To build the Android app, run:

npm run build:local:android

This command generates the following file: build/polaris.apk.

For the iOS app, run:

npm run build:local:ios

This command generates the following files: build/polaris.tgz and build/polaris.app (Appium uses the latter).

Run the tests

Before running the test, follow the "Introduction to Appium" and "Getting Started" tutorials and install:

• the appium server (required to run the tests)
• the Appium desktop app (useful to inspect the app while you are writing the tests)

After that, you are ready to run the tests. First, run the Appium server (a possible improvement could be to set-up the Appium Service that runs the server automatically before the tests):

appium

And in another terminal, run the Android test:

node e2e/index.js 

Or the iOS one:

node e2e/index.js ios

As you can see, there isn't a package.json script in place yet. The test (e2e/index.js) is a Node program that load the iOS or Android configuration based on the first argument: android (the default) or ios.

The main idea is to write a single test that can run on both platforms.

I'm close to this objective but not there yet, let's dive into the code.

Test internals

All the code for the mobile e2e tests is located insider the e2e/ folder. Here you can find 3 files:

• config.android.js: the Appium configuration files for Android
• config.ios.js: the Appium configuration files for iOS
• index.js: the test

The test

The test is the same implemented for the web (cypress/integration/i18n.spec.js): from the home screen clicks the settings button, in the settings page changes the language from English to Italian and checks a label.

As you can see, the test does not use any testing library (no Jest, or Mocha, etc.), it simply uses the webdriverio to control the app and get info from it. There are no assertions in the test, yet. My focus was to be able to navigate the app and get the values from it.

Still missing: configure a testing framework to run the test. Polaris is using Jest for unit tests, so we should try to use that for this e2e test.

How to identify and select an element in the app

As you can see from the code, webdriverio uses a JQuery-style selector $(). This selector takes in input XPath and IDs.
The XPath is usually platform-dependant (for example: /hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/...) so they are not reusable in both platform. For this reason, I tried to use IDs. The format of the ID selector is a tilde plus the ID:

 const settingsButton = await client.$('~go-to-settings');

Here, got-to-settings is the ID and '~go-to-settings' is the selector.

Next question/step is: how do I define an ID in the app's code so that we can reference it in the test? Well, in iOS we can use the testID prop, on Android we should use accessibilityLabel prop. Unfortunately, we cannot use testID on Android too, as explained in this React Native's PR (It's from 2016, but it's still valid).

So, if you set testID="go-to-settings" in the "go to settings link", the test running on iOS is able to find the button, but the one on Android is not. On the other side if you set both testID="go-to-settings" (for iOS compatibility) and accessibilityLabel="go-to-settings", the Android's test is able to locate the link.

Unfortunately, the use of accessibilityLabel="go-to-settings" creates a problem in the iOS test, because webdriverio is not able to locate the link using testID, or to be more accurate, it founds more than one element with that ID and returns the wrong one.

This issue is clear when you try to inspect the iOS app using the Appium's Desktop application. When accessibilityLabel is not used, only one element has the go-to-settings ID set correctly:

This element (the more internal one) is the one that must be clicked to open the new screen.

At the contrary, if bothtestID and accessibilityLabel are set, two iOS elements have the same name, and the wrong one is clicked:

To solve this issue, the accessibilityLabel prop is set on Android's platform only:

<Component
  style={style}
  title={titleAsProp ? title : null}
  onPress={() => navigate(path, params)}
  testID={testID}
  accessibilityLabel={Platform.OS === 'android' ? testID : null}
>

This solve the issue to identify all (almost) elements in the app to interact with. Unfortunately, there is still an issue on Android: find a way to locate and click the "Italian" item in the Android's picker. In iOS the following lines of code work correctly:

  const italianLanguage = await client.$('~Italiano');
  await italianLanguage.click();

But on Android, no item is found with that selector. I tried to add an accessibilityLabel to the Picker.Item in src/components/atoms/picker-sheet/index.jsx but it didn't work. If I inspect the Android's app using Appium's Desktop inspector, I see no ID set for the items' list.

The Appium configuration files for iOS and Android

The two files e2e/config.android.js and e2e/config.ios.js contain the configuration that allows Appium to connect to the emulator/simulator and install the app:

// e2e/config.ios.js
const configuration = {
  path: '/wd/hub',
  port: 4723,
  capabilities: {
    platformName: 'iOS',
    automationName: 'XCUITest',
    deviceName: 'iPhone 11',
    app: path.normalize(`${__dirname}/../build/polaris.app`)
  }
};

// e2e/config.android.js
const configuration = {
  path: '/wd/hub',
  port: 4723,
  capabilities: {
    platformName: 'Android',
    automationName: 'UiAutomator2',
    deviceName: 'Android Emulator',
    app: path.normalize(`${__dirname}/../build/polaris.apk`),
    avd: 'Nexus_5_API_29'
  }
};

In both configurations, is the capabilities part that changes. In particular:

• automationName defines the driver used by Appium to control the simulator/emulator (XCUITest on iOS and UiAutomator2 on Android)
• deviceName on iOS defines the name of the simulator to use. The string used here has to match the name on an installed iOS simulator on your machine. In my test, I used a "iPhone 11 simulator". On Android is a simple, descriptive string.
• app is the absolute path of the app to install and run.
• avd is an Android-only field and define the name of the installed emulator to run and use for the tests. In my test, I used a "Nexus 5, with API v29".

Please note, you can use the capabilities above to run the inspector provided by the Appium Desktop Application:

• In the console, start the appium server.
• Open the desktop app.
• Select: Appium -> New Session Window.
• Copy the capabilities above (with the path in app resolved to the actual path) in the "JSON Representation` section:

• Click on "Start Session" and inspect the app.

[simoneb]

Adding another one to the list: https://cavy.app/

[simoneb]

Current status is:

• detox doesn't support managed expo, so not an option
• cavy doesn't support managed expo, so not an option

[andreaforni]

This week I worked on Simone's PR about "e2e web tests via appium", and this is what I've found:

• We can automate Web testings using WebdriverIO directly.
• Appium uses WebdriverIO to automate and control a mobile application, but WebdriverIO can control everything that speaks the WebDriver Protocol: browsers (directly) or mobile apps (thanks to Appium). So a test written with WebDriverIO can run on Android, iOS or a web browser; we just need to provide the correct configuration for every environment (we also need to find a way to use the same selector in every environment, see "How to identify and select an element in the app" section in my previous comment)
• WebdriverIO implements the DevTool Protocol too, so it can control Chromium-based browser. Not needed here, but it's good to know.
• WebdriverIO provides many methods that we can use in our tests and not listed in Appium's documentation. So, it's important to look at WebdriverIO documentation too. In this way, I've found the waitForDisplayed used to wait that the home screen is fully loaded before continuing with the test:

  const welcomeMessageText = await client.$(selector)
  await welcomeMessageText.waitForDisplayed({ timeout: 10 * 1000 })

• WebdriverIO provides a testrunner too and integration with Mocha, Jasmine and Cucumber. So it's not just an automation tool, but it can also be used as a test framework. Unfortunately it the documentation doesn't mention Jest.