Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Update e2e-test:native-web-tap tests for WebDriverIO compatibility #1490

Merged
merged 24 commits into from
Feb 6, 2023

Conversation

mwakizaka
Copy link
Contributor

@mwakizaka mwakizaka commented Jan 22, 2023

@mykola-mokhnach
Copy link
Contributor

mykola-mokhnach commented Jan 26, 2023

@mwakizaka Shall we merge this PR or you'd like to do more changes (I can observe some failures during the most recent CI run)?

@mwakizaka
Copy link
Contributor Author

@mwakizaka Shall we merge this PR or you'd like to do more changes (I can observe some failures during the most recent CI run)?

oh, let me check. thank you for letting me know.

Comment on lines 19 to 20
'appium:fullReset': true,
'appium:noReset': false,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this is an appropriate way to avoid ci tests fail (I'd like to just close all tabs in mobile safari before running these tests)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a possibility to influence global safari settings in Simulator: https://appium.github.io/appium-xcuitest-driver/4.16/execute-methods/#mobile-updatesafaripreferences

Maybe this could help.
Enabling fullReset may significantly slow down tests as it resets the whole simulator

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry to be late, so far I haven't managed to find a way to close tabs via mobile: updateSafariPreferences. To me it looks infeasible.

If no problem, I'm thinking about trying to open Preferences app and close the safari tabs via GUI at before mocha function.

Comment on lines +25 to +29
{w: 1080, h: 2340}, // 13 mini, 12 mini
{w: 1170, h: 2532}, // 14, 13, 13 Pro, 12, 12 Pro
{w: 1284, h: 2778}, // 14 Plus, 13 Pro Max, 12 Pro Max
{w: 1179, h: 2556}, // 14 Pro
{w: 1290, h: 2796}, // 14 Pro Max
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need this to fix nativeWebTapStrict feature for these devices.

Copy link
Contributor

@mykola-mokhnach mykola-mokhnach Jan 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for sure we need: appium/appium#17014

@KazuCocoa
Copy link
Member

Maybe we can merge this once..?

@mwakizaka
Copy link
Contributor Author

Maybe we can merge this once..?

sorry, but maybe I don't have enough time to examine further. if no problem, I'd like to merge this once.

@mykola-mokhnach
Copy link
Contributor

Maybe we can merge this once..?

sorry, but maybe I don't have enough time to examine further. if no problem, I'd like to merge this once.

I would propose to simply mark unstable tests with return this.skip() for now, so the CI is always green

before(async function () {
skipped = false;
try {
await closeAllTabsViaSettingsApp(deviceName);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed my mind and modified to close the safari tabs via GUI.

[MEMO]

  1. As for https://appium.github.io/appium-xcuitest-driver/4.16/execute-methods/#mobile-updatesafaripreferences, I tried removing the contents of BrowserControllersSavedState property in Library/Preferences/com.apple.mobilesafari.plist file but it doesn't help.
  2. While playing with Preferences app, I saw a line of device log like default 22:07:24.666146+0900 Preferences -[UIApplication _beginBackgroundTaskWithName:expirationHandler:]: Will add backgroundTask with taskName: com.apple.mobilesafari.settings.DeleteAllDataAndCachesTask, expirationHandler: <__NSMallocBlock__: 0x600001680a20>. I guess it is probably a function embedded in com.apple.Preferences app which closes safari tabs. If it's possible to kick via CUI, it's much better but I don't have any good idea.

@KazuCocoa
Copy link
Member

KazuCocoa commented Feb 4, 2023

The idb test failures were invalid Xcode path things. Maybe we should fix the path. You can ignore it in this PR.
hm.. but other passed ones also had the same path... maybe something was wrong in the idb setup.

@mwakizaka
Copy link
Contributor Author

thank you. I'll check e2e_web_iPhone13_16_2 failures soon

@mwakizaka
Copy link
Contributor Author

thank you. I'll check e2e_web_iPhone13_16_2 failures soon

I checked the log and concluded that it is not likely related to my changes (most likely similar to appium/appium#15325?). so I'd like to merge this as is once. In case when it turns out to be too flaky, then I'd like to apply #1490 (comment).

2023-02-04T08:24:48.8673980Z �[38;5;0m[HTTP]�[0m �[37m-->�[39m �[37mPOST�[39m �[37m/session/5f4c9435-1dc6-488e-975c-b8343b671e38/url�[39m
2023-02-04T08:24:48.8776540Z �[38;5;0m[HTTP]�[0m {"url":"http://127.0.0.1:4567/test/guinea-pig-app-banner"}
2023-02-04T08:24:48.8879820Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m Calling AppiumDriver.setUrl() with args: ["http://127.0.0.1:4567/test/guinea-pig-app-banner","5f4c9435-1dc6-488e-975c-b8343b671e38"]
2023-02-04T08:24:48.9009250Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m Executing command 'setUrl'
2023-02-04T08:24:48.9112260Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m Attempting to set url 'http://127.0.0.1:4567/test/guinea-pig-app-banner'
2023-02-04T08:24:48.9216230Z [debug] �[38;5;144m[RemoteDebugger]�[0m Navigating to new URL: 'http://127.0.0.1:4567/test/guinea-pig-app-banner'
2023-02-04T08:24:48.9320870Z [debug] �[38;5;144m[RemoteDebugger]�[0m Waiting for frame navigated message...
2023-02-04T08:24:48.9602620Z [debug] �[38;5;144m[RemoteDebugger]�[0m Sending '_rpc_forwardSocketData:' message to app 'PID:94989', page '1', target 'page-11' (id: 50): 'Page.navigate'
2023-02-04T08:24:48.9706700Z [debug] �[38;5;144m[RemoteDebugger]�[0m Received data response from send (id: 50): '{}'
2023-02-04T08:24:48.9920520Z [debug] �[38;5;144m[RemoteDebugger]�[0m Sending to Web Inspector took 87ms
2023-02-04T08:24:49.0114000Z �[38;5;0m[HTTP]�[0m �[37m-->�[39m �[37mGET�[39m �[37m/test/guinea-pig-app-banner�[39m
2023-02-04T08:24:49.0217710Z �[38;5;0m[HTTP]�[0m {}
2023-02-04T08:24:49.0323720Z [debug] �[38;5;0m[HTTP]�[0m Sending guinea pig response with params: {"throwError":"","serverTime":"2023-02-04T08:24:48.357Z","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15","comment":"None"}
2023-02-04T08:24:49.0428710Z �[38;5;0m[HTTP]�[0m <-- GET /test/guinea-pig-app-banner 200 13 ms - 3537
2023-02-04T08:24:49.0535880Z �[38;5;0m[HTTP]�[0m 
2023-02-04T08:24:49.0641980Z [debug] �[38;5;144m[RemoteDebugger]�[0m Frame navigated in 174ms from: 'Page.frameNavigated' event
2023-02-04T08:24:49.0757360Z �[38;5;0m[HTTP]�[0m <-- GET /js/jquery.min.js 304 3 ms - -
2023-02-04T08:24:49.0859770Z �[38;5;0m[HTTP]�[0m 
2023-02-04T08:24:49.4077070Z [debug] �[38;5;144m[RemoteDebugger]�[0m Waiting for dom...
2023-02-04T08:24:49.4179720Z [debug] �[38;5;144m[RemoteDebugger]�[0m Page loaded, verifying whether ready
2023-02-04T08:24:49.5858390Z [debug] �[38;5;144m[RemoteDebugger]�[0m Application 'PID:94989' disconnected. Removing from app dictionary.
2023-02-04T08:24:49.6240650Z [debug] �[38;5;144m[RemoteDebugger]�[0m Current app is 'PID:94989'
2023-02-04T08:24:49.6255170Z [debug] �[38;5;144m[RemoteDebugger]�[0m No longer have app id. Attempting to find new one.
2023-02-04T08:24:49.9062680Z [debug] �[38;5;144m[RemoteDebugger]�[0m Not connected to an application. Ignoring page load
2023-02-04T08:24:49.9162990Z [debug] �[38;5;144m[RemoteDebugger]�[0m Page load completed in 503ms
2023-02-04T08:24:49.9270150Z [debug] �[38;5;144m[RemoteDebugger]�[0m Sending '_rpc_forwardSocketData:' message, page '1' (id: 52): 'Console.enable'
2023-02-04T08:24:49.9370650Z [debug] �[38;5;144m[RemoteDebugger]�[0m Sending to Web Inspector took 29ms
2023-02-04T08:24:49.9473940Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m Encountered internal error running command: TypeError: Cannot read properties of undefined (reading 'bplistOverride')
2023-02-04T08:24:49.9651960Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at toEntries (/Users/runner/work/1/s/node_modules/bplist-creator/bplistCreator.js:324:13)
2023-02-04T08:24:49.9768830Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at /Users/runner/work/1/s/node_modules/bplist-creator/bplistCreator.js:430:22
2023-02-04T08:24:49.9873640Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at Array.forEach (<anonymous>)
2023-02-04T08:24:49.9977780Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at toEntriesObject (/Users/runner/work/1/s/node_modules/bplist-creator/bplistCreator.js:429:21)
2023-02-04T08:24:50.0082160Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at toEntries (/Users/runner/work/1/s/node_modules/bplist-creator/bplistCreator.js:360:14)
2023-02-04T08:24:50.0088850Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at /Users/runner/work/1/s/node_modules/bplist-creator/bplistCreator.js:430:22
2023-02-04T08:24:50.0208870Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at Array.forEach (<anonymous>)
2023-02-04T08:24:50.0382300Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at toEntriesObject (/Users/runner/work/1/s/node_modules/bplist-creator/bplistCreator.js:429:21)
2023-02-04T08:24:50.0490520Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at toEntries (/Users/runner/work/1/s/node_modules/bplist-creator/bplistCreator.js:360:14)
2023-02-04T08:24:50.0605200Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at module.exports (/Users/runner/work/1/s/node_modules/bplist-creator/bplistCreator.js:25:17)
2023-02-04T08:24:50.0720410Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at Object.createBinaryPlist (/Users/runner/work/1/s/node_modules/@appium/support/lib/plist.js:103:22)
2023-02-04T08:24:50.0831920Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at PlistServiceEncoder._encode (/Users/runner/work/1/s/node_modules/appium-ios-device/lib/plist-service/transformer/plist-service-encoder.js:18:31)
2023-02-04T08:24:50.0936670Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at PlistServiceEncoder._transform (/Users/runner/work/1/s/node_modules/appium-ios-device/lib/plist-service/transformer/plist-service-encoder.js:13:20)
2023-02-04T08:24:50.1037520Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at PlistServiceEncoder.Transform._write (node:internal/streams/transform:205:23)
2023-02-04T08:24:50.1156400Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at writeOrBuffer (node:internal/streams/writable:391:12)
2023-02-04T08:24:50.1261580Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at _write (node:internal/streams/writable:332:10)
2023-02-04T08:24:50.1367780Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at PlistServiceEncoder.write (node:internal/streams/writable:336:10)
2023-02-04T08:24:50.1471610Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at WebInspectorService.sendMessage (/Users/runner/work/1/s/node_modules/appium-ios-device/lib/webinspector/index.js:152:19)
2023-02-04T08:24:50.1583550Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at /Users/runner/work/1/s/node_modules/appium-remote-debugger/lib/rpc/rpc-client-simulator.js:131:20
2023-02-04T08:24:50.1687380Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at Promise.cancellationExecute [as _execute] (/Users/runner/work/1/s/node_modules/bluebird/js/release/debuggability.js:406:9)
2023-02-04T08:24:50.1787040Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at Promise._resolveFromExecutor (/Users/runner/work/1/s/node_modules/bluebird/js/release/promise.js:518:18)
2023-02-04T08:24:50.1889860Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at new Promise (/Users/runner/work/1/s/node_modules/bluebird/js/release/promise.js:103:10)
2023-02-04T08:24:50.1993550Z �[38;5;0m[HTTP]�[0m <-- POST /session/5f4c9435-1dc6-488e-975c-b8343b671e38/url 500 1676 ms - 696
2023-02-04T08:24:50.2096430Z �[38;5;0m[HTTP]�[0m 
2023-02-04T08:24:50.2198840Z �[90m2023-02-04T08:24:49.937Z�[39m �[33mWARN�[39m �[97mwebdriver�[39m: Request failed with status 500 due to An unknown server-side error occurred while processing the command. Original error: Cannot read properties of undefined (reading 'bplistOverride')
2023-02-04T08:24:50.2337410Z �[90m2023-02-04T08:24:49.937Z�[39m �[96mINFO�[39m �[97mwebdriver�[39m: Retrying 1/1
2023-02-04T08:24:50.2438330Z �[90m2023-02-04T08:24:49.938Z�[39m �[96mINFO�[39m �[97mwebdriver�[39m: [POST] http://127.0.0.1:4567/session/5f4c9435-1dc6-488e-975c-b8343b671e38/url
2023-02-04T08:24:50.2541960Z �[90m2023-02-04T08:24:49.938Z�[39m �[96mINFO�[39m �[97mwebdriver�[39m: �[33mDATA�[39m { url: 'http://127.0.0.1:4567/test/guinea-pig-app-banner' }
2023-02-04T08:24:50.2644210Z �[38;5;0m[HTTP]�[0m �[37m-->�[39m �[37mPOST�[39m �[37m/session/5f4c9435-1dc6-488e-975c-b8343b671e38/url�[39m
2023-02-04T08:24:50.2745130Z �[38;5;0m[HTTP]�[0m {"url":"http://127.0.0.1:4567/test/guinea-pig-app-banner"}
2023-02-04T08:24:50.2849390Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m Calling AppiumDriver.setUrl() with args: ["http://127.0.0.1:4567/test/guinea-pig-app-banner","5f4c9435-1dc6-488e-975c-b8343b671e38"]
2023-02-04T08:24:50.3054000Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m Executing command 'setUrl'
2023-02-04T08:24:50.3157910Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m Attempting to set url 'http://127.0.0.1:4567/test/guinea-pig-app-banner'
2023-02-04T08:24:50.4185710Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m Encountered internal error running command: Error: Missing parameter: appIdKey
2023-02-04T08:24:50.4428300Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at checkParams (/Users/runner/work/1/s/node_modules/appium-remote-debugger/lib/utils.js:174:11)
2023-02-04T08:24:50.4467770Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at RemoteDebugger.navToUrl (/Users/runner/work/1/s/node_modules/appium-remote-debugger/lib/mixins/navigate.js:102:14)
2023-02-04T08:24:50.4851570Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at XCUITestDriver.setUrl (/Users/runner/work/1/s/lib/commands/general.js:230:23)
2023-02-04T08:24:50.4853400Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at commandExecutor (/Users/runner/work/1/s/node_modules/@appium/base-driver/lib/basedriver/driver.js:122:18)
2023-02-04T08:24:50.4961220Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at /Users/runner/work/1/s/node_modules/async-lock/lib/index.js:171:12
2023-02-04T08:24:50.5064880Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at AsyncLock._promiseTry (/Users/runner/work/1/s/node_modules/async-lock/lib/index.js:304:31)
2023-02-04T08:24:50.5169360Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at exec (/Users/runner/work/1/s/node_modules/async-lock/lib/index.js:170:9)
2023-02-04T08:24:50.5272240Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at AsyncLock.acquire (/Users/runner/work/1/s/node_modules/async-lock/lib/index.js:187:3)
2023-02-04T08:24:50.5375430Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at XCUITestDriver.executeCommand (/Users/runner/work/1/s/node_modules/@appium/base-driver/lib/basedriver/driver.js:138:39)
2023-02-04T08:24:50.5478370Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at runMicrotasks (<anonymous>)
2023-02-04T08:24:50.5581200Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at processTicksAndRejections (node:internal/process/task_queues:96:5)
2023-02-04T08:24:50.5687260Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at XCUITestDriver.executeCommand (/Users/runner/work/1/s/lib/driver.js:784:12)
2023-02-04T08:24:50.5821120Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at defaultBehavior (/Users/runner/hostedtoolcache/node/16.19.0/x64/lib/node_modules/appium/lib/appium.js:673:14)
2023-02-04T08:24:50.5924720Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at AppiumDriver.executeWrappedCommand (/Users/runner/hostedtoolcache/node/16.19.0/x64/lib/node_modules/appium/lib/appium.js:763:16)
2023-02-04T08:24:50.6028360Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at AppiumDriver.executeCommand (/Users/runner/hostedtoolcache/node/16.19.0/x64/lib/node_modules/appium/lib/appium.js:685:17)
2023-02-04T08:24:50.6131440Z [debug] �[38;5;32m[XCUITestDriver@d9b1 (5f4c9435)]�[0m     at asyncHandler (/Users/runner/hostedtoolcache/node/16.19.0/x64/lib/node_modules/appium/node_modules/@appium/base-driver/lib/protocol/protocol.js:387:19)
2023-02-04T08:24:50.6247130Z �[90m2023-02-04T08:24:49.978Z�[39m �[31mERROR�[39m �[97mwebdriver�[39m: Request failed with status 500 due to unknown error: An unknown server-side error occurred while processing the command. Original error: Missing parameter: appIdKey
2023-02-04T08:24:50.6350760Z �[38;5;0m[HTTP]�[0m <-- POST /session/5f4c9435-1dc6-488e-975c-b8343b671e38/url 500 33 ms - 626

@KazuCocoa
Copy link
Member

@KazuCocoa
Copy link
Member

thanks for the update. Re-ran a couple of times, then it passed. I have disabled idb tests as preparation failures.

@KazuCocoa KazuCocoa merged commit 12b7ab8 into appium:master Feb 6, 2023
@mwakizaka
Copy link
Contributor Author

I see. thank you for your cooperation!

@mwakizaka mwakizaka deleted the test/safari-nativewebtap branch February 6, 2023 02:49
KazuCocoa added a commit that referenced this pull request Feb 6, 2023
KazuCocoa added a commit that referenced this pull request Feb 6, 2023
github-actions bot pushed a commit that referenced this pull request Feb 6, 2023
## [4.16.12](v4.16.11...v4.16.12) (2023-02-06)

### Bug Fixes

* Update NOTCHED_DEVICE_SIZES for nativeWebTapStrict (part of [#1490](#1490)) ([#1497](#1497)) ([e2bbd94](e2bbd94))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants