iOS Simulator e2e tests are extremely flaky on GH Actions

[piotruela]

The tests that we run on the iOS Simulator in the GitHub workflow keep failing.
The main cause seems to be this problem:
iOS Simulators no longer launching.
When I try to run tests on iOS Simulator with Xcode > 15.0 or/and iOS SDK > 17.0 the process is so slow that it easily exceeds timeout duration (and also many tests fails randomly).
I tried to use previous versions of macOS/Xcode/iOS and so far the best configuration was [macOS-14, Xcode 14.3.1, iOS 16.4]. By 'the best' I mean that it works quite fast but still, many tests fail randomly without a clear reason.
When running the same tests locally everything work as expected.

[mrizzello-compass]

I'm experiencing the same problem. My tests all pass locally but when in GA with the same configurations as local, I get failures. I'm seeing a lot of this:

2024-08-12 15:57:17.371548+0000 RunnerUITests-Runner[26672:91503] [connection] nw_socket_handle_socket_event [C3.1.1:2] Socket SO_ERROR [61: Connection refused] 2024-08-12 15:57:17.377934+0000 RunnerUITests-Runner[26672:91503] [connection] nw_endpoint_flow_failed_with_error [C3.1.1 '00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01'.8082 in_progress socket-flow (satisfied (Path is satisfied), interface: lo0)] already failing, returning 2024-08-12 15:57:17.382967+0000 RunnerUITests-Runner[26672:91503] [connection] nw_socket_handle_socket_event [C3.1.2:2] Socket SO_ERROR [61: Connection refused] 2024-08-12 15:57:17.385492+0000 RunnerUITests-Runner[26672:91503] [connection] nw_endpoint_flow_failed_with_error [C3.1.2 16777343:8082 in_progress socket-flow (satisfied (Path is satisfied), interface: lo0)] already failing, returning 2024-08-12 15:57:17.405939+0000 RunnerUITests-Runner[26672:91503] [connection] nw_endpoint_flow_failed_with_error [C3.1.2 16777343:8082 cancelled socket-flow ((null))] already failing, returning 2024-08-12 15:57:17.408596+0000 RunnerUITests-Runner[26672:91503] [Default] Connection 3: received failure notification 2024-08-12 15:57:17.416671+0000 RunnerUITests-Runner[26672:91503] [Default] Connection 3: failed to connect 1:61, reason -1 2024-08-12 15:57:17.418293+0000 RunnerUITests-Runner[26672:91503] [Default] Connection 3: encountered error(1:61) 2024-08-12 15:57:17.430618+0000 RunnerUITests-Runner[26672:88814] [Default] Task <1F7007BE-B3FB-4AAE-8BA6-FD5E090770B1>.<1> HTTP load failed, 0/0 bytes (error code: -1004 [1:61]) 2024-08-12 15:57:17.465546+0000 RunnerUITests-Runner[26672:88814] [Default] Task <1F7007BE-B3FB-4AAE-8BA6-FD5E090770B1>.<1> finished with error [-1004] Error Domain=NSURLErrorDomain Code=-1004 "Could not connect to the server." UserInfo={_kCFStreamErrorCodeKey=61, NSUnderlyingError=0x600003c096e0 {Error Domain=kCFErrorDomainCFNetwork Code=-1004 "(null)" UserInfo={_NSURLErrorNWPathKey=satisfied (Path is satisfied), interface: en0, _kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <1F7007BE-B3FB-4AAE-8BA6-FD5E090770B1>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=( "LocalDataTask <1F7007BE-B3FB-4AAE-8BA6-FD5E090770B1>.<1>" ), NSLocalizedDescription=Could not connect to the server., NSErrorFailingURLStringKey=http://localhost:8082/runDartTest, NSErrorFailingURLKey=http://localhost:8082/runDartTest, _kCFStreamErrorDomainKey=1}

[piotruela]

@mrizzello-compass We decided to move our internal e2e tests to other CI until this problem on Github will be solved. (More specifically, we moved them to Bitrise which is much more stable when it comes to running patrol tests on iOS Simulators).

[mrizzello-compass]

@piotruela i'm still seeing this error in bitrise, but they all pass locally

2024-08-29 20:28:25.494309+0000 RunnerUITests-Runner[22104:70810] PatrolAppServiceClient.runDartTest(tests.account_settings_test Verify whether an SSO user does not have the update password option in account settings page) 2024-08-29 20:28:25.527622+0000 RunnerUITests-Runner[22104:72963] [connection] nw_socket_handle_socket_event [C3.1.1:2] Socket SO_ERROR [61: Connection refused] 2024-08-29 20:28:25.530066+0000 RunnerUITests-Runner[22104:72963] [connection] nw_socket_handle_socket_event [C3.1.2:2] Socket SO_ERROR [61: Connection refused] 2024-08-29 20:28:25.542646+0000 RunnerUITests-Runner[22104:72963] Connection 3: received failure notification 2024-08-29 20:28:25.548864+0000 RunnerUITests-Runner[22104:72963] Connection 3: failed to connect 1:61, reason -1 2024-08-29 20:28:25.549815+0000 RunnerUITests-Runner[22104:72963] Connection 3: encountered error(1:61) 2024-08-29 20:28:25.555861+0000 RunnerUITests-Runner[22104:73407] Task <7E1CE2B5-53C9-45A4-AAF7-6B2F8F1819DC>.<1> HTTP load failed, 0/0 bytes (error code: -1004 [1:61]) 2024-08-29 20:28:25.578452+0000 RunnerUITests-Runner[22104:71043] Task <7E1CE2B5-53C9-45A4-AAF7-6B2F8F1819DC>.<1> finished with error [-1004] Error Domain=NSURLErrorDomain Code=-1004 "Could not connect to the server." UserInfo={_kCFStreamErrorCodeKey=61, NSUnderlyingError=0x6000027db510 {Error Domain=kCFErrorDomainCFNetwork Code=-1004 "(null)" UserInfo={_NSURLErrorNWPathKey=satisfied (Path is satisfied), interface: en0, _kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <7E1CE2B5-53C9-45A4-AAF7-6B2F8F1819DC>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=( "LocalDataTask <7E1CE2B5-53C9-45A4-AAF7-6B2F8F1819DC>.<1>" ), NSLocalizedDescription=Could not connect to the server., NSErrorFailingURLStringKey=http://localhost:8082/runDartTest, NSErrorFailingURLKey=http://localhost:8082/runDartTest, _kCFStreamErrorDomainKey=1} 2024-08-29 20:28:25.664606+0000 RunnerUITests-Runner[22104:71043] runDartTest("tests.account_settings_test Verify whether an SSO user does not have the update password option in account settings page"): call finished, test result: CRASHED 2024-08-29 20:28:25.773770+0000 RunnerUITests-Runner[22104:71043] PatrolServer: INFO: App reported that it is ready /Users/vagrant/git/ios/RunnerUITests/RunnerUITests.m:5: error: -[RunnerUITests tests.account_settings_test Verify whether an SSO user does not have the update password option in account settings page] : ((passed) is true) failed - (no details - app likely crashed) t = 16.85s Tear Down Test Case '-[RunnerUITests tests.account_settings_test Verify whether an SSO user does not have the update password option in account settings page]' failed (17.615 seconds).

[chiphan-quickcare]

Use patrol 3.0.9 fix the problems

[mrizzello-compass]

Use patrol 3.0.9 fix the problems

Do you mean 3.9.0?

[nursultan-akhmetzhanov]

Has this issue been resolved? I am experiencing the same problem on Codemagic, 80-90% of my tests fail.

[karim-cardeo]

I am experiencing a similar issue on Codemagic, when trying to run my tests on an iPhone 16 Pro simulator. Upgrading to the latest version of patrol as suggested by @chiphan-quickcare did not help.