Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

All the tests after the first test execution stuck on "Test starting...", then fail #1555

Closed
nzkn opened this issue Jul 19, 2023 · 8 comments
Closed

All the tests after the first test execution stuck on "Test starting...", then fail #1555

nzkn opened this issue Jul 19, 2023 · 8 comments
Assignees
@fylyppo
Labels
waiting for response Waiting for user's response

Comments

@nzkn
Copy link

nzkn commented Jul 19, 2023 •
edited
Loading

Steps to reproduce

  1. Create a project with multiple end-to-end test files
  2. Start iOS Simulator
  3. Run tests with the command:
    patrol test --target integration_test/a_test.dart --target integration_test/b_test.dart --target integration_test/c_test.dart

Actual results

The a_test.dart executes fine. After the first test execution something strange starts happening - tests randomly start and end, then tests stuck on the "Test is starting..." screen, nothing happens after that.The same tests execution works fine when running them with separate commands one after another.

Logs

Logs 
✓ Completed building app with entrypoint test_bundle.dart for iOS simulator (414.4s)
Will uninstall apps before running tests
$ xcrun simctl uninstall B05C9460-2347-4160-AC7E-EC8F16D0F192 one.homes.sand
$ xcrun simctl uninstall B05C9460-2347-4160-AC7E-EC8F16D0F192 one.homes.sand.RunnerUITests.xctrunner
• Running app with entrypoint test_bundle.dart for iOS simulator on simulator iPhone 8...
$ xcodebuild -showsdks -json
Assuming SDK version 16.1 for iphonesimulator
Looking for .xctestrun matching dev_iphonesimulator16.1*.xctestrun at /Users/nkahaniak/Pickett/unite-mobile/build/ios_integ/Build/Products
Found 1 match(es), the first one will be used
Found /Users/nkahaniak/Pickett/unite-mobile/build/ios_integ/Build/Products/dev_iphonesimulator16.1-x86_64.xctestrun
$ xcodebuild test-without-building -xctestrun /Users/nkahaniak/Pickett/unite-mobile/build/ios_integ/Build/Products/dev_iphonesimulator16.1-x86_64.xctestrun -only-testing RunnerUITests -destination platform=iOS Simulator,name=iPhone 8
        Command line invocation:
            /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun /Users/nkahaniak/Pickett/unite-mobile/build/ios_integ/Build/Products/dev_iphonesimulator16.1-x86_64.xctestrun -only-testing RunnerUITests -destination "platform=iOS Simulator,name=iPhone 8"
        
        User defaults from command line:
            IDEPackageSupportUseBuiltinSCM = YES
        
        2023-07-19 16:43:34.713 xcodebuild[30713:14338484] [MT] IDETestOperationsObserverDebug: 441.274 elapsed -- Testing started completed.
        2023-07-19 16:43:34.713 xcodebuild[30713:14338484] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
        2023-07-19 16:43:34.713 xcodebuild[30713:14338484] [MT] IDETestOperationsObserverDebug: 441.274 sec, +441.274 sec -- end
        2023-07-19 16:43:34.822 xcodebuild[30713:14338484] [MT] IDEResultKit: Warning: While writing a result bundle to /Users/nkahaniak/Library/Developer/Xcode/DerivedData/Runner-chrmpeedzampjkejhdqkitzqvdgd/Logs/Test/Test-dev-2023.07.19_16-36-13-+0300.xcresult, not all contents have been imported, as the Staging directory still contains the following files/directories: ["1_Test"]
        
        Test session results, code coverage, and logs:
                /Users/nkahaniak/Library/Developer/Xcode/DerivedData/Runner-chrmpeedzampjkejhdqkitzqvdgd/Logs/Test/Test-dev-2023.07.19_16-36-13-+0300.xcresult
        
        Failing tests:
                -[RunnerUITests activateStrategyTest]
                -[RunnerUITests createStrategyTest]
                -[RunnerUITests editStrategyTest]
        
        ** TEST EXECUTE FAILED **
        
        Testing started
        Test suite 'RunnerUITests' started on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)'
        Test case '-[RunnerUITests aAuthenticationTest]' passed on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)' (12.138 seconds)
        Test case '-[RunnerUITests activateStrategyTest]' failed on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)' (73.699 seconds)
        Test case '-[RunnerUITests createStrategyTest]' failed on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)' (133.516 seconds)
        Test case '-[RunnerUITests editStrategyTest]' failed on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)' (192.453 seconds)
✗ Failed to execute tests of app with entrypoint test_bundle.dart for iOS simulator on simulator iPhone 8 (xcodebuild exited with code 65) (444.3s)
Error: xcodebuild exited with code 65
#0      throwToolExit (package:patrol_cli/src/base/exceptions.dart:7:3)
#1      IOSTestBackend.execute.<anonymous closure> (package:patrol_cli/src/ios/ios_test_backend.dart:181:9)
<asynchronous suspension>
#2      DisposeScope.run (package:dispose_scope/src/dispose_scope.dart:46:7)
<asynchronous suspension>
#3      IOSTestBackend.execute (package:patrol_cli/src/ios/ios_test_backend.dart:147:5)
<asynchronous suspension>
#4      TestCommand._execute (package:patrol_cli/src/commands/test.dart:269:7)
<asynchronous suspension>
#5      TestCommand.run (package:patrol_cli/src/commands/test.dart:169:23)
<asynchronous suspension>
#6      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#7      PatrolCommandRunner.runCommand (package:patrol_cli/src/runner/patrol_command_runner.dart:310:18)
<asynchronous suspension>
#8      PatrolCommandRunner.run (package:patrol_cli/src/runner/patrol_command_runner.dart:254:18)
<asynchronous suspension>
#9      patrolCommandRunner (package:patrol_cli/src/runner/patrol_command_runner.dart:72:20)
<asynchronous suspension>
#10     main (file:///Users/nkahaniak/.pub-cache/hosted/pub.dev/patrol_cli-2.0.4/bin/main.dart:6:20)
<asynchronous suspension>

See the logs above to learn what happened. Also consider running with --verbose. If the logs still aren't useful, then it's a bug - please report it.
$ xcrun simctl uninstall B05C9460-2347-4160-AC7E-EC8F16D0F192 one.homes.sand-unite
$ xcrun simctl uninstall B05C9460-2347-4160-AC7E-EC8F16D0F192 one.homes.sand-unite.RunnerUITests.xctrunner

Patrol version

patrol: 2.0.4

Patrol Doctor output

Patrol Doctor output 
You're using Patrol CLI 2.0, which has breaking changes.
Read the migration guide at https://patrol.leancode.co/v2.
Disable this warning by setting the PATROL_MIGRATED environment variable.

Patrol CLI version: 2.0.4
Program adb found in /Users/nkahaniak/Library/Android/sdk/platform-tools/adb
Env var $ANDROID_HOME set to /Users/nkahaniak/Library/Android/sdk
Program xcodebuild found in /usr/bin/xcodebuild
Program ideviceinstaller found in /usr/local/bin/ideviceinstaller

Flutter Doctor output

Flutter Doctor output 
Doctor summary (to see all details, run flutter doctor -v):
[!] Flutter (Channel unknown, 3.10.2, on macOS 13.4.1 22F82 darwin-x64, locale en-UA)
    ! Flutter version 3.10.2 on channel unknown at /Users/nkahaniak/Downloads/flutter
      Currently on an unknown channel. Run `flutter channel` to switch to an official channel.
      If that doesn't fix the issue, reinstall Flutter by following instructions at https://flutter.dev/docs/get-started/install.
    ! Unknown upstream repository.
      Reinstall Flutter by following instructions at https://flutter.dev/docs/get-started/install.
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[✓] Xcode - develop for iOS and macOS (Xcode 14.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.2)
[✓] IntelliJ IDEA Ultimate Edition (version 2022.2.1)
[✓] VS Code (version 1.65.2)
[✓] Connected device (3 available)
[✓] Network resources
@jBorkowska jBorkowska assigned jBorkowska and fylyppo and unassigned jBorkowska Jul 19, 2023
@fylyppo
Copy link
Collaborator

fylyppo commented Jul 19, 2023 •
edited
Loading

Hi, two questions:

  • can you confirm that you have disabled parallel execution properly? Here's how to do it
  • is this happening only on iOS simulator?

@nzkn
Copy link
Author

nzkn commented Jul 20, 2023

@fylyppo I can confirm that the issue occurs with parallel execution disabled. I did not test with a real device, only with Simulator

@fylyppo
Copy link
Collaborator

fylyppo commented Jul 20, 2023

And do you still have similar logs like:

Testing started
        Test suite 'RunnerUITests' started on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)'
        Test case '-[RunnerUITests aAuthenticationTest]' passed on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)' (12.138 seconds)
        Test case '-[RunnerUITests activateStrategyTest]' failed on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)' (73.699 seconds)
        Test case '-[RunnerUITests createStrategyTest]' failed on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)' (133.516 seconds)
        Test case '-[RunnerUITests editStrategyTest]' failed on 'Clone 1 of iPhone 8 - RunnerUITests-Runner (30968)' (192.453 seconds)
✗ Failed to execute tests of app with entrypoint test_bundle.dart for iOS simulator on simulator iPhone 8 (xcodebuild exited with code 65) (444.3s)

?
Clone 1 of iPhone 8 means that it created a clone exactly for parallel execution. Maybe you have flavors and you changed it in a different one than you use for testing.

@nzkn
Copy link
Author

nzkn commented Jul 21, 2023

I do have flavors in the app, but parallel execution is disabled in for all the schemes. I will double test it today and let you know if the issue is still reproducible

@nzkn
Copy link
Author

nzkn commented Jul 24, 2023 •
edited
Loading

I can confirm that issue is still reproducible, the parallel execution is disabled
Screenshot 2023-07-24 at 11 41 35
Screenshot 2023-07-24 at 11 41 17

@fylyppo
Copy link
Collaborator

fylyppo commented Jul 25, 2023

Can you show me the code of two tests where its reproducible?

@nzkn
Copy link
Author

nzkn commented Jul 25, 2023

Will set up a project with minimal reproducible code later today and send you the link

@jBorkowska jBorkowska added the waiting for response Waiting for user's response label Jul 26, 2023
@jBorkowska jBorkowska mentioned this issue Jul 27, 2023
Closed
@bartekpacia
Copy link
Contributor

Closing - lack of response.

This is very likely a problem with setup not being done correctly.

@bartekpacia bartekpacia closed this as not planned Won't fix, can't repro, duplicate, stale Sep 14, 2023
@bartekpacia bartekpacia pinned this issue Sep 14, 2023
@bartekpacia bartekpacia unpinned this issue Sep 14, 2023
@leancodepl leancodepl locked and limited conversation to collaborators Sep 14, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@fylyppo fylyppo

Labels
waiting for response Waiting for user's response
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bartekpacia @fylyppo @nzkn @jBorkowska