iOS Simulators no longer launching #9591
Comments
|
Hey @ds8k, |
|
@shamil-mubarakshin @ds8k I believe I have the same problem. You can have a look on https://github.com/kirillzyusko/react-native-keyboard-controller repository and iOS e2e action/failed jobs. I am observing such problem from Mar 19. Basically some tests are failing because they exceed a timeout of 2 minutes (120000ms). I've tried to increase timeout to 13 mins but still tests are very flaky. On my local machine everything works fine. Another my observation - after 1 or 2 failing tests everything becomes to work faster again and subsequent tests can be executed successfully. As a temporary workaround I'm using |
|
@shamil-mubarakshin Since the week before last week, everything starts to fail because of this error: The destination id we set is the "iPhone 14 pro" device id. Here is a link to our pipeline: https://identitydivision.visualstudio.com/IDDP/_build/results?buildId=1278003&view=logs&j=f71b8be3-a86a-5c85-a350-6d71c1f97c14&t=c67dfeb8-f426-5ec6-237b-6331a4e8e3ff This is blocking our release currently. Could you please help with high priority? |
|
Same issue here, using |
|
@shamil-mubarakshin I'm seeing this as well - although in our case, the iOS simulator is starting, but it takes a lot longer than it once did. We're only testing using macOS-14 runner. As recently as Monday, our full iOS test suite took 3 minutes to complete, including building the app, starting the simulator, and running the full test suite. At this point, the test was running on macOS 14.2.1, with an image that was tagged 20240116.1 (although the hyperlink in the log doesn't point at a release - the tag exists, but the release doesn't appear to exist any more, and the hyperlinks are broken). As of the update, it can take 10 minutes or longer to start the iOS simulator (example 1, example 2). The slowdown is then causing other problems that appear to cause the test suite to crash (fairly consistently at the same place) after about 20% of the tests. These two tests are running on macOS 14.4, with an image tagged 20240325.1. I've also seen this problem with macOS 14.4.1, with the image tagged 20240402.2 (example 1, example 2); on the more recent image, it appears that the simulator might be slightly faster, but still much slower than before. One manifestation of the slowdown can be observed around line 270 of the of the "Test App" logs. In the old configuration, there would be 2-3 repeats of the "... still waiting" indicator (printed 10 seconds apart). In the new configuration, there are 20 or more iterations of "... still waiting". The "Testbed (macOS-arm64)" tests take the same time to run, so the issue doesn't appear to be a general problem with the macOS-14 runner; it's specific to the iOS simulator running on the macOS-14 runner. The test suite is using a tool called briefcase to run the test suite; internally, briefcase is starting the simulator using: where 9531516F-3EE1-4463-AFCF-9BE73B199E7B is the UDID of the iPhone SE (3rd generation) simulator running iOS 17.4. On the older (successful) runs, the UDID is E33AE7FB-692F-4481-B3CB-23A7BE8F17DE, as the simulator is running iOS 17.2. I have also tried running the test suite on the new configuration, but selecting an iOS 17.2 simulator, and running after explicitly selecting Xcode 15.2; neither approach restores the historical simulator speed. |
|
Hello. We are encountering the same problem. We had a similar problem here with Xcode 15.0. |
|
@freakboy3742 have you tried to run simulator 16.x (for example 16.4)? I've heard it can give back full speed 👀 For me it looks like new iOS 17 simulators do a heavy CPU work for a first run. I thought to cache simulator and restore for all subsequent runs, but on my local machine the size of simulator folder is about 4 GB and with limit of 10GB cache it's very unreasonable to try to cache it 😔 @Kyome22 I used |
|
@kirillzyusko Thanks for that suggestion - it appears you're correct. Reverting to Xcode 14.3.1 and an iOS 16.4 simulator restores the previous testing speed. That isn't a long term fix, if only because I'd predict Xcode 14.3.1 will be dropped from the runner image in the not-too-distant-future - but it at least it unsticks my immediate CI issue. |
|
@freakboy3742 you can add iOS 16.4 in XCode 15 too, I believe. But you'll need to define additional steps 👀 But I totally agree with you - it should be fixed on image level somehow 👍 |
|
@kirillzyusko I did try that; but it looks like the 16.4 API is only included with the older Xcode. It's probably possible to download, but pulling a 7GB image in CI is a bit of a tall order. |
|
@freakboy3742 direct downloading will take ~5 mins, you're right. But you can also create a symbolic link #8402 (comment) I haven't tried it and again you're right - if you go to symbolic link approach then you still depends on XCode 14.3.1 (which is not good). And downloading 7GB and further installation is also an expensive operation. |
|
Hey @sssuourabh! |
|
It appears that the |
|
...or maybe I've spoken too soon - this job passed, but when I tried to apply the same change to another PR, it fails. |
|
Same thing happening with me. Sometimes it passes and next Time it fails.
Sourabh Shekhar Singh
+91-7838135034
https://www.linkedin.com/in/sourabhshekhar/
…On Fri, 12 Apr 2024 at 1:21 PM, Russell Keith-Magee < ***@***.***> wrote:
...or maybe I've spoken too soon - this job passed
<https://github.com/beeware/toga/actions/runs/8658180101/job/23741567362>,
but when I tried to apply the same change to another PR, it fails
<https://github.com/beeware/toga/actions/runs/8658471154/job/23743197182>.
—
Reply to this email directly, view it on GitHub
<#9591 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAWIRLLDUGUKKPJFPUO44STY46HAJAVCNFSM6AAAAABFOVDOUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRGIYTANJVHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
We are running into the same issues with our build infrastructure using GitHub hosted runners: https://github.com/StanfordBDHG/SwiftPackageTemplate/actions/runs/8680699610/job/23803319639 Some of the tests timeout due to missing resources and sometimes the simulator does not even start, e.g., using the |
|
Just adding our anecdata to this, we have a workflow that first boots a named simulator with the following command: DEVICE_TYPE_ID="com.apple.CoreSimulator.SimDeviceType.iPhone-14-Pro-Max"
RUNTIME_ID="com.apple.CoreSimulator.SimRuntime.iOS-17-2"
SIMULATOR_NAME="simulator name goes here"
xcrun simctl bootstatus $(xcrun simctl create $NAME $DEVICE_TYPE_ID $RUNTIME_ID) -b...then once the simulator is finished booting, loads a specific URL into Mobile Safari with the following command: URL="url goes here"
xcrun simctl openurl booted $URLWe have observed the following timings:
(Note: The "Open URL time" for the last 2 rows is "> 20s" because 20s is the arbitrary timeout we have on this command; hence those workflow runs failed and we cannot confirm how long it actually takes to load the URL) (Also worth noting that on my local machine (M1 Macbook Pro), the |
|
The simulator may be able to start stably with a combination of Xcode 15.1 and iOS 17.2. |
|
The 20240415.6 image hasn't resolved the issue. Is there any possibility that someone from Github might respond? I've been advised via a ticket raised on an enterprise support contract that I should monitor the public ticket for updates, which would be great, if only there were updates of any kind from Github. |
|
Hey @freakboy3742. issue is currently being investigated. There is an issue with ios-17.4 simulator and CPU consumption. Checking if it is relevant and there is a command line equivalent to disabling Thread Performance Checker. |
|
I don't know, what happened, but today I discovered that tests are executing fast again 🚀 (under 5 minutes). |
|
Confirming the same. To further add to my table from above:
It appears that the latest image version has almost (but not quite) returned to the same level of performance as the 20240116.1 image version. It would be great to understand what the change was that resulted in this improvement. |
Agree. From what I observed - it's again not possible to programmatically set time for iOS simulator (was a known issue startig from XCode 14). So just curious what actually was changed in latest runner version 🤔 |
|
The iOS 17.5 simulator that is the default on 20240422.3 definitely appears to run better in my experience, but it's still significantly slower than before - my test suite used to run to completion in 3-4 minutes; I'm now getting a test timeout 15 mins. The "Time to first test" has almost doubled - previously the suite would build and start running in <2 minutes; it now takes 4-5 minutes to do the same build. The test appears to be stalling on one particular test; it takes almost 8 minutes to get to that point in the suite. |
|
still facing the issue |
|
Found also a similar problem: The |
I think this should be filed as a separate issue. |
|
This issue is still happening. It appears that maybe the simulator is running, but all of my tests are still failing. The same tests pass locally on MacOS 14 with Xcode 15.3 and on the macos-12 image using Xcode 14. |
|
Hello! We understand there's been performance degradation in the simulator, primarily due to the latest Xcode version rather than our runner images.While we've noticed poor performance on our standard runners, we've observed better results on larger macOS runners. If this is crucial for your needs, we recommend switching to the larger macOS runners. |
|
@vinaynaikr08 Unfortunately, switching to larger runners isn't a viable option, as my usage is for an Open Source project using the free tier plan, without the budget to support the commercial runner tiers. |
|
I was able to solve the issue by doing this https://www.youtube.com/watch?v=9bDDWNoibIk&t=12s&ab_channel=SgtSam |
|
@Sameer-Ijaz this is not relevant, you can't clean up temporary files in a simulator, and it wouldn't make sense to since each bootup should be from a clean image. |
…rs timing out according to actions/runner-images#9591 (comment)
…simulators timing out according to actions/runner-images#9591 (comment)" This reverts commit 0f08a42. We can't use larger runners in this repo.
Description
Something changed within the past couple weeks that is preventing any iOS simulators from launching successfully, which is breaking our E2E Detox tests. They used to successfully run, but since Xcode 15.3 was released it appears the entire process has broken.
I've tried running tests on macos-13 and macos-14 with the same results.
Platforms affected
Runner images affected
Image version and build link
Image: macos-14-arm64
Version: 20240219.1
Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/2024021[9](https://github.com/axioscode/axios-app/actions/runs/8475906614/job/23224707831#step:1:10).1/images/macos/macos-14-arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20240219.1
Is it regression?
https://github.com/axioscode/axios-app/actions/runs/8209405392/job/22454863523
Expected behavior
Detox tests are able to launch the designated simulator and successfully run
Actual behavior
iOS Simulator never launches and tests time out
Repro steps
The text was updated successfully, but these errors were encountered: