Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OSX machines are de-provisioned during CI / PR runs leading to failures #34472

Closed
jaredpar opened this issue Apr 2, 2020 · 16 comments
Closed

OSX machines are de-provisioned during CI / PR runs leading to failures #34472

jaredpar opened this issue Apr 2, 2020 · 16 comments
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' blocking-official-build
Milestone
Future

Comments

@jaredpar
Copy link
Member

jaredpar commented Apr 2, 2020 •
edited
Loading

Runfo Tracking Issue: OSX deprovision in runtime

Definition Build Kind Job Name
runtime 1288514 PR 52695 CoreCLR Product Build OSX arm64 release
runtime 1284973 PR 57116 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1284614 PR 57076 Build MacCatalyst x64 Debug AllSubsets_Mono
runtime 1266365 PR 56406 Libraries Test Run release mono Linux x64 Debug
runtime 1266272 PR 56502 CoreCLR Product Build OSX x64 release
runtime 1266203 PR 56398 Mono Product Build OSX x64 release
runtime 1265838 PR 56537 CoreCLR Product Build OSX x64 checked
runtime 1265709 PR 56531 Mono crossaot Product Build OSX x64 release
runtime 1264483 PR 56431 Mono iOS AOT offsets
runtime 1264469 PR 56411 Build OSX x64 Release AllSubsets_Mono_LLVMJIT
runtime 1264447 PR 56490 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1264431 PR 56486 Build tvOSSimulator x64 Debug AllSubsets_Mono
runtime 1264431 PR 56486 Build MacCatalyst arm64 Debug AllSubsets_Mono
runtime 1263889 PR 56471 CoreCLR Product Build OSX arm64 checked
runtime 1263861 PR 56411 Build MacCatalyst arm64 Debug AllSubsets_Mono
runtime 1263853 PR 56316 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1262979 PR 56434 Installer Build and Test coreclr OSX_x64 Release
runtime 1246278 PR 55939 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1246251 PR 55925 Mono crossaot Product Build OSX x64 release

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
0 2 18
@jaredpar jaredpar added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Apr 2, 2020
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Apr 2, 2020
@Dotnet-GitSync-Bot
Copy link
Collaborator

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

@jaredpar
Copy link
Member Author

jaredpar commented Apr 2, 2020

FYI this is not runtime specific. Hitting other repos as well

Definition Build Kind Timeline Record
runtime 585548 Rolling Mono Product Build iOS arm64 debug
runtime 585548 Rolling Mono LLVM Product Build OSX x64 debug
runtime 585497 PR #33701 Installer Build and Test OSX_x64 Debug
runtime 585541 PR #34229 Mono LLVM Product Build OSX x64 debug
coreclr 585166 Rolling CoreCLR Pri1 Test Run OSX x64 checked
MachineLearning-CI 584884 PR dotnet/machinelearning#4925 MacOS_x64_NetCoreApp21 Release_Build

Evaluated 500 builds
Impacted 5 bulids
Impacted 6 jobs

@jkotas jkotas mentioned this issue Apr 4, 2020
Merged
@jaredpar jaredpar mentioned this issue Apr 9, 2020
Closed
2 tasks
@jaredpar jaredpar mentioned this issue Apr 21, 2020
Open
@dougbu
Copy link
Member

dougbu commented Apr 21, 2020

This is also tracked at dotnet/core-eng#9448. Correct?

@MattGal
Copy link
Member

MattGal commented Apr 21, 2020

This is also tracked at dotnet/core-eng#9448. Correct?

Correct.

@jashook jashook mentioned this issue Apr 23, 2020
Closed
@EgorBo EgorBo mentioned this issue May 6, 2020
Merged
@MihaZupan MihaZupan mentioned this issue May 14, 2020
Merged
@AraHaan
Copy link
Member

AraHaan commented May 19, 2020

aww that link 404d for me.

@MattGal
Copy link
Member

MattGal commented May 19, 2020

aww that link 404d for me.

Private repo; I can publicly tell you however that the problem continues to be investigated and seems to be at the MacOS-VM-provider level.

@AraHaan
Copy link
Member

AraHaan commented May 19, 2020

Seems on my pr 36694 that this might be a cause when I read this: https://helix.dot.net/api/2019-06-17/jobs/b0b1c468-c4c5-4f2d-a2a6-3c831ed6ef20/workitems/System.Net.Sockets.Tests/console

@MattGal
Copy link
Member

MattGal commented May 19, 2020

Seems on my pr 36694 that this might be a cause when I read this: https://helix.dot.net/api/2019-06-17/jobs/b0b1c468-c4c5-4f2d-a2a6-3c831ed6ef20/workitems/System.Net.Sockets.Tests/console

No that log is from a plain old crash on a Helix machine. This issue is specific to hosted Azure Devops build macs, which are in no way connected.

@CoffeeFlux
Copy link
Contributor

Because of the AzDO limitation where you can't rerun individual lanes until all of them are completed, this issue is particularly painful when trying to land a PR :(

@layomia layomia mentioned this issue May 22, 2020
Merged
@wfurt wfurt mentioned this issue May 27, 2020
Merged
@dougbu dougbu mentioned this issue May 30, 2020
Merged
@MattGal MattGal mentioned this issue Jun 2, 2020
Merged
@wfurt wfurt mentioned this issue Jun 16, 2020
Merged
@ViktorHofer ViktorHofer removed the untriaged New issue has not been triaged by the area owner label Jul 12, 2020
@ViktorHofer ViktorHofer added this to the 5.0.0 milestone Jul 12, 2020
@myblindy
Copy link

Any idea why my https://dev.azure.com/dnceng/public/_build/results?buildId=726102 build was included in this? It didn't fail any tests or build steps as far as I can tell.

@MattGal
Copy link
Member

MattGal commented Jul 13, 2020

Any idea why my https://dev.azure.com/dnceng/public/_build/results?buildId=726102 build was included in this? It didn't fail any tests or build steps as far as I can tell.

It does on Attempt #1 :
image

So, if you didn't click retry, some nice person did for you :)

@MattGal
Copy link
Member

MattGal commented Jul 13, 2020

Update from my end; chatted with AzDO folks in charge of the hosted macs today, the previous ETA of 7/15 to migrate the hosted agents seems to have not been accurate. I am politely asking for an updated time.

@am11 am11 mentioned this issue Jul 29, 2020
Closed
2 tasks
@ViktorHofer ViktorHofer modified the milestones: 5.0.0, Future Aug 4, 2020
@benaadams benaadams mentioned this issue Aug 11, 2020
Merged
@josalem josalem mentioned this issue Aug 13, 2020
Merged
@directhex
Copy link
Contributor

Seemingly still a thing https://dev.azure.com/dnceng/internal/_build/results?buildId=810664&view=logs&j=04b1677b-aa91-548a-e789-89c3d998d0f7

@MattGal
Copy link
Member

MattGal commented Sep 11, 2020 •
edited
Loading

Seemingly still a thing https://dev.azure.com/dnceng/internal/_build/results?buildId=810664&view=logs&j=04b1677b-aa91-548a-e789-89c3d998d0f7

Yes, definitely still a thing (and why the issue is still open). I checked this instance out, and it's definitely on the v2 pool so the migration is not something we can hope for to help us here any more. I did share this specific instance back with the MMS team though they are tracking them en masse as well via telemetry.

@ViktorHofer ViktorHofer mentioned this issue Oct 1, 2020
Closed
@ViktorHofer
Copy link
Member

@MattGal we are now live updating this issues top post. Maybe that helps?

@jaredpar jaredpar mentioned this issue Oct 6, 2020
Closed
@ViktorHofer
Copy link
Member

Got fixed by AzDO. Closing.

@ghost ghost locked as resolved and limited conversation to collaborators May 12, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' blocking-official-build
Projects
None yet
Milestone
Future
Development

No branches or pull requests
10 participants
@jaredpar @directhex @myblindy @CoffeeFlux @dougbu @ViktorHofer @MattGal @jeffschwMSFT @AraHaan @Dotnet-GitSync-Bot