OS X CI strategy roadmap

[sam-github]

There are a number of activities going on in parallel with OS X in CI, but its hard to get a sense of the overall strategy. Some decisions have been made, and open issues track the progress on them, but some more decisions are upcoming. @AshCripps and I will keep this up to date.

---

Notes below are in rough chronological order, some parallelization is posssible:

• 10.15+notarization runing at nearform - this unblocks Catalina issues (ci setup) (notarization) (bug report)
• 10.15 test (and release+notarization maybe?) running in permanent orka hosts at macstadium (some initial steps taken, paused pending Ash completing above)(assumes Orka will donate permanent hosts to us) (tracking issue)

^-- At this point we will have to choose whether to use an Orka or Nearform host for https://ci-release.nodejs.org, because either would work but we build releases only once, on a single machine.

• Benchmark: at this point, we can see relative build speed of Orka vs Nearform, this information may guide decisions about where we put effort in next (though it is possible "ethereal" hosts will benchmark differently than permanent)

• POC of Orka Jenkins plugin that runs workers on "ethereal" hosts

  • Macstadium provides the plugin, so in theory we just enable it on the job, point it at the images we want it to use for the build job, and it will spin up a container, run the build, and spin down the container. This sounds very promising.
  • Q: does it work for us?
  • Q: is it all-or-nothing, ie. can we have node-test-commit-osx run some jobs on nearform, others on Orka?
  • Q: can we run older OS X (10.12 perhaps) as ethereal build jobs (just for test, not release)?

^--- If its one or the other, we may have to choose, and might even stand down the Nearform machines. Its not clear whether using a mixture of near-duplicate build infrastructure makes us more robust to failure, or just gives us more things to manage that have their own independent failure modes.

^--- At this point, we can consider (if all went well) removing OS X 10.10 and 10.11 test hosts from the older macstadium to Orka ethereal jobs. We will have to consider what we want to move --- what selection of older OS X versions do we want to run tests on? Ccurrently, we run only on 10.10 and 10.11. We have 2 instances of 10.12 test hosts, but no builds run on them, and it doesn't appear OS X of those ancient vintages are very popular stats.

[slohiya]

This is blocking our release. We cannot release the mac version of our app. Any update on when this will be addressed is appreciated.

[rvagg]

current, more granular, progress on where we're at right now can probably be followed in nodejs/node#31459

[AshCripps]

Tracking issue for setting up ORKA - #2236

[BethGriggs]

When manually testing the binary of v10.20.0 on my MacOS Catalina I noticed that I am now unable to run the binary that is extracted from the tar.gz. It looks like Apple may have made the notarization checks more strict? 😨

I downloaded https://nodejs.org/download/rc/v10.20.0-rc.1/node-v10.20.0-rc.1-darwin-x64.tar.gz, extracted the file, and tried to run ./bin/node.

/cc @rvagg @AshCripps

[rvagg]

Yikes, they were going to get more "strict" at the beginning of this month but I didn't know it would mean this. I still don't have Catalina running on my laptop (because of chaos like this!) so I can't check right now, but if you go to Security & Privacy in your settings do you have "App Store and identified developers" checked?

I wonder if you did a curl|tar style install if it would be fine. If not then Homebrew must be having trouble with this, surely. This must be because it's been tagged as coming from the web via Chrome.

The plan was to get notarization out in this next 13.x (it's queued, I don't think released yet), then in the next 10.x and 12.x straight after that once we've tested it in a 13.x and made sure there aren't any major screams from users. I think we should probably stick with that but it might mean we should bring forward the next 10.x and 12.x to deal specifically with this in a more timely manner, a bit like pushing out a security release. @nodejs/releasers?

[BethGriggs]

App Store and identified developers is checked. This is a new laptop that I haven't particularly played with any settings yet, so quite close to a fresh install.

nvm works fine, so I tried wget and it doesn't show the warning that way either. The issue only seems to show when I click to download the .tar.gz, odd. I'll keep digging. Another colleague on the same version of MacOS Catalina is seeing different results.

[AshCripps]

@rvagg do we notorize the tar files aswell or just the pkd? just incase this is going to be the new norm for apple to require every zipped folder to come from a "trusted source"

[BethGriggs]

I'm guessing this shouldn't hold up nodejs/node#31984, as we are already aware of notarization issues and there are various work arounds.

Timing of upcoming releases is due to be discussed in the Release WG meeting tomorrow (nodejs/Release#554), I'll bring getting notarized binaries out into that discussion.

[richardlau]

@BethGriggs What we could do is put a note in the release post that it is a known issue that will be addressed in a future release?

[slohiya]

We also use the tar.gz package of nodejs in our application that gets installed on mac. Before this proposed notarization change we were not able to notarize as we were getting errors during notarization on the nodejs files. I am hoping that after this change we will be able to notarize our app that contains tar.gz. Please confirm. Thanks for all the help.

[rvagg]

No, we don't notarize the .tar.gz, and can't. But I would assume that people downloading with their browser are going to be choosing the .pkg, not the .tar.?z file. But that doesn't help in this instance since the .pkg isn't notarized and won't even install.

@slohiya check with a recent nightly build to see if that works: https://nodejs.org/download/nightly/v14.0.0-nightly20200325f4153c2fe7/, our signing on master for those nightly builds includes the necessary entitlements that may be causing your hiccup. The additional term-size binary that ships with npm is in there too if you're bundling npm with your application as well.

v13.12.0 is due to go out any day now, see nodejs/node#32376, and it has all of this stuff in it. When that's released you could put that through your bundling process to see if it's resolved.

[slohiya]

@rvagg : I understand tar.gz will not be notarized. As you suspect, we were running into issues notarizing our app that includes content of tar.gz probably because of missing entitlements and term-size issue. I will try the nightly build or 13.X if released and see if that works for me. Thanks for quick response and help on this.

[slohiya]

@rvagg : My preliminary testing using the above mentioned nightly build shows that including the contents of the nodejs tar.gz in our app does not give any notarization issue. So I think its looking good. I will just have to wait till this fix is back ported to 12.X (LTS). We will perform some more testing on this. Thank you and others again for your help on this.

[slohiya]

@rvagg: Repeated our testing on the 13.12.0 release and notarization looks good. Do you know when this notarization fix back porting to 12.X (LTS) is scheduled? Or where can I track the progress of the back port to 12.X?

[rvagg]

Backports for notarization:

• [v10.x] backport npm 6.14.4 and notarization for macOS node#32528 10.x
• [v12.x] backport npm updates and macOS notarization node#32527 12.x

I'd expect these to land easily, so far I'm not seeing feedback about the 13.x notarization impacting anything but I'm not well tuned in to where people would report such problems. Barring any problems, the next 10.x and 12.x should be when we see releases:

• Release plan - v10.x Maintenance Release#504 10.x release plan
• Release plan - v12.x Maintenance Release#494 12.x release plan

Both have the 21st of April listed as their next dates. I believe the release team has been discussing timing in light of notarization but I don't know if this will change anything.

Any of the release folks on this thread have anything to add?

[richardlau]

The notarization changes happened and we're also using Orka in the CI. @AshCripps Is there anything remaining to do for this issue?

[AshCripps]

whats left is removal of the old hosting solution and possibly trialing the jenkins plugin but those can be/are tracked in other issues so closing