[Script Recorder] Notarise Electron App

[drewpost]

On macOS, we need to notarise our application to prevent Gatekeeper from throwing an error when it's run that requires a user to process a security exception. This requires admin access which a lot of users won't have, is a frustrating step for users who do have permissions and is reputation ally damaging.

https://kilianvalkhof.com/2019/electron/notarizing-your-electron-application/

[justinkambic]

There's some administrative headache associated with accomplishing this. We're going to need a developer account with Apple. This is going to require working together with IT to get an account set up for us I suppose.

@vigneshshanmugam any thoughts on this?

[vigneshshanmugam]

Yes, we would need a company account for signing the distributable and notarizing the app.

[paulb-elastic]

Do we know if we already have some facility elsewhere in Elastic for this?

[vigneshshanmugam]

I thought about it yesterday and the best thing would be to ask the team responsible for "Elastic Shield" application. Probably IT helpdesk or infosec, dont have much clue.

[elasticdog]

We do use an Apple developer account for notarization of artifacts that are published as part of our automated Release Manager process, but I don't know too many details surrounding how Apple structures permissions for the organization (if we're using a single shared account or if there is a concept of multiple users that fall under a shared developer agreement).

IT manages our Apple account, so they would be best to answer the structure question, but the @elastic/infra-release area members might be able to provide some technical guidance on the existing notarization mechanisms we have in place.

[brunerd]

Hi,
I'm the macOS Endpoint engineer for our Elastic fleet and I've also have admin duties on our Apple Dev account, this issue came to me via Salesforce ticket 00829187.

Currently we've got App IDs for a few Elastic products:

The Endgame and Agent folks are the main active ones, I usually do renewal on the certs and profiles, or approved new devices. You may only need a Developer ID to sign your self-distributed apps, but I'm not sure of what the entitlements that your Synthetics recorder might be using? Maybe none?

Otherwise it looks like we'd need to:

• Add both @vigneshshanmugam and @justinkambic to the Dev program
• Create a new App ID (e.g. co.elastic.syntheticsthingamabob)
• Create a "Developer ID Application" certificate to sign the app (if there is a pkg installer that is a separate cert)

I am might be missing some things since I admin this but am not a dev using these services day-today. So indeed you might benefit from asking the other teams at Elastic who are releasing about this. Once we've got a good idea of App ID names, any entitlements or Services used, etc. we can move forward. Also just checking since this app is already being used in production, InfoSec has already had a chance to do whatever reviews they need to?

Also, here's Apple's minimalist Help pages on these topics: https://help.apple.com/developer-account/#/

[justinkambic]

@brunerd looking at the list of entitlements, at a glance I don't think we need any of them per se.

Also just checking since this app is already being used in production, InfoSec has already had a chance to do whatever reviews they need to?

The app is published as a dev preview at the moment, it's not seeing actual production use yet. We'll start the process with InfoSec soon though.

Add both @vigneshshanmugam and @justinkambic to the Dev program

Could we add a group instead of individuals?

Create a new App ID (e.g. co.elastic.syntheticsthingamabob)

This should be co.elastic.synthetics-recorder or co.elastic.syntheticsRecorder AFAIK. cc @drewpost

[justinkambic]

As an aside, this is the reason we want to get progress on this issue. Users are greeted with this if they try to open from the Applications menu:

If they do the "command+click" method of opening it, they're still given a scary warning message:

[brunerd]

Apple Developer accounts are created per user in "App Store Connect" and it can't leverage our Okta groups, it doesn't have that kind of integration.

Requests come in via the it-eng github, here's some examples of the format and types of requests:
Add new employee to Apple Developer Account - https://github.com/elastic/it-eng/issues/850
Add new Endpoint developer VM UUID to Apple Provisioning Profiles - https://github.com/elastic/it-eng/issues/839

We'll also need to set up the App ID (the bundle ID of the app), the Provisioning Profiles and assign tose profiles to the UUIDs of the Macs being used for dev work.

[justinkambic]

Update here; we've created an issue to track the addition of users and provisioning of an app ID: (private repo) https://github.com/elastic/it-eng/issues/1146.

We need some additional information, namely the UUID of the other developers' laptops not included in that list, as well as a final decision on the App ID to use when signing the bundle.

When we're ready we can let the IT team know and they'll set up and provision the necessary resources for us. The rest of the team is out until the new year so we will need to wait a bit before we can complete this setup process. Once we're past that we can work to notarize the application and issue a new release that won't issue the warning.

[brunerd]

OK, I've created both the App ID co.elastic.synthetics-recorder and a "Developer ID Application" Provisioning Profile named "Synthetics Recorder" associated the certificate Elasticsearch, Inc (Developer ID Application), Jun 02, 2023 (there are 3 in the portal, I chose the date furthest out). Looking closer at the way the Elastic Agent folks have things set up, the UUID assignment only comes into play for Development Profiles. Since this is a "Developer ID Application" it isn't even capable of assignment like that, so we indeed should not need this. Although already added the UUIDs for @lucasfcosta @justinkambic but @vigneshshanmugam I didn't add yours, so that'll be a fine test, it shouldn't be needed for this type of Profile. All the invites are away just now, so look out for those, accept them and sign into XCode and fiddle around 🎻 let me know if you need additional administrative fiddling.

Also, the allow-unsigned-executable-memory entitlement seems to only be a local setting you need to set, it's not something done at the Developer portal level.

[justinkambic]

Thanks @brunerd - I will try to let you know today if I encounter any snags.

[justinkambic]

As an update here - we have successfully signed our binaries with developer certs, and all seems to work fine to a point. We are unable to notarize our binaries presently, details follow.

The next stage as I understand it is to collaborate with the Infra team to create a process by which we can sign our binaries with Developer ID Installer certificates. These certificates will be signed with Elastic's org name (as opposed to the developer-signed binaries we can create locally, which use our developer name, i.e. Justin Kambic in my case). As such, the org does not distribute these certificates for unsupervised use by developers.

I'll update back where when we have moved the new introduction of this process beyond initial stages.

[drewpost]

Hey @justinkambic - any update on this?

[justinkambic]

Hi @drewpost - yes we are progressing on this. Notarization is a late-stage milestone of https://github.com/elastic/observability-robots/issues/1113, which is tracking the overall automation of all the release stages we'll need in order to be GA.

We've recently merged #238, which takes care of signing the binaries with Elastic's official developer certificate. We will be doing some experimentation around notarizing these binaries after we ship our next release, which I'm hoping will be this week sometime.

Once we have shipped a signed binary, I'll experiment with Apple's notarization service to see what additional work is needed (specifying entitlements, is it the right kind of cert, etc.), and we'll make whatever requisite changes to the CI process as needed. At that point, we should be able to automatically trigger a release that will take care of the signing, communication with the Apple service, and publishing the final release binary. It is possible there's a significant amount of discrete scope in that last bit that we won't uncover until we're working on it.

[justinkambic]

After some additional investigation, you cannot submit to Apple's notarization service without access to the keychain that contains the certificate used for signing, so we're going to have to develop this piece of the pipeline completely within our CI environment. cc @v1v

[v1v]

After some additional investigation, you cannot submit to Apple's notarization service without access to the keychain that contains the certificate used for signing, so we're going to have to develop this piece of the pipeline completely within our CI environment. cc @v1v

As long as the the notorization runs in the same CI worker where the electron-build was executed previously, then it should be ok, right? If that's the case, then there is only need to build the notarization script that consumes the list of signed artifacts to be submitted to the Apple's notorization.

Let me ping @mgreau , since he might have some ideas about this.

The original support to sign the apple binaries was done in -> #238

[justinkambic]

@v1v yes I believe you are correct in that we should have no problem executing it as part of the signing process. There is an npm package that handles all of this with some minimal hookup code, but we've abstained from contributing it thus far as we were thinking of being able to leverage an existing process.

The package that handles this does it in an automated way a la Electron Builder.

[kyungeunni]

I've started creating a draft adding notarization script to afterSign hook: #290
I've pushed the custom tag and triggered the release pipeline, but it failed because the worker has xcode 12 or earlier installed so it can not use the latest tool, notarytool.
Even though we decide to use the legacy tool, we are missing some credentials in the keychain, so here are two options we can take:

option 1. Using Worker having xcode13 installed:

• We need the profile in the keychain mentioned in this comment (Is this the same item in the keychain entry as we use for signing? if yes, we don't need to add anything)

option 2. Using current Worker:

• We need to provide appleId and appleIdPassword mentioned in the Legacy option: https://github.com/electron/electron-notarize#method-notarizeopts-promisevoid
• Also need to follow safety recommendations when using appleIdPassword: https://github.com/electron/electron-notarize#safety-when-using-appleidpassword)

@v1v Could you advise which option would be possible to take for us going forward?

[v1v]

@v1v Could you advise which option would be possible to take for us going forward?

We might need to involve the release and CI systems team as they are the system owners and might know further about what certificates or xcode are available in the existing workers.

@elastic/release-eng and @elastic/ci-systems , would you mind helping on this? @kyungeunni asked a few questions in #51 (comment) that you probably can provide further insights.

[nkammah]

We do have a task to upgrade the macOS signing workers to a newer OS, which would include upgrading Xcode. From that angle, I can see going down the option1.

With that said, the work has been put on hold so far due to:

• how short staff the Release Eng team is
• how critical the 2 signing macs are within the release workflow

@kyungeunni what is your rough timeline for wrapping this up ? Would you potentially have some bandwidth in helping to manually provision (instructions) and validate the macOS worker if that's an option - the Release Eng team would normally do it, but as mentioned above, we have no bandwidth at the moment, and a collaboration may be the most effective approach.

cc @mgreau

[kyungeunni]

Hello @nkammah!
the aim is to get it done before 8.5 release date. I'd love to try manually provisioning it with the given instruction. Thanks for the pointer!

[nkammah]

@kyungeunni I re-imaged worker-c07yc0cejyvy and it now has Xcode 13.4.1 and

bash-3.2$ xcrun notarytool
OVERVIEW: Manage submissions to the Apple notary service

USAGE: notarytool <subcommand>

OPTIONS:
  --version               Show the version.
  -h, --help              Show help information.

SUBCOMMANDS:
  store-credentials       Save Notary service credentials to the Keychain. Leave credential options not specified for interactive prompts.
  submit                  Submit an archive to the Notary service
  info                    Get status information for a submission
  wait                    Wait for completion of a previous submission
  history                 Get a list of previous submissions for your team
  log                     Retrieve notarization log for a single completed submission

  See 'notarytool help <subcommand>' for detailed help.

I am validating that worker onto https://internal-ci.elastic.co/. Do you have a Jenkins job we can use to validate that it works on this worker for your app?

[kyungeunni]

Hi @nkammah, thanks for this update!! 🥇

I've triggered a Jenkins job that uses notarytool and seems like it works! (The job failed due to a credential misconfiguration, not because of the worker)

[nkammah]

The job failed due to a credential misconfiguration

Is this something to be fixed on your end or on the worker's end?

[kyungeunni]

I wasn't sure which credentials are needed. Now, I found out we need to generate app-specific password and store it in the keychain, if it hasn't been done yet.

• Generating app-specific password
• Storing credentials to a keychain using notarytool:

xcrun notarytool store-credentials <profile-name>
    --apple-id <developer-apple-id>   
    --team-id <developer-team-id>
    --password <app-specific-password>
    --keychain <optional-keychain-path>

Then I'd need to use the <profile-name> from above in my script. Also, not sure which keychain this information needs to be stored in, but we use [this keychain] (https://github.com/elastic/synthetics-recorder/blob/main/.ci/scripts/release-ci.sh#L7) for signing. I'd really appreciate some help from someone who has an Apple developer account set up and who knows about credential management / adding new credentials to the keychain.

---

References:

• Apple documentation: link from Apple
• Blog post: link

[nkammah]

Our Release Eng. team signing expert is not available atm, but it seems like this is how we do signing of other apps - would a similar approach work for your app?

[kyungeunni]

Thanks for sharing the reference! 👍 I'm trying out Electron focused notarisation tooling (https://github.com/electron/electron-notarize) so once we have the proper credential available for the recorder(I can see Endgame team also uses app-specific password), I think that should be an excellent first step.

Also, I noticed we'd need to review the entitlements list too. It'd be great to consult with the expert in Realease Eng. team! (ofc when they are available :) )

[kyungeunni]

Hi @v1v

Is it possible for us to use notarize-macos-artifacts job for notarising? I found out we use the same credentials for all the mac apps. The scripts seems to be quite general so I was wondering if it'd be possible and also make sense to reuse the existing script. (Also I'd like to avoid exporting credentials to be used in the release scripts as we had security concerns)

Here you can find threads and references regarding the notarisation: https://github.com/elastic/it-eng/issues/1654#issuecomment-1247620277

[nkammah]

@kyungeunni @v1v feel free to take a stab at using the existing Jenkins job and reporting what you find out

[kyungeunni]

@nkammah Thanks for the link 🎉 I've tried it, but it seems to fail when copying files around 🤔 Here's the latest test I've done.

[nkammah]

@kyungeunni I believe the problem is not with failing to copy the files, but rather than the app, or some portion of it, was already signed. We do have exceptions on some files already signed (see here) but it seems like in this case, this one is not listed.

Did you sign './elastic-synthetics-recorder-notarize-2.zip.extracted/Elastic Synthetics Recorder.app/Contents/Frameworks/Elastic Synthetics Recorder Helper.app'. before submitting it to the Jenkins job? If so, can you try again with an un-signed version?

[kyungeunni]

Actually, I uploaded un-signed one(the one I built in my local machine). It is odd that it says it's already signed 🤔 .

I thought it was something related to the artifacts not found in the expected directory as I was seeing these error code:

[nkammah]

The signing script starts by detecting all the path within the .zip file to be signed - in your app's case, it looks as below:

Elastic Synthetics Recorder.app/Contents/Resources/app.asar.unpacked/node_modules/sharp/vendor/8.12.2/darwin-x64/lib/libvips-cpp.42.dylib
Elastic Synthetics Recorder.app/Contents/Resources/app.asar.unpacked/node_modules/sharp/vendor/8.12.2/darwin-arm64v8/lib/libvips-cpp.42.dylib
Elastic Synthetics Recorder.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework
Elastic Synthetics Recorder.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libEGL.dylib
Elastic Synthetics Recorder.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libvk_swiftshader.dylib
Elastic Synthetics Recorder.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libGLESv2.dylib
Elastic Synthetics Recorder.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libffmpeg.dylib
Elastic Synthetics Recorder.app/Contents/Frameworks/ReactiveObjC.framework/Versions/A/ReactiveObjC
Elastic Synthetics Recorder.app/Contents/Frameworks/Squirrel.framework/Versions/A/Squirrel
Elastic Synthetics Recorder.app/Contents/Frameworks/Mantle.framework/Versions/A/Mantle
Elastic Synthetics Recorder.app
Elastic Synthetics Recorder.app/Contents/Frameworks/Elastic Synthetics Recorder Helper.app
Elastic Synthetics Recorder.app/Contents/Frameworks/Elastic Synthetics Recorder Helper (Plugin).app
Elastic Synthetics Recorder.app/Contents/Frameworks/Elastic Synthetics Recorder Helper (GPU).app
Elastic Synthetics Recorder.app/Contents/Frameworks/Elastic Synthetics Recorder Helper (Renderer).app

The error we were receiving was that ./elastic-synthetics-recorder-notarize-2.zip.extracted/Elastic Synthetics Recorder.app/Contents/Frameworks/Elastic Synthetics Recorder Helper.app was already signed.

Looking at the order of path to sign, it seems like we're signing, in that order

• Elastic Synthetics Recorder.app
• Elastic Synthetics Recorder.app/Contents/Frameworks/Elastic Synthetics Recorder Helper.app

Given that the command sign options we use are as below

--deep --force --timestamp --options runtime --strict --verbose=9

my initial theory was that the --deep argument was causing a recursive signing, and that indeed Elastic Synthetics Recorder Helper.app was already signed once when we sign Elastic Synthetics Recorder.app.

For experimentation sake, I removed the --deep flag locally, and re-attempted the signing. It failed with a stapling error:

Processing: /private/var/lib/jenkins/workspace/elastic+unified-release+master+notarize-macos-single-package/cd/release/elastic-macos-notarizer/workspace/extracted-artifacts/synthetics-recorder-0.0.1-notarization.3-mac-arm64.zip.extracted/Elastic Synthetics Recorder.app
CloudKit query for Elastic Synthetics Recorder.app (2/d64416bdc2248c4768a1961aed70afd6d873298d) failed due to "Record not found".
Could not find base64 encoded ticket in response for 2/d64416bdc2248c4768a1961aed70afd6d873298d
The staple and validate action failed! Error 65.

I don't know if this is because I tried to sign an arm64 binary on a non-arm mac.

[nkammah]

@kyungeunni I've been going on and on, trying different things, but it seems like the stapling is failing and I suspect because of an issue with the code structure.

[notarizer] Some notarizations completed with errors.

It links to a more detailed log visible here

{
  "logFormatVersion": 1,
  "jobId": "a9ac778e-a9e1-4510-862a-d95b6256763d",
  "status": "Invalid",
  "statusSummary": "Archive contains critical validation errors",
  "statusCode": 4000,
  "archiveFilename": "synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip",
  "uploadDate": "2022-09-22T13:37:17Z",
  "sha256": "16537c39c6a2de6ed437d384077ae1506fe85f54947b031f90b4ebfb9d5d9d5d",
  "ticketContents": null,
  "issues": [
    {
      "severity": "error",
      "code": null,
      "path": "synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip/workspace/extracted-artifacts/synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.extracted/Elastic Synthetics Recorder.app/Contents/MacOS/Elastic Synthetics Recorder",
      "message": "The signature of the binary is invalid.",
      "docUrl": null,
      "architecture": "arm64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip/workspace/extracted-artifacts/synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.extracted/Elastic Synthetics Recorder.app/Contents/Resources/local-browsers/chromium-978106/chrome-mac/Chromium.app/Contents/MacOS/Chromium",
      "message": "The signature of the binary is invalid.",
      "docUrl": null,
      "architecture": "arm64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip/workspace/extracted-artifacts/synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.extracted/Elastic Synthetics Recorder.app/Contents/Resources/local-browsers/chromium-978106/chrome-mac/Chromium.app/Contents/Frameworks/Chromium Framework.framework/Chromium Framework",
      "message": "The signature of the binary is invalid.",
      "docUrl": null,
      "architecture": "arm64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip/workspace/extracted-artifacts/synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.extracted/Elastic Synthetics Recorder.app/Contents/Frameworks/Electron Framework.framework/Electron Framework",
      "message": "The signature of the binary is invalid.",
      "docUrl": null,
      "architecture": "arm64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip/workspace/extracted-artifacts/synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.extracted/Elastic Synthetics Recorder.app/Contents/Frameworks/ReactiveObjC.framework/ReactiveObjC",
      "message": "The signature of the binary is invalid.",
      "docUrl": null,
      "architecture": "arm64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip/workspace/extracted-artifacts/synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.extracted/Elastic Synthetics Recorder.app/Contents/Frameworks/Squirrel.framework/Squirrel",
      "message": "The signature of the binary is invalid.",
      "docUrl": null,
      "architecture": "arm64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip/workspace/extracted-artifacts/synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.extracted/Elastic Synthetics Recorder.app/Contents/Frameworks/Mantle.framework/Mantle",
      "message": "The signature of the binary is invalid.",
      "docUrl": null,
      "architecture": "arm64"
    }
  ]
}

I have no idea why the file is called synthetics-recorder-0.0.1-beta.6-mac-arm64.zip.zip - but regardless, could you kindly check that your app is properly following Apple guidelines with regards to Nested Code.

[kyungeunni]

A bit of update:

@nkammah and I caught up in Slack and verified that we can notarize the app that is signed by our CD with xcrun notarytool submit command. The next step will be creating a Jenkins job that signs, notarizes, and packages the Electron-based app.

In the meantime, we can notarize the latest release manually so that it is not blocked by the mentioned task above.

[v1v]

The next step will be creating a Jenkins job that signs, notarizes, and packages the Electron-based app.

IIUC, that's already in place in the existing Release stage ->

synthetics-recorder/.ci/Jenkinsfile

Lines 148 to 183 in 1559e65

	stage('Release') {
	options { skipDefaultCheckout() }
	when {
	beforeAgent true
	allOf {
	tag pattern: 'v\\d+\\.\\d+.*', comparator: 'REGEXP'
	expression { isInternalCI() }
	}
	}
	environment {
	BUCKET_NAME = 'internal-ci-artifacts'
	BUCKET_SUBFOLDER = "${env.REPO}/${env.TAG_NAME}/${env.BUILD_ID}"
	BUCKET_PATH = "gs://${env.BUCKET_NAME}/${env.BUCKET_SUBFOLDER}"
	BUCKET_CREDENTIALS = 'internal-ci-gcs-plugin'
	SIGNED_ARTIFACTS = 'signed-artifacts'
	BUCKET_SUBFOLDER_SIGNED_ARTIFACTS = "${env.BUCKET_SUBFOLDER}/${env.SIGNED_ARTIFACTS}"
	BUCKET_SIGNED_ARTIFACTS_PATH = "gs://${env.BUCKET_NAME}/${env.BUCKET_SUBFOLDER_SIGNED_ARTIFACTS}"
	DIST_FOLDER = 'dist'
	RELEASE_URL_MESSAGE = "(<https://github.com/elastic/${env.REPO}/releases/tag/${env.TAG_NAME}|${env.TAG_NAME}>)"
	}
	// There are some environmental issues with some of the CI workers
	// let's use the one we know it works.
	// agent { label 'macos-code-signer' }
	agent { label 'worker-c07yc0cejyvy' }
	stages {
	stage('Dist') {
	options { skipDefaultCheckout() }
	steps {
	deleteDir()
	unstash 'source'
	withNodeJSEnv() {
	dir("${BASE_DIR}"){
	runReleaseWithKeychain()
	}
	}
	}

, which uses:

• the CI worker -> agent { label 'worker-c07yc0cejyvy' }
• the function runReleaseWithKeychain, which it is the function that was implemented in ci: move release process to the internal system (sign linux and macos) #238 to sign the macos artifacts using the keychain stored in Vault.

If the notorizing is something to be done by another CI job provided by the Release team, then it could be similarly done as the one for signing the artifacts with the gpg in

synthetics-recorder/.ci/Jenkinsfile

Lines 221 to 224 in 1559e65

	build(job: 'elastic+unified-release+master+sign-artifacts-with-gpg',
	parameters: [string(name: 'gcs_input_path', value: "${env.BUCKET_PATH}")],
	wait: true,
	propagate: true)

, pretty much trigger another build from the CI pipeline and wait for it.

Let me know if the above is not the way to do it, and how you envision the macos notarisation.