[aws mac1.metal] [iOS Big Sur][codesign] buildklite-agent get errSecInternalComponent when trying to codesign

[OliverKoo]

Environment
Hi I am using buildkite on my aws mac1.metal instances. These agents acts as CI agents in our org. Instances now launch buildkite-agent when the instance is booted (done vis plist in /Library/LauchDaemon). Here is my plist

Issue
The agent trys to run the following command using python subprocess

subprocess.CalledProcessError: Command '['/usr/bin/codesign', '-v', '--sign', 'XXXXXXXXXXXXXXXXXXXXXXXXX', '--entitlements', 'bazel-out/darwin-opt/bin/MYORG/iOS/WidgetExtension/WidgetExtension_entitlements.entitlements', '--force', 'bazel-out/darwin-opt/bin/MYORG/iOS/WidgetExtension/WidgetExtension_archive-root/WidgetExtension.appex']

resulting this error:

ERROR:
--
  | bazel-out/darwin-opt/bin/MYORG/iOS/WidgetExtension/WidgetExtension_archive-root/WidgetExtension.appex: errSecInternalComponent

Interestingly running the same job with agent that I launch locally. (ssh into the machine, then run buildkite-agent start) then it worked flawlessly. So I am wondering if there are some kind of permission difference between launching buildkite-agent as a daemon via launchd and starting it locally. (similar to this issue I opened before where bk agent failed a job as launchd daemon but succeed if launch locally)

Things I have tried to fix/Debugging

• errSecInternalComponent seems like a common issue with codesign so I tried locking and unlocking the keychain with security unlock-keychain -p password path/to/key

• people who use other CI system/tool also encounter the same issue but largely solve when setting SessionCreate in their plist. . Tried but still failing

• VNC into the mac1.metal ec2 instance and gave /usr/local/bin/buildkite-agent FDA - still failed the job

• VNC in and open KeyChain access and give /usr/local/bin/buildkite-agent access to the private key it is having issue signing - still failed

• everything here

Wondering if any of you buildkite wizard 🧙‍♀️ 🧙‍♂️ knows the difference between launchd buildkite-agent and agent launch locally? I feel like there is a big gotcha I am missing. Perhaps an attribute missing in my plist.

[OliverKoo]

also tried this - https://stackoverflow.com/questions/16550594/jenkins-xcode-build-works-codesign-fails
creating a temp key in post checkout hook but still failed

[sj26]

Hi @OliverKoo! Sorry I'm not sure of the details, but I think there are many ways you can run processes and daemons on macOS. I know that codesign is a pretty sensitive bit of macOS and requires elevated privileges and access to keychains. I believe we've solved this using something like SessionCreate in the past, but I'm not sure we have an easy example to point at sorry. I'll ping some other brains and see if anybody has any bright ideas.

[OliverKoo]

Hi @sj26 thanks for getting back to me so quickly 😃

I have tried the SessionCreate method before. I have linked the post in my post under "things I have tried (second bullet point)" but wasn't able to get around it.

Definitely let me know what other buildkite wizard thinks please

[sj26]

@OliverKoo hmm, I found some more breadcrumbs — I think for Xcode to work correctly the mac needs to be logged in as a user, and the buildkite agent needs to be started as part of that user session. You might need to enable something like this to auto-login:
https://support.apple.com/en-au/HT201476

There are some related notes here for the GitLab Runner, which runs similarly:
https://docs.gitlab.com/runner/install/osx.html#limitations-on-macos

[OliverKoo]

@sj26 great find. I will try migrate plist to LaunchAgent from LaunchDaemon and set up auto login, then report back

[OliverKoo]

@sj26 launch agent from /Users/buildkite-agent/Library/LaunchAgents/com.buildkite.buildkite-agent.plist with auto-login enabled solved this 🎉 . closing this ticket.