Occasional abort trap 6 error when signing application bundles

[ofek]

By occasional I mean basically half of the time. Restarting the job that signs the binaries has no effect on this step for some reason and I must restart the entire pipeline, building from scratch.

https://github.com/pypa/hatch/actions/runs/5301640159/jobs/9596106008

+ ls -la staged/Hatch.app
total 0
drwxr-xr-x  3 runner  staff   96 Jun 18 04:04 .
+ ls -la staged/Hatch.app/Contents
drwxr-xr-x  3 runner  staff   96 Jun 18 04:04 ..
drwxr-xr-x  4 runner  staff  128 Jun 18 04:04 Contents
total 8
drwxr-xr-x  4 runner  staff  128 Jun 18 04:04 .
drwxr-xr-x  3 runner  staff   96 Jun 18 04:04 ..
+ ls -la staged/Hatch.app/Contents/MacOS
-rw-r--r--  1 runner  staff  576 Jun 18 04:04 Info.plist
drwxr-xr-x  3 runner  staff   96 Jun 18 04:04 MacOS
total 16688
drwxr-xr-x  3 runner  staff       96 Jun 18 04:04 .
drwxr-xr-x  4 runner  staff      128 Jun 18 04:04 ..
-rwxrwx---  1 runner  staff  8543032 Jun 18 04:04 hatch

+ rcodesign sign -vvv --pem-source /tmp/certificate.pem --pem-source /tmp/private-key.pem "staged/Hatch.app" "signed/Hatch.app"
[2023-06-18T04:04:36Z WARN  apple_codesign::cli] reading PEM data from /tmp/certificate.pem
[2023-06-18T04:04:36Z WARN  apple_codesign::cli] reading PEM data from /tmp/private-key.pem
[2023-06-18T04:04:36Z WARN  apple_codesign::cli] registering signing key
[2023-06-18T04:04:36Z WARN  apple_codesign::cli] automatically registered Apple CA certificate: Developer ID Certification Authority
[2023-06-18T04:04:36Z WARN  apple_codesign::cli] automatically registered Apple CA certificate: Apple Root CA
[2023-06-18T04:04:36Z WARN  apple_codesign::cli] using time-stamp protocol server http://timestamp.apple.com/ts01
[2023-06-18T04:04:36Z WARN  apple_codesign::cli] automatically setting team ID from signing certificate: BMM5U3QVKW
[2023-06-18T04:04:36Z WARN  apple_codesign::cli] signing staged/Hatch.app to signed/Hatch.app
[2023-06-18T04:04:36Z WARN  apple_codesign::signing] signing bundle at staged/Hatch.app
[2023-06-18T04:04:36Z WARN  apple_codesign::bundle_signing] signing 0 nested bundles in the following order:
[2023-06-18T04:04:36Z WARN  apple_codesign::bundle_signing] signing bundle at staged/Hatch.app into signed/Hatch.app
[2023-06-18T04:04:36Z DEBUG goblin::mach] Ctx: Ctx *** container: Big, le: Little ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] Mach-o header: Header *** magic: 0xfeedfacf, cputype: 16777228, cpusubtype: 0x0, filetype: "EXECUTE", ncmds: 20, sizeofcmds: 2080, flags: 0xa00085, reserved: 0x0 ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 0 - LoadCommand *** offset: 32, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 72, segname: [95, 95, 80, 65, 71, 69, 90, 69, 82, 79, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 4294967296, fileoff: 0, filesize: 0, maxprot: 0, initprot: 0, nsects: 0, flags: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 1 - LoadCommand *** offset: 104, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 632, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 4294967296, vmsize: 3506176, fileoff: 0, filesize: 3506176, maxprot: 5, initprot: 5, nsects: 7, flags: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 2 - LoadCommand *** offset: 736, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 232, segname: [95, 95, 68, 65, 84, 65, 95, 67, 79, 78, 83, 84, 0, 0, 0, 0], vmaddr: 4298473472, vmsize: 212992, fileoff: 3506176, filesize: 212992, maxprot: 3, initprot: 3, nsects: 2, flags: 16 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 3 - LoadCommand *** offset: 968, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 552, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 4298686464, vmsize: 16384, fileoff: 3719168, filesize: 16384, maxprot: 3, initprot: 3, nsects: 6, flags: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 4 - LoadCommand *** offset: 1520, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 72, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 4298702848, vmsize: 245760, fileoff: 3735552, filesize: 231632, maxprot: 1, initprot: 1, nsects: 0, flags: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 5 - LoadCommand *** offset: 1592, command: DyldChainedFixups(LinkeditDataCommand *** cmd: 2147483700, cmdsize: 16, dataoff: 3735552, datasize: 3008 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 6 - LoadCommand *** offset: 1608, command: DyldExportsTrie(LinkeditDataCommand *** cmd: 2147483699, cmdsize: 16, dataoff: 3738560, datasize: 168928 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 7 - LoadCommand *** offset: 1624, command: Symtab(SymtabCommand *** cmd: 2, cmdsize: 24, symoff: 3918104, nsyms: 159, stroff: 3921888, strsize: 2288 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 8 - LoadCommand *** offset: 1648, command: Dysymtab(DysymtabCommand *** cmd: 11, cmdsize: 80, ilocalsym: 0, nlocalsym: 1, iextdefsym: 1, nextdefsym: 1, iundefsym: 2, nundefsym: 157, tocoff: 0, ntoc: 0, modtaboff: 0, nmodtab: 0, extrefsymoff: 0, nextrefsyms: 0, indirectsymoff: 3920648, nindirectsyms: 310, extreloff: 0, nextrel: 0, locreloff: 0, nlocrel: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 9 - LoadCommand *** offset: 1728, command: LoadDylinker(DylinkerCommand *** cmd: 14, cmdsize: 32, name: 12 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 10 - LoadCommand *** offset: 1760, command: Uuid(UuidCommand *** cmd: 27, cmdsize: 24, uuid: [232, 130, 244, 78, 105, 206, 52, 108, 147, 35, 71, 57, 62, 219, 10, 19] ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 11 - LoadCommand *** offset: 1784, command: Unimplemented(LoadCommandHeader *** cmd: 50, cmdsize: 32 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 12 - LoadCommand *** offset: 1816, command: SourceVersion(SourceVersionCommand *** cmd: 42, cmdsize: 16, version: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 13 - LoadCommand *** offset: 1832, command: Main(EntryPointCommand *** cmd: 2147483688, cmdsize: 24, entryoff: 35824, stacksize: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 14 - LoadCommand *** offset: 1856, command: LoadDylib(DylibCommand *** cmd: 12, cmdsize: 96, dylib: Dylib *** name: 24, timestamp: 2, current_version: 3959700504, compatibility_version: 65536 *** ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 15 - LoadCommand *** offset: 1952, command: LoadDylib(DylibCommand *** cmd: 12, cmdsize: 56, dylib: Dylib *** name: 24, timestamp: 2, current_version: 458752, compatibility_version: 458752 *** ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 16 - LoadCommand *** offset: 2008, command: LoadDylib(DylibCommand *** cmd: 12, cmdsize: 56, dylib: Dylib *** name: 24, timestamp: 2, current_version: 86441984, compatibility_version: 65536 *** ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 17 - LoadCommand *** offset: 2064, command: FunctionStarts(LinkeditDataCommand *** cmd: 38, cmdsize: 16, dataoff: 3907488, datasize: 10616 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 18 - LoadCommand *** offset: 2080, command: DataInCode(LinkeditDataCommand *** cmd: 41, cmdsize: 16, dataoff: 3918104, datasize: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 19 - LoadCommand *** offset: 2096, command: CodeSignature(LinkeditDataCommand *** cmd: 29, cmdsize: 16, dataoff: 3924176, datasize: 43008 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] Ctx: Ctx *** container: Big, le: Little ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] Mach-o header: Header *** magic: 0xfeedfacf, cputype: 16777223, cpusubtype: 0x3, filetype: "EXECUTE", ncmds: 19, sizeofcmds: 2256, flags: 0xa00085, reserved: 0x0 ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 0 - LoadCommand *** offset: 32, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 72, segname: [95, 95, 80, 65, 71, 69, 90, 69, 82, 79, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 4294967296, fileoff: 0, filesize: 0, maxprot: 0, initprot: 0, nsects: 0, flags: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 1 - LoadCommand *** offset: 104, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 712, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 4294967296, vmsize: 4063232, fileoff: 0, filesize: 4063232, maxprot: 5, initprot: 5, nsects: 8, flags: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 2 - LoadCommand *** offset: 816, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 232, segname: [95, 95, 68, 65, 84, 65, 95, 67, 79, 78, 83, 84, 0, 0, 0, 0], vmaddr: 4299030528, vmsize: 212992, fileoff: 4063232, filesize: 212992, maxprot: 3, initprot: 3, nsects: 2, flags: 16 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 3 - LoadCommand *** offset: 1048, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 632, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 4299243520, vmsize: 16384, fileoff: 4276224, filesize: 16384, maxprot: 3, initprot: 3, nsects: 7, flags: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 4 - LoadCommand *** offset: 1680, command: Segment64(SegmentCommand64 *** cmd: 25, cmdsize: 72, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 4299259904, vmsize: 262144, fileoff: 4292608, filesize: 252728, maxprot: 1, initprot: 1, nsects: 0, flags: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 5 - LoadCommand *** offset: 1752, command: DyldInfoOnly(DyldInfoCommand *** cmd: 2147483682, cmdsize: 48, rebase_off: 4292608, rebase_size: 11456, bind_off: 4304064, bind_size: 104, weak_bind_off: 0, weak_bind_size: 0, lazy_bind_off: 4304168, lazy_bind_size: 3384, export_off: 4307552, export_size: 169696 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 6 - LoadCommand *** offset: 1800, command: Symtab(SymtabCommand *** cmd: 2, cmdsize: 24, symoff: 4491912, nsyms: 164, stroff: 4495832, strsize: 2400 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 7 - LoadCommand *** offset: 1824, command: Dysymtab(DysymtabCommand *** cmd: 11, cmdsize: 80, ilocalsym: 0, nlocalsym: 1, iextdefsym: 1, nextdefsym: 1, iundefsym: 2, nundefsym: 162, tocoff: 0, ntoc: 0, modtaboff: 0, nmodtab: 0, extrefsymoff: 0, nextrefsyms: 0, indirectsymoff: 4494536, nindirectsyms: 323, extreloff: 0, nextrel: 0, locreloff: 0, nlocrel: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 8 - LoadCommand *** offset: 1904, command: LoadDylinker(DylinkerCommand *** cmd: 14, cmdsize: 32, name: 12 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 9 - LoadCommand *** offset: 1936, command: Uuid(UuidCommand *** cmd: 27, cmdsize: 24, uuid: [85, 19, 165, 180, 183, 119, 48, 225, 172, 149, 174, 139, 191, 71, 254, 194] ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 10 - LoadCommand *** offset: 1960, command: Unimplemented(LoadCommandHeader *** cmd: 50, cmdsize: 32 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 11 - LoadCommand *** offset: 1992, command: SourceVersion(SourceVersionCommand *** cmd: 42, cmdsize: 16, version: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 12 - LoadCommand *** offset: 2008, command: Main(EntryPointCommand *** cmd: 2147483688, cmdsize: 24, entryoff: 52160, stacksize: 0 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 13 - LoadCommand *** offset: 2032, command: LoadDylib(DylibCommand *** cmd: 12, cmdsize: 96, dylib: Dylib *** name: 24, timestamp: 2, current_version: 3959700504, compatibility_version: 65536 *** ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 14 - LoadCommand *** offset: 2128, command: LoadDylib(DylibCommand *** cmd: 12, cmdsize: 56, dylib: Dylib *** name: 24, timestamp: 2, current_version: 458752, compatibility_version: 458752 *** ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 15 - LoadCommand *** offset: 2184, command: LoadDylib(DylibCommand *** cmd: 12, cmdsize: 56, dylib: Dylib *** name: 24, timestamp: 2, current_version: 86441984, compatibility_version: 65536 *** ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 16 - LoadCommand *** offset: 2240, command: FunctionStarts(LinkeditDataCommand *** cmd: 38, cmdsize: 16, dataoff: 4477248, datasize: 11096 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 17 - LoadCommand *** offset: 2256, command: DataInCode(LinkeditDataCommand *** cmd: 41, cmdsize: 16, dataoff: 4488344, datasize: 3568 ***) ***
[2023-06-18T04:04:36Z DEBUG goblin::mach] 18 - LoadCommand *** offset: 2272, command: CodeSignature(LinkeditDataCommand *** cmd: 29, cmdsize: 16, dataoff: 4498232, datasize: 47104 ***) ***
[2023-06-18T04:04:36Z WARN  apple_codesign::bundle_signing] collecting code resources files
/Users/runner/work/_temp/e08e2dd0-4a91-408d-ab80-c2bdf0a32da4.sh: line 1:  3467 Abort trap: 6           rcodesign sign -vvv --pem-source /tmp/certificate.pem --pem-source /tmp/private-key.pem "staged/Hatch.app" "signed/Hatch.app"

[roblabla]

So I managed to reproduce the crash by forking your repo, setting all the certs up, and ssh-ing into the runner. I recovered a bunch of information. First of all, this seems to be either a double free or otherwise a case of freeing a pointer that wasn't malloc'd (memory corruption?), according to this output I found while running rcodesign directly in a terminal in the CI:

rcodesign(11383,0x118b64600) malloc: *** error for object 0x600001de0000: pointer being freed was not allocated
rcodesign(11383,0x118b64600) malloc: *** set a breakpoint in malloc_error_break to debug

Running the same command on the same bundle with the same certs on my local mac does not trigger the crash. So I guess there's something different between the CI and my local environments. Of course, there's a lot of subtle differences (for instance, I'm using macOS 13.3.1 on an M1 mac, the runner is on 12.2.6 on an x86_64 VM).

Here's the unsigned bundle: staged.zip

The crash report summary

Process:               rcodesign [3213]
Path:                  /usr/local/bin/rcodesign
Identifier:            rcodesign
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        Exited process [3212]
Responsible:           bash [457]
User ID:               501

Date/Time:             2023-06-18 10:00:44.5017 +0000
OS Version:            macOS 12.6.6 (21G646)
Report Version:        12
Anonymous UUID:        28C488D7-2871-0137-7EA3-2D329D89124C


Time Awake Since Boot: 1000 seconds

System Integrity Protection: enabled

Crashed Thread:        0  main  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
abort() called


Thread 0 Crashed:: main Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	    0x7ff80ea81ffe __pthread_kill + 10
1   libsystem_pthread.dylib       	    0x7ff80eab81ff pthread_kill + 263
2   libsystem_c.dylib             	    0x7ff80ea03d24 abort + 123
3   libsystem_malloc.dylib        	    0x7ff80e8e1357 malloc_vreport + 551
4   libsystem_malloc.dylib        	    0x7ff80e8e452b malloc_report + 151
5   rcodesign                     	       0x102905de6 walkdir::IntoIter::pop::h7b7004a68910f74d + 230
6   rcodesign                     	       0x10290494d _$LT$walkdir..IntoIter$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::next::h2c8fc4520b1e67bf + 1325
7   rcodesign                     	       0x1028fbde3 apple_bundles::directory_bundle::DirectoryBundle::nested_bundles::h0c101ca1570a2027 + 403
8   rcodesign                     	       0x1028fb94e apple_bundles::directory_bundle::DirectoryBundle::files::h2a2eeacbb24c371f + 46
9   rcodesign                     	       0x10205efba apple_codesign::bundle_signing::SingleBundleSigner::write_signed_bundle::h381f90355b23f35c + 5258
10  rcodesign                     	       0x10205b4c7 apple_codesign::bundle_signing::BundleSigner::write_signed_bundle::ha518127d405caf05 + 1655
11  rcodesign                     	       0x102093e04 apple_codesign::signing::UnifiedSigner::sign_path::had65b4bfe9eefd68 + 436
12  rcodesign                     	       0x101f394f8 apple_codesign::cli::command_sign::he4fe5fd0e5439083 + 13912
13  rcodesign                     	       0x101f45158 apple_codesign::cli::main_impl::hc3c74fbaf30fe330 + 35480
14  rcodesign                     	       0x101e98f28 rcodesign::main::h6c4b4400db661d0e + 24
15  rcodesign                     	       0x101e993f6 std::sys_common::backtrace::__rust_begin_short_backtrace::h8aefd5ecbb734caf + 6
16  rcodesign                     	       0x101e995ac std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::hffee58d366c3b5e2 + 12
17  rcodesign                     	       0x102a013d4 std::rt::lang_start_internal::h9ca2efac34d80f78 + 708
18  rcodesign                     	       0x101e98ffc main + 44
19  dyld                          	       0x112df252e start + 462


Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000  rbx: 0x0000000112e6d600  rcx: 0x00007ff7be061918  rdx: 0x0000000000000000
  rdi: 0x0000000000000103  rsi: 0x0000000000000006  rbp: 0x00007ff7be061940  rsp: 0x00007ff7be061918
   r8: 0x0000000000000001   r9: 0x0000000000000000  r10: 0x0000000000000000  r11: 0x0000000000000246
  r12: 0x0000000000000103  r13: 0x0000000000000050  r14: 0x0000000000000006  r15: 0x0000000000000016
  rip: 0x00007ff80ea81ffe  rfl: 0x0000000000000246  cr2: 0x00007ff8501ef008
  
Logical CPU:     0
Error Code:      0x02000148 
Trap Number:     133


Binary Images:
    0x7ff80ea7a000 -     0x7ff80eab1fff libsystem_kernel.dylib (*) <35a199f3-e389-3202-916b-d6662a391f7e> /usr/lib/system/libsystem_kernel.dylib
    0x7ff80eab2000 -     0x7ff80eabdfff libsystem_pthread.dylib (*) <e5d44afd-2577-3cee-8711-9d8d426229e0> /usr/lib/system/libsystem_pthread.dylib
    0x7ff80e982000 -     0x7ff80ea0afff libsystem_c.dylib (*) <e42e9d7a-03b4-340b-b61e-dcd45fd4acc0> /usr/lib/system/libsystem_c.dylib
    0x7ff80e8d2000 -     0x7ff80e8fdfff libsystem_malloc.dylib (*) <47042acd-a337-322a-8db7-ecd59cc60d92> /usr/lib/system/libsystem_malloc.dylib
       0x101e94000 -        0x102c8ffff rcodesign (*) <6c427287-0796-33d2-a878-1c515e58ba48> /usr/local/bin/rcodesign
       0x112ded000 -        0x112e58fff dyld (*) <2517e9fe-884a-3855-8532-92bffba3f81c> /usr/lib/dyld

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=541.7M resident=0K(0%) swapped_out_or_unallocated=541.7M(100%)
Writable regions: Total=537.4M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=537.4M(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Kernel Alloc Once                    8K        1 
MALLOC                           145.2M       12 
MALLOC guard page                   16K        4 
MALLOC_NANO (reserved)           384.0M        1         reserved VM address space (unallocated)
Stack                             8188K        1 
Stack Guard                       56.0M        1 
VM_ALLOCATE                         20K        5 
VM_ALLOCATE (reserved)             128K        1         reserved VM address space (unallocated)
__DATA                            2489K      119 
__DATA_CONST                      5197K       83 
__DATA_DIRTY                       209K       50 
__LINKEDIT                       444.0M        6 
__OBJC_RO                         82.8M        1 
__OBJC_RW                         3200K        2 
__TEXT                            97.7M      131 
__UNICODE                          592K        1 
dyld private memory               1024K        1 
shared memory                        8K        1 
===========                     =======  ======= 
TOTAL                              1.2G      421 
TOTAL, minus reserved VM space   846.2M      421 

The crash.ips file

{"app_name":"rcodesign","timestamp":"2023-06-18 10:00:48.00 +0000","app_version":"","slice_uuid":"6c427287-0796-33d2-a878-1c515e58ba48","build_version":"","platform":1,"share_with_app_devs":0,"is_first_party":1,"bug_type":"309","os_version":"macOS 12.6.6 (21G646)","incident_id":"F1FF7853-A0C8-456D-984B-C7AA9F6C396A","name":"rcodesign"}
{
  "uptime" : 1000,
  "procLaunch" : "2023-06-18 10:00:44.4841 +0000",
  "procRole" : "Unspecified",
  "version" : 2,
  "userID" : 501,
  "deployVersion" : 210,
  "modelCode" : "VMware7,1",
  "procStartAbsTime" : 1009291506387,
  "coalitionID" : 447,
  "osVersion" : {
    "train" : "macOS 12.6.6",
    "build" : "21G646",
    "releaseType" : "User"
  },
  "captureTime" : "2023-06-18 10:00:44.5017 +0000",
  "incident" : "F1FF7853-A0C8-456D-984B-C7AA9F6C396A",
  "bug_type" : "309",
  "pid" : 3213,
  "procExitAbsTime" : 1009308700017,
  "cpuType" : "X86-64",
  "procName" : "rcodesign",
  "procPath" : "\/usr\/local\/bin\/rcodesign",
  "parentProc" : "Exited process",
  "parentPid" : 3212,
  "coalitionName" : "runner-provisioner",
  "crashReporterKey" : "28C488D7-2871-0137-7EA3-2D329D89124C",
  "responsiblePid" : 457,
  "responsibleProc" : "bash",
  "sip" : "enabled",
  "isCorpse" : 1,
  "exception" : {"codes":"0x0000000000000000, 0x0000000000000000","rawCodes":[0,0],"type":"EXC_CRASH","signal":"SIGABRT"},
  "asi" : {"libsystem_c.dylib":["abort() called"]},
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":16355,"threadState":{"r13":{"value":80},"rax":{"value":0},"rflags":{"value":582},"cpu":{"value":0},"r14":{"value":6},"rsi":{"value":6},"r8":{"value":1},"cr2":{"value":140704472821768},"rdx":{"value":0},"r10":{"value":0},"r9":{"value":0},"r15":{"value":22},"rbx":{"value":4612085248,"symbolLocation":0,"symbol":"_main_thread"},"trap":{"value":133},"err":{"value":33554760},"r11":{"value":582},"rip":{"value":140703374516222,"matchesCrashFrame":1},"rbp":{"value":140702021720384},"rsp":{"value":140702021720344},"r12":{"value":259},"rcx":{"value":140702021720344},"flavor":"x86_THREAD_STATE","rdi":{"value":259}},"name":"main","queue":"com.apple.main-thread","frames":[{"imageOffset":32766,"symbol":"__pthread_kill","symbolLocation":10,"imageIndex":0},{"imageOffset":25087,"symbol":"pthread_kill","symbolLocation":263,"imageIndex":1},{"imageOffset":531748,"symbol":"abort","symbolLocation":123,"imageIndex":2},{"imageOffset":62295,"symbol":"malloc_vreport","symbolLocation":551,"imageIndex":3},{"imageOffset":75051,"symbol":"malloc_report","symbolLocation":151,"imageIndex":3},{"imageOffset":10952166,"symbol":"walkdir::IntoIter::pop::h7b7004a68910f74d","symbolLocation":230,"imageIndex":4},{"imageOffset":10946893,"symbol":"_$LT$walkdir..IntoIter$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::next::h2c8fc4520b1e67bf","symbolLocation":1325,"imageIndex":4},{"imageOffset":10911203,"symbol":"apple_bundles::directory_bundle::DirectoryBundle::nested_bundles::h0c101ca1570a2027","symbolLocation":403,"imageIndex":4},{"imageOffset":10910030,"symbol":"apple_bundles::directory_bundle::DirectoryBundle::files::h2a2eeacbb24c371f","symbolLocation":46,"imageIndex":4},{"imageOffset":1879994,"symbol":"apple_codesign::bundle_signing::SingleBundleSigner::write_signed_bundle::h381f90355b23f35c","symbolLocation":5258,"imageIndex":4},{"imageOffset":1864903,"symbol":"apple_codesign::bundle_signing::BundleSigner::write_signed_bundle::ha518127d405caf05","symbolLocation":1655,"imageIndex":4},{"imageOffset":2096644,"symbol":"apple_codesign::signing::UnifiedSigner::sign_path::had65b4bfe9eefd68","symbolLocation":436,"imageIndex":4},{"imageOffset":677112,"symbol":"apple_codesign::cli::command_sign::he4fe5fd0e5439083","symbolLocation":13912,"imageIndex":4},{"imageOffset":725336,"symbol":"apple_codesign::cli::main_impl::hc3c74fbaf30fe330","symbolLocation":35480,"imageIndex":4},{"imageOffset":20264,"symbol":"rcodesign::main::h6c4b4400db661d0e","symbolLocation":24,"imageIndex":4},{"imageOffset":21494,"symbol":"std::sys_common::backtrace::__rust_begin_short_backtrace::h8aefd5ecbb734caf","symbolLocation":6,"imageIndex":4},{"imageOffset":21932,"symbol":"std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::hffee58d366c3b5e2","symbolLocation":12,"imageIndex":4},{"imageOffset":11981780,"symbol":"std::rt::lang_start_internal::h9ca2efac34d80f78","symbolLocation":708,"imageIndex":4},{"imageOffset":20476,"symbol":"main","symbolLocation":44,"imageIndex":4},{"imageOffset":21806,"symbol":"start","symbolLocation":462,"imageIndex":5}]}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703374483456,
    "size" : 229376,
    "uuid" : "35a199f3-e389-3202-916b-d6662a391f7e",
    "path" : "\/usr\/lib\/system\/libsystem_kernel.dylib",
    "name" : "libsystem_kernel.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703374712832,
    "size" : 49152,
    "uuid" : "e5d44afd-2577-3cee-8711-9d8d426229e0",
    "path" : "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "name" : "libsystem_pthread.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703373467648,
    "size" : 561152,
    "uuid" : "e42e9d7a-03b4-340b-b61e-dcd45fd4acc0",
    "path" : "\/usr\/lib\/system\/libsystem_c.dylib",
    "name" : "libsystem_c.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703372746752,
    "size" : 180224,
    "uuid" : "47042acd-a337-322a-8db7-ecd59cc60d92",
    "path" : "\/usr\/lib\/system\/libsystem_malloc.dylib",
    "name" : "libsystem_malloc.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4327030784,
    "size" : 14663680,
    "uuid" : "6c427287-0796-33d2-a878-1c515e58ba48",
    "path" : "\/usr\/local\/bin\/rcodesign",
    "name" : "rcodesign"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4611559424,
    "size" : 442368,
    "uuid" : "2517e9fe-884a-3855-8532-92bffba3f81c",
    "path" : "\/usr\/lib\/dyld",
    "name" : "dyld"
  }
],
  "sharedCache" : {
  "base" : 140703371456512,
  "size" : 19331678208,
  "uuid" : "f1151485-63d5-339e-9a30-e15d7842ae9f"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=541.7M resident=0K(0%) swapped_out_or_unallocated=541.7M(100%)\nWritable regions: Total=537.4M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=537.4M(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nKernel Alloc Once                    8K        1 \nMALLOC                           145.2M       12 \nMALLOC guard page                   16K        4 \nMALLOC_NANO (reserved)           384.0M        1         reserved VM address space (unallocated)\nStack                             8188K        1 \nStack Guard                       56.0M        1 \nVM_ALLOCATE                         20K        5 \nVM_ALLOCATE (reserved)             128K        1         reserved VM address space (unallocated)\n__DATA                            2489K      119 \n__DATA_CONST                      5197K       83 \n__DATA_DIRTY                       209K       50 \n__LINKEDIT                       444.0M        6 \n__OBJC_RO                         82.8M        1 \n__OBJC_RW                         3200K        2 \n__TEXT                            97.7M      131 \n__UNICODE                          592K        1 \ndyld private memory               1024K        1 \nshared memory                        8K        1 \n===========                     =======  ======= \nTOTAL                              1.2G      421 \nTOTAL, minus reserved VM space   846.2M      421 \n",
  "legacyInfo" : {
  "threadTriggered" : {
    "name" : "main",
    "queue" : "com.apple.main-thread"
  }
},
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "61301e3a61217b3110231469",
      "factorPackIds" : {
        "SIRI_FIND_MY_CONFIGURATION_FILES" : "6348493aa52bb16adc4e4d06"
      },
      "deploymentId" : 240000023
    },
    {
      "rolloutId" : "607844aa04477260f58a8077",
      "factorPackIds" : {
        "SIRI_MORPHUN_ASSETS" : "6103050cbfe6dc472e1c982a"
      },
      "deploymentId" : 240000066
    }
  ],
  "experiments" : [
    {
      "treatmentId" : "c28e4ee6-1b08-4f90-8e05-2809e78310a3",
      "experimentId" : "6317d2003d24842ff850182a",
      "deploymentId" : 400000013
    },
    {
      "treatmentId" : "6dd670af-0633-45e4-ae5f-122ae4df02be",
      "experimentId" : "64406ba83deb637ac8a04419",
      "deploymentId" : 900000005
    }
  ]
}
}

[roblabla]

Another interesting thing: I ran cargo install apple-codesign and the resulting binary did not crash while signing the bundle in CI. Could the latest release have perhaps an old, buggy version of a dependency leading to the crash?

In the meantime, @ofek you may be able to self-build apple-codesign and use that instead of the release to fix your immediate issue.

[ofek]

Very interesting, I will try that now thanks! BTW how did you SSH into the runner? I didn't know that was possible

[ofek]

Building anew does indeed fix the issue, thank you! I will leave this open until there is a new release however since building adds at least 9 minutes to CI.

[roblabla]

Very interesting, I will try that now thanks! BTW how did you SSH into the runner? I didn't know that was possible

Using tmate, see this commit . It uses mxschmitt/action-tmate which takes care of installing tmate, starting the tmate session, and giving the ssh command in the job log output. Very convenient to debug these kind of issues ^^.

[indygreg]

Thanks for grabbing the crash info, @roblabla!

The crashing stack has all the hallmarks of a memory corruption bug. Crashing code is in apple-bundles crate, which doesn't have any use of unsafe. And walkdir is generally solid as a rock.

My knee jerk is this must be a binary portability issue. Mach-O binary built with newer macOS SDK crashing when run on older macOS machine. This type of thing is known to happen. But it shouldn't happen and most often points to a bug in the macOS SDK, potentially a bug in the clang toolchain in it.

Could you please try grabbing an rcodesign binary from GitHub Actions and see if that still crashes? This may use a newer, fixed macOS SDK to build.

[ofek]

Yes that works I tried multiple times pypa/hatch#900

[indygreg]

While I'd like to understand how this crashed, I haven't seen any other reports of rcodesign crashing. I suspect this has to do with a portable Mach-O binary issue. I'm going to close this due to lack of actionable details.

I should have a new release out shortly.