Segmentation Fault During PIRProcessDatabase Execution

[alexlin-gogolook]

Environment

• macOS 15.0
• Xcode 16.0
• SwiftProtobuf 1.28.1

Description

I followed the steps outlined in the TestingInstructions.md. When I executed the command:

ConstructDatabase --icon-directory icons input.txtpb block.binpb identity.binpb

I encountered an error. It seems that there is an EXC_BAD_ACCESS happening on line 61 of ConstructDatabase.swift at the call to example.textFormatString().

To troubleshoot, I commented out the exampleIdentities and directly initialized inputIdentities instead of decoding it from textFormatString. After this change, I was able to generate block.binpb and identity.binpb.

However, when I followed the instructions further and executed:

PIRProcessDatabase block-config.json

I encountered the following error:

2024-09-30T18:03:26+0800 info PIRProcessDatabase : [PIRProcessDatabase] Processing database with configuration: {
  "algorithm" : "mulPir",
  "cuckooTableConfig" : {
    "bucketCount" : {
      "allowExpansion" : {
        "expansionFactor" : 1.1,
        "targetLoadFactor" : 0.9
      }
    },
    "hashFunctionCount" : 2,
    "maxEvictionCount" : 100,
    "maxSerializedBucketSize" : 1024,
    "multipleTables" : true,
    "slotCount" : 255
  },
  "inputDatabase" : "block.binpb",
  "keyCompression" : "noCompression",
  "outputDatabase" : "block-SHARD_ID.bin",
  "outputPirParameters" : "block-SHARD_ID.params.txtpb",
  "rlweParameters" : "n_4096_logq_27_28_28_logt_5",
  "sharding" : {
    "entryCountPerShard" : 1000
  },
  "trialsPerShard" : 5,
  "useMaxSerializedBucketSize" : false
}
2024-09-30T18:03:26+0800 info PIRProcessDatabase : [PIRProcessDatabase] Sharded database into 1 shards
2024-09-30T18:03:26+0800 info PIRProcessDatabase : [PIRProcessDatabase] Processing shard 0 with 3 rows
2024-09-30T18:03:26+0800 info PIRProcessDatabase : [PIRProcessDatabase] Created cuckoo table CuckooTableInformation(entryCount: 3, bucketCount: 2, emptyBucketCount: 1, loadFactor: 0.017089844)
2024-09-30T18:03:26+0800 info PIRProcessDatabase : [PIRProcessDatabase] Validating shard 0
zsh: segmentation fault  PIRProcessDatabase block-config.json

Any insight into what might be causing the segmentation fault or the initial EXC_BAD_ACCESS would be greatly appreciated!

[fboemer]

Any insight into what might be causing the segmentation fault or the initial EXC_BAD_ACCESS would be greatly appreciated!

Can you run this command with lldb? I.e., run

lldb PIRProcessDatabase block-config.json

Then, in the (lldb) prompt, enter run,

(lldb) run

Hopefully, you'll see the crash, and be able to enter backtrace.

(lldb) backtrace

which will return a backtrace of the issue.
If you can post the backtrace here, this will help us diagnose what the issue is.

[alexlin-gogolook]

Thank you for your reply.

Backtrace:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
  * frame #0: 0x000000019473975c libswiftCore.dylib`swift_getAssociatedTypeWitness + 28
    frame #1: 0x000000010025bd44 PIRProcessDatabase`BinaryEncodingSizeVisitor.visitMapField<SwiftProtobuf.ProtobufUInt64, HomomorphicEncryptionProtobuf.Apple_SwiftHomomorphicEncryption_V1_SerializedKeySwitchKey>(fieldType=SwiftProtobuf._ProtobufMessageMap<τ_0_0, τ_0_1>, value=Swift.Dictionary<Swift.UInt64, HomomorphicEncryptionProtobuf.Apple_SwiftHomomorphicEncryption_V1_SerializedKeySwitchKey> @ 0x0000000115e04300, fieldNumber=1) at <stdin>:0 [opt]
    frame #2: 0x000000010025ca38 PIRProcessDatabase`protocol witness for Visitor.visitMapField<A, B>(fieldType:value:fieldNumber:) in conformance BinaryEncodingSizeVisitor at <compiler-generated>:0 [opt]
    frame #3: 0x000000010017cb24 PIRProcessDatabase`Apple_SwiftHomomorphicEncryption_V1_SerializedGaloisKey.traverse<SwiftProtobuf.BinaryEncodingSizeVisitor>(visitor=(serializedSize = 0)) at apple_swift_homomorphic_encryption_v1_he.pb.swift:694:19 [opt]
    frame #4: 0x000000010017cd50 PIRProcessDatabase`protocol witness for Message.traverse<A>(visitor:) in conformance Apple_SwiftHomomorphicEncryption_V1_SerializedGaloisKey at <compiler-generated>:0 [opt]
    frame #5: 0x000000010028cabc PIRProcessDatabase`Message.serializedDataSize() at Message+BinaryAdditions.swift:74:9 [opt]
    frame #6: 0x000000010025a550 PIRProcessDatabase`BinaryEncodingSizeVisitor.visitSingularMessageField<τ_0_0>(value=<unavailable>, fieldNumber=<unavailable>) at BinaryEncodingSizeVisitor.swift:351:33 [opt]
    frame #7: 0x000000010025c768 PIRProcessDatabase`protocol witness for Visitor.visitSingularMessageField<A>(value:fieldNumber:) in conformance BinaryEncodingSizeVisitor at <compiler-generated>:0 [opt]
    frame #8: 0x000000010017e0fc PIRProcessDatabase`closure #1 in Apple_SwiftHomomorphicEncryption_V1_SerializedEvaluationKey.traverse<SwiftProtobuf.BinaryEncodingSizeVisitor>(, visitor=(serializedSize = 0)) at apple_swift_homomorphic_encryption_v1_he.pb.swift:800:19 [opt]
    frame #9: 0x000000010017e028 PIRProcessDatabase`Apple_SwiftHomomorphicEncryption_V1_SerializedEvaluationKey.traverse<SwiftProtobuf.BinaryEncodingSizeVisitor>(visitor=(serializedSize = 0)) at apple_swift_homomorphic_encryption_v1_he.pb.swift:799:9 [opt]
    frame #10: 0x000000010017e354 PIRProcessDatabase`protocol witness for Message.traverse<A>(visitor:) in conformance Apple_SwiftHomomorphicEncryption_V1_SerializedEvaluationKey at <compiler-generated>:0 [opt]
    frame #11: 0x000000010028c810 PIRProcessDatabase`Message.serializedBytes<HomomorphicEncryptionProtobuf.Apple_SwiftHomomorphicEncryption_V1_SerializedEvaluationKey>(partial:options:) at Message+BinaryAdditions.swift:74:9 [opt]
    frame #12: 0x000000010028c7e0 PIRProcessDatabase`Message.serializedBytes<HomomorphicEncryptionProtobuf.Apple_SwiftHomomorphicEncryption_V1_SerializedEvaluationKey>(partial=<unavailable>, options=(useDeterministicOrdering = false)) at Message+BinaryAdditions.swift:41:28 [opt]
    frame #13: 0x0000000100217b08 PIRProcessDatabase`EvaluationKey.size() [inlined] generic specialization <HomomorphicEncryptionProtobuf.Apple_SwiftHomomorphicEncryption_V1_SerializedEvaluationKey> of SwiftProtobuf.Message.serializedData(partial: Swift.Bool) throws -> Foundation.Data at <compiler-generated>:0 [opt]
    frame #14: 0x0000000100217ab8 PIRProcessDatabase`EvaluationKey.size() at ConversionPir.swift:230:33 [opt]
    frame #15: 0x00000001001a0fbc PIRProcessDatabase`merged generic specialization <HomomorphicEncryption.Bfv<Swift.UInt64>> of PrivateInformationRetrieval.ProcessKeywordDatabase.ShardValidationResult.description() throws -> Swift.String + 504
    frame #16: 0x000000010019d764 PIRProcessDatabase`specialized ProcessDatabase.process<A>(config=<unavailable>) at ProcessDatabase.swift:386:57 [opt]
    frame #17: 0x00000001001a2cd0 PIRProcessDatabase`ProcessDatabase.run() at ProcessDatabase.swift:419:17 [opt]
    frame #18: 0x00000001001a2f98 PIRProcessDatabase`protocol witness for ParsableCommand.run() in conformance ProcessDatabase at <compiler-generated>:0 [opt]
    frame #19: 0x00000001001da6c8 PIRProcessDatabase`specialized static ProcessDatabase.$main() [inlined] generic specialization <PIRProcessDatabase.ProcessDatabase> of static ArgumentParser.ParsableCommand.main(Swift.Optional<Swift.Array<Swift.String>>) -> () at <compiler-generated>:0 [opt]
    frame #20: 0x00000001001da67c PIRProcessDatabase`specialized static ProcessDatabase.$main() [inlined] generic specialization <PIRProcessDatabase.ProcessDatabase> of static ArgumentParser.ParsableCommand.main() -> () at <compiler-generated>:0 [opt]
    frame #21: 0x00000001001da67c PIRProcessDatabase`specialized static ProcessDatabase.$main() at <compiler-generated>:299:1 [opt]
    frame #22: 0x00000001001a3140 PIRProcessDatabase`PIRProcessDatabase_main [inlined] static PIRProcessDatabase.ProcessDatabase.$main() -> () at ProcessDatabase.swift:0 [opt]
    frame #23: 0x00000001001a313c PIRProcessDatabase`PIRProcessDatabase_main at ProcessDatabase.swift:0 [opt]
    frame #24: 0x00000001833bc274 dyld`start + 2840

[fboemer]

@alexlin-gogolook , thanks for reporting the issue.
Can you try the latest main (which updates to https://github.com/apple/swift-homomorphic-encryption/releases/tag/1.0.1)?

[alexlin-gogolook]

TL;DR:

• PIRProcessDatabase appears to be working fine.
• ConstructDatabase is still broken (arm64/Swift 6.0).
• PIRService still cannot build (arm64/Swift 6.0).
• Running it on Docker was always feasible. (aarch64/Swift 6.0.1 )

---

Here is more information. The following is my environment (it has not changed since I first reported the issue):

swift-driver version: 1.115
Apple Swift version 6.0 (swiftlang-6.0.0.9.10 clang-1600.0.26.2)
Target: arm64-apple-macosx15.0

Then, before revert 'Adopt Swift 6', running the same tests under Docker seemed fine, and test number recognition worked correctly. Below is the Docker environment, using the latest Swift image (as of 2024/10/1).

Swift version 6.0.1 (swift-6.0.1-RELEASE)
Target: aarch64-unknown-linux-gnu

I tried the latest main branch, but building with
swift package experimental-install -c release --product PIRService fails.
This is the same issue as in #36.

My workaround is to copy Hummingbird.MemoryPersistDriver.swift into the PIRService directory and add import Hummingbird.

Then, running
ConstructDatabase --icon-directory icons input.txtpb block.binpb identity.binpb
throws an error.
Here’s the backtrace:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
  * frame #0: 0x000000019473975c libswiftCore.dylib`swift_getAssociatedTypeWitness + 28
    frame #1: 0x00000001002710d0 ConstructDatabase`TextFormatEncodingVisitor.visitMapField<SwiftProtobuf.ProtobufString, ConstructDatabase.InputCallIdentity>(fieldType=SwiftProtobuf._ProtobufMessageMap<τ_0_0, τ_0_1>, value=Swift.Dictionary<Swift.String, ConstructDatabase.InputCallIdentity> @ 0x000000013ef060e0, fieldNumber=1) at TextFormatEncodingVisitor.swift:666:11 [opt]
    frame #2: 0x0000000100271568 ConstructDatabase`protocol witness for Visitor.visitMapField<A, B>(fieldType:value:fieldNumber:) in conformance TextFormatEncodingVisitor at <compiler-generated>:0 [opt]
    frame #3: 0x00000001000ec140 ConstructDatabase`InputIdentities.traverse<SwiftProtobuf.TextFormatEncodingVisitor>(visitor=SwiftProtobuf.TextFormatEncodingVisitor @ 0x000000016fdfe428) at InputIdentities.pb.swift:142:19 [opt]
    frame #4: 0x00000001000ec298 ConstructDatabase`protocol witness for Message.traverse<A>(visitor:) in conformance InputIdentities at <compiler-generated>:0 [opt]
    frame #5: 0x000000010024c224 ConstructDatabase`Message.textFormatString(options=<unavailable>) at Message+TextFormatAdditions.swift:52:12 [opt]
    frame #6: 0x00000001000e9ec8 ConstructDatabase`specialized static ConstructDatabase.exampleIdentities.getter [inlined] generic specialization <ConstructDatabase.InputIdentities> of SwiftProtobuf.Message.textFormatString() -> Swift.String at <compiler-generated>:0 [opt]
    frame #7: 0x00000001000e9ea0 ConstructDatabase`specialized static ConstructDatabase.exampleIdentities.getter at ConstructDatabase.swift:61:24 [opt]
    frame #8: 0x00000001000ea994 ConstructDatabase`specialized ConstructDatabase.init() [inlined] static ConstructDatabase.ConstructDatabase.exampleIdentities.getter : Swift.String at ConstructDatabase.swift:0 [opt]
    frame #9: 0x00000001000ea990 ConstructDatabase`specialized ConstructDatabase.init() at ConstructDatabase.swift:64:6 [opt]
    frame #10: 0x00000001000e8f28 ConstructDatabase`protocol witness for ParsableArguments.init() in conformance ConstructDatabase [inlined] ConstructDatabase.ConstructDatabase.init() -> ConstructDatabase.ConstructDatabase at <compiler-generated>:0 [opt]
    frame #11: 0x00000001000e8f20 ConstructDatabase`protocol witness for ParsableArguments.init() in conformance ConstructDatabase at <compiler-generated>:30:8 [opt]
    frame #12: 0x00000001000515a4 ConstructDatabase`specialized ArgumentSet.init(type=ConstructDatabase.ConstructDatabase, visibility=ArgumentParser.ArgumentVisibility @ 0x000000016fdfe8c4, parent=nil) at ParsableArguments.swift:287:52 [opt]
    frame #13: 0x0000000100024d40 ConstructDatabase`CommandParser.parseCurrent(_:) [inlined] ArgumentParser.ArgumentSet.init(_: ArgumentParser.ParsableArguments.Type, visibility: ArgumentParser.ArgumentVisibility, parent: Swift.Optional<ArgumentParser.InputKey>) -> ArgumentParser.ArgumentSet at <compiler-generated>:0 [opt]
    frame #14: 0x0000000100024d24 ConstructDatabase`CommandParser.parseCurrent(_:) [inlined] ArgumentParser.LenientParser.init(command=ConstructDatabase.ConstructDatabase, split=ArgumentParser.SplitArguments @ 0x0000600000497920) -> ArgumentParser.LenientParser at ArgumentSet.swift:214:24 [opt]
    frame #15: 0x0000000100024d24 ConstructDatabase`CommandParser.parseCurrent(split=ArgumentParser.SplitArguments @ 0x000000016fdff0c0) at CommandParser.swift:145:18 [opt]
    frame #16: 0x00000001000251f0 ConstructDatabase`CommandParser.descendingParse(split=ArgumentParser.SplitArguments @ 0x000000016fdff0c0) at CommandParser.swift:184:31 [opt]
    frame #17: 0x0000000100024314 ConstructDatabase`CommandParser.parse(arguments=0 values) at CommandParser.swift:243:11 [opt]
    frame #18: 0x0000000100053500 ConstructDatabase`static ParsableCommand.parseAsRoot(arguments=<unavailable>) at ParsableCommand.swift:67:23 [opt]
    frame #19: 0x00000001000ea258 ConstructDatabase`specialized static ConstructDatabase.$main() [inlined] generic specialization <ConstructDatabase.ConstructDatabase> of static ArgumentParser.ParsableCommand.main(Swift.Optional<Swift.Array<Swift.String>>) -> () at <compiler-generated>:0 [opt]
    frame #20: 0x00000001000ea234 ConstructDatabase`specialized static ConstructDatabase.$main() [inlined] generic specialization <ConstructDatabase.ConstructDatabase> of static ArgumentParser.ParsableCommand.main() -> () at <compiler-generated>:0 [opt]
    frame #21: 0x00000001000ea234 ConstructDatabase`specialized static ConstructDatabase.$main() at <compiler-generated>:29:1 [opt]
    frame #22: 0x00000001000e8fcc ConstructDatabase`ConstructDatabase_main [inlined] static ConstructDatabase.ConstructDatabase.$main() -> () at ConstructDatabase.swift:0 [opt]
    frame #23: 0x00000001000e8fc8 ConstructDatabase`ConstructDatabase_main at ConstructDatabase.swift:0 [opt]
    frame #24: 0x00000001833bc274 dyld`start + 2840

This is the same error as before, so I used the same workaround by generating InputIdentities from code instead of the text format string input. After this, PIRProcessDatabase seems to work correctly.

[fboemer]

Thanks @alexlin-gogolook, seems like one more change is needed: #39

[alexlin-gogolook]

@fboemer Thanks this issue was resolved