Fix Santa sync protocol for versions of Santa >= 2024.6

[pmarkowsky]

to: @natesinger @mike-flowers-airbnb
cc: @airbnb/rudolph-maintainers

Background

In the Santa 2024.6 release the parsing of the JSON in the sync protocol was handed to the protobuf library which uses the Protobuf to JSON mapping. This had the side effect of making Santa more strict about the protocol fields than it was before.

The cursor field in the RuleDownload request is expected to be an opaque field of that's a string. However Rudolph is currently serializing the cursor object into the field directly in the request and the response. This mismatch causes Rudolph to be incompatible with Santa for versions >= 2024.6.

Changes

• This fixes Rudolph does not sync with Santa agents starting with v2024.6 #52
• This PR changes how the Cursor field is handled by the RuleDownload Request and Response structs
• First the request is parsed directly and then the cursor field is parsed again into the Cursor struct
• For responses, the Cursor struct is first serialized to a json string then added to the Response struct.

Testing

• Unit tests have been updated
• Testing was performed by users who'd applied the patch to their versions of rudolph. See this chat in the #santa channel on the mac admins slack.

Manual Testing Steps

• Download and install Santa version 2024.9
• Apply the patch to rudolph and setup an instance of Rudolph
• Configure santa to use your Rudolph instance as a sync service set SyncBaseURL
• Run santactl sync from the terminal

[radsec]

Starting to review this, going to propagate this to my test environment and will merge/approve once a sync occurs successfully.

[radsec]

Testing confirmed to work on Santa versions prior to 2024.6 running 2024.2 and upgraded to google/santa-2024.9

Test Logs:

Last login: Sun Nov 17 11:12:41 on console
ryan_diers@small-xray-two-vm ~ % sudo santactl status
Password:
>>> Daemon Info
  Mode                      | Monitor
  Transitive Rules          | Yes
  Log Type                  | file
  File Logging              | Yes
  USB Blocking              | No
  On Start USB Options      | None
  Watchdog CPU Events       | 0  (Peak: 11.67%)
  Watchdog RAM Events       | 0  (Peak: 92.84MB)
>>> Cache Info
  Root cache count          | 28
  Non-root cache count      | 0
>>> Database Info
  Binary Rules              | 2964
  Certificate Rules         | 533
  TeamID Rules              | 12
  SigningID Rules           | 16
  Compiler Rules            | 295
  Transitive Rules          | 0
  Events Pending Upload     | 1
>>> Static Rules
  Rules                     | 28
>>> Watch Items
  Enabled                   | No
>>> Sync Info
  Sync Server               | https://rad-rudolph.XXX.XXX/
  Clean Sync Required       | No
  Last Successful Full Sync | 2024/11/17 11:18:34 -0800
  Last Successful Rule Sync | 2024/11/17 11:18:34 -0800
  Push Notifications        | Disconnected
  Bundle Scanning           | No
ryan_diers@small-xray-two-vm ~ % sudo santactl sync --clean-all
Password:
Preflight starting
Clean All sync requested by user
Performing request, attempt 1 (of 5 maximum)...
Server Trust: /O=(null)/OU=(null)/CN=rad-rudolph.XXX.XXX/SHA-1=XXX
Clean sync requested by server
Preflight complete
Event upload starting
Event upload complete
Rule download starting
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 16 rules
Performing request, attempt 1 (of 5 maximum)...
Received 0 rules
Processed 3116 rules
Rule download complete
Postflight starting
Performing request, attempt 1 (of 5 maximum)...
Postflight complete
Sync completed successfully
ryan_diers@small-xray-two-vm ~ % sudo santactl status
>>> Daemon Info
  Mode                      | Lockdown
  Log Type                  | file
  File Logging              | Yes
  USB Blocking              | No
  On Start USB Options      | None
  Watchdog CPU Events       | 0  (Peak: 11.67%)
  Watchdog RAM Events       | 0  (Peak: 113.69MB)
>>> Cache Info
  Root cache count          | 2
  Non-root cache count      | 0
>>> Database Info
  Binary Rules              | 2568
  Certificate Rules         | 522
  TeamID Rules              | 10
  SigningID Rules           | 16
  Compiler Rules            | 276
  Transitive Rules          | 0
  Events Pending Upload     | 2
>>> Static Rules
  Rules                     | 28
>>> Watch Items
  Enabled                   | No
>>> Sync Info
  Sync Server               | https://rad-rudolph.XXX.XXX/
  Clean Sync Required       | No
  Last Successful Full Sync | 2024/11/17 11:26:24 -0800
  Last Successful Rule Sync | 2024/11/17 11:26:24 -0800
  Push Notifications        | Disconnected
  Bundle Scanning           | No
ryan_diers@small-xray-two-vm ~ % sudo santactl version
santad          | 2024.2 (build 605404402)
santactl        | 2024.2 (build 605404402)
SantaGUI        | 2024.2 (build 605404402)
ryan_diers@small-xray-two-vm ~ % sudo santactl version
santad          | 2024.9 (build 674285143)
santactl        | 2024.9 (build 674285143)
SantaGUI        | 2024.9 (build 674285143)
ryan_diers@small-xray-two-vm ~ % sudo santactl sync --clean-all
Preflight starting
Clean All sync requested by user
Performing request, attempt 1 (of 5 maximum)...
Server Trust: /O=(null)/OU=(null)/CN=rad-rudolph.XXX.XXX/SHA-1=XXX
Clean sync requested by server
Preflight complete
Event upload starting
Event upload complete
Rule download starting
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 50 rules
Performing request, attempt 1 (of 5 maximum)...
Received 16 rules
Performing request, attempt 1 (of 5 maximum)...
Received 0 rules
Processed 3116 rules
Rule download complete
Postflight starting
Performing request, attempt 1 (of 5 maximum)...
Postflight complete
Sync completed successfully
ryan_diers@small-xray-two-vm ~ % sudo santactl sync
Preflight starting
Performing request, attempt 1 (of 5 maximum)...
Server Trust: /O=(null)/OU=(null)/CN=rad-rudolph.XXX.XXX/SHA-1=XXX
Preflight complete
Event upload starting
Performing request, attempt 1 (of 5 maximum)...
Uploaded 3 events
Event upload complete
Rule download starting
Performing request, attempt 1 (of 5 maximum)...
Received 0 rules
Performing request, attempt 1 (of 5 maximum)...
Received 0 rules
Rule download complete
Postflight starting
Performing request, attempt 1 (of 5 maximum)...
Postflight complete
Sync completed successfully
ryan_diers@small-xray-two-vm ~ % sudo santactl status
>>> Daemon Info
  Mode                      | Lockdown
  Log Type                  | file
  File Logging              | Yes
  USB Blocking              | No
  On Start USB Options      | None
  Watchdog CPU Events       | 0  (Peak: 1.33%)
  Watchdog RAM Events       | 0  (Peak: 55.80MB)
>>> Cache Info
  Root cache count          | 31
  Non-root cache count      | 0
>>> Database Info
  Binary Rules              | 2568
  Certificate Rules         | 522
  TeamID Rules              | 10
  SigningID Rules           | 16
  CDHash Rules              | 0
  Compiler Rules            | 276
  Transitive Rules          | 0
  Events Pending Upload     | 28
>>> Static Rules
  Rules                     | 28
>>> Watch Items
  Enabled                   | No
>>> Sync Info
  Sync Server               | https://rad-rudolph.XXX.XXX/
  Clean Sync Required       | No
  Last Successful Full Sync | 2024/11/17 11:29:29 -0800
  Last Successful Rule Sync | 2024/11/17 11:29:29 -0800
  Push Notifications        | Disconnected
  Bundle Scanning           | No

[radsec]

LGTM. Thanks again for shipping this update!