feat(tee): use hex serialization for RPC responses

[pbeza]

What ❔

Following Anton's suggestion, we have switched to hex serialization for API/RPC requests and responses. Previously, we used default JSON serialization for Vec, which resulted in a lengthy comma-separated list of integers.

This change standardizes serialization, making it more efficient and reducing the size of the responses. The previous format, with a series of comma-separated integers for pubkey-like fields, looked odd.

Then:

curl -X POST\
     -H "Content-Type: application/json" \
     --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [491882, "sgx"] }' \
        https://mainnet.era.zksync.io
{"jsonrpc":"2.0","result":[{"attestation":[3,0,2,0,0,0,0,0,10,<dozens of comma-separated integers here>

Now:

$ curl -X POST \
       -H "Content-Type: application/json" \
       --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [1, "sgx"] }' \
          http://localhost:3050
{"jsonrpc":"2.0","result":[{"l1BatchNumber":1,"teeType":"sgx","pubkey":"0506070809","signature":"0001020304","proof":"0a0b0c0d0e","provedAt":"2024-09-16T11:53:38.253033Z","attestation":"0403020100"}],"id":1}

This change needs to be deployed in lockstep with:
matter-labs/teepot#209.

Why ❔

To improve user experience (UX) and optimize the RPC response size.

Checklist

• PR title corresponds to the body of PR (we generate changelog entries from PRs).
• Tests for the changes have been added / updated.
• Documentation comments have been added / updated.
• Code has been formatted via zk fmt and zk lint.

[haraldh]

is that backwards compatible?

[pbeza]

is that backwards compatible?

Definitely not, which means we need to update both the prover and verifier in sync. I see three options here:

1. Since, realistically, no one is relying on the unstable_getTeeProofs TEE RPC endpoint yet (note the unstable keyword), we can use this time to iterate quickly and ignore breaking compatibility changes, postponing API/RPC versioning until later.
2. We can introduce versioning for the API/RPC interface.
3. We can quietly support both formats in the TEE proof verifier using a series of if statements, without explicit versioning.

Any other ideas or preferences? It’d be great to know how API/RPC versioning is handled in other parts of the project. Links to relevant code showing how we do versioning would be super helpful! (cc @popzxc)

[haraldh]

I don't understand this change... nobody is looking at the raw data anyway. And potential http compression makes the byte saving negligible

[pbeza]

I don't understand this change... nobody is looking at the raw data anyway. And potential http compression makes the byte saving negligible

This change was suggested by Anton (@nithalak), and I think it’s purely a matter of UX/DevEx preference. Comma-separated integers in JSON as a representation of a signature are a bit uncommon – they’re usually represented in hex or Base64. I’m not sure how we typically handle byte arrays across other RPC endpoints, but we should probably align with the existing approach. Let’s have @popzxc @RomanBrodetski be the tiebreaker.

[RomanBrodetski]

This change will make serialization more efficient and reduce the size of the responses.

I don't think we care about serialization efficiency in this case, but I agree it doesn't look right to have those integer arrays. Indeed raw data is not often observed, but I imagine us having to document it - would look less awkward with hex serialization. Not saying it's critical, but since we have it i implemented here already - please finalize the draft and let's merge

[pbeza]

tl;dr Feel free to re-run the tests later and merge. ✅

---

@haraldh @RomanBrodetski This PR is ready to be merged, but the flaky state of the tests is preventing them from passing (as discussed here):

429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Regarding backward compatibility: If we deploy PR matter-labs/teepot#209 in lockstep with this one, we should be fine. The only tool consuming the JSON fields that are now being serialized differently is the one from matter-labs/teepot#209.

[pbeza]

JFYI: PR #2912 should hopefully fix the test flakiness that is blocking this PR.

[pbeza]

@haraldh @RomanBrodetski It seems all the tests have finally passed. Please feel free to review/merge.