feat(zebra-checkpoints): make zebra-checkpoints work for zebrad backend

[oxarbitrage]

Motivation

Zebra has an utility tool to create checkpoints. Until now this tool only work if zcashd is used as the backend.

This PR make the changes to make it work with zebrad as the backend while zcashd is still supported.

Close #5852

Solution

When i started on this i was only expecting to make changes to the zebra-checkpoints binary, however i realized later we were going to need changes in the getblock rpc method.

The binary is expecting height, hash and size fields of a getblock call with verbosity = 1. Currently (and because ligthwalletd only make use of the tx field of the object response), we are not returning any additional data.

this PR updated the getblock rpc method to return the additional data needed to make the tool work.

Also, in the zebra-checkpoints we changed the use of getblockcount to the blocks field of getblockchaininfo call as suggested in the proposed changes section of the ticket.

Review

This is optional and low priority. It can wait until people get back. Anyone can review.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

• Update some docs mentioning this in zebra book or checkpoint README ?
• Do manual tests of differences between zcashd and zebrad generated checkpoints.
• Compare the speed of using zebrad against zcashd as the backend.

[oxarbitrage]

I did some manual test for comparison and performance.

In regards to time, zebra started a bit faster but they both ended up taking pretty much the same amount of time to get a full list of up to date checkpoints in the Mainnet. Here what i did:

zebra (i have a synchronized chain at port 6666):

$ time ./target/debug/zebra-checkpoints --cli /media/zcash/zcash/src/zcash-cli  -- -rpcport=6666 > mainnet_checkpoints_zebrad.txt

real	381m56,356s
user	60m57,259s
sys	36m19,635s
$ 

zcashd (synchronized and running in default rpc port):

$ time ./target/debug/zebra-checkpoints --cli /media/zcash/zcash/src/zcash-cli -- -rpccookiefile="/media/chain/zcash/.cookie" > mainnet_checkpoints_zcashd.txt
real	399m10,090s
user	67m53,213s
sys	37m6,481s
$

A comparassion of this 2 generated files show that they are exactly the same.

$ diff --color mainnet_checkpoints_zebrad.txt mainnet_checkpoints_zcashd.txt 
$

[codecov]

Codecov Report

Merging #5894 (3117bb7) into main (67194c4) will decrease coverage by 0.12%.
The diff coverage is 0.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5894      +/-   ##
==========================================
- Coverage   78.01%   77.90%   -0.12%     
==========================================
  Files         312      312              
  Lines       38997    39022      +25     
==========================================
- Hits        30425    30401      -24     
- Misses       8572     8621      +49     

[arya2]

Looks good to me.

[arya2]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[teor2345]

This looks good, and I'd really like to merge it. But it seems to cause a performance issue with lightwalletd.

I spent some time trying to work out how to fix it, let me know what you think of my suggestions!

[mpguerra]

If we can't get this merged with minimal changes, e.g timeboxed to 1 day + 1 more round of reviews then we should park it for now

[oxarbitrage]

If we can't get this merged with minimal changes, e.g timeboxed to 1 day + 1 more round of reviews then we should park it for now

I think this is easy enough for a fix in one day and another round of reviews. Working on it now.

[arya2]

Thanks for the workaround to the performance issue!

Performance issues were fixed

[teor2345]

Looks good, just doing some testing to make sure all the backends produce the same results.

[teor2345]

Looks good, just doing some testing to make sure all the backends produce the same results.

All 3 backend/node combinations produce the same results for new checkpoints, I'll run a full set of checkpoints overnight to be sure. I think it's worth doing this test because checkpoints are consensus-critical.

[teor2345]

The full checkpoint files were exactly the same as each other, and they didn't change any of the existing checkpoints.