Document how to run and query the results of the blockchain scanning MVP

[mpguerra]

Motivation

We want a "naive" zebra user to be able to use zebra-scan to query the Zcash blockchain with a viewing key and get a list of the transactions related with it

Scope

The documentation should provide details of:

• how to compile and run zebra with the zebra-scan feature enabled
• how to specify the viewing key(s), e.g location and format for the config file containing the keys
• how to query the database to retrieve data about the transactions found

It would be nice if we could also post the same instructions to the blog and include screenshots and examples, e.g using the ZECPages viewing key.

The incomplete draft document is here:
https://github.com/ZcashFoundation/zebra/blob/main/book/src/user/shielded-scan.md

[teor2345]

Before we do the instructions, let's decide if #8048 is a blocker for the MVP?

If we put secret keys in the existing config file, we might need to change its permissions, but that is a breaking change if someone has deliberately made their file group or world (all users) readable.

[teor2345]

So far I think there are two ways to read the file:

• using the read results method in the Rust API

  • this code needs to run inside Zebra, or while Zebra is shut down
  • if we want it to run while Zebra is running, we can add a zebra-state argument that opens a secondary database
  • we could write a zebrad command that takes a key (or a key index for security), and prints the heights and transaction IDs

• using ldb to dump the database

  • open the database in secondary read-only mode using command-line arguments
  • use --from and --to arguments to get the entire range of results for the key, by changing the height to ffffff
  • text processing to split up the stored data into individual 64-hex digit hashes
  • this is a less usable method

[mpguerra]

Hey team! Please add your planning poker estimate with Zenhub @arya2 @oxarbitrage @teor2345 @upbqdn

[teor2345]

In the Zebra sprint planning we decided on ldb and #8050.

[teor2345]

Here's some experiments with the zecpages key:

Showing all the transactions:

ldb --try_load_options --db="$HOME/.cache/zebra-custom/private-scan/v1/mainnet" --secondary_path= --column_family=sapling_tx_ids --hex scan
...
0x7A78766965777331713064757974676371717171707172653236776B6C343567767777776437303678773630386875636D7666616C72373539656A7766377173686A663572396161373332337A756C767A36706C68747470356D6C747163677339743033396378326430396D67713035747336336E387533356879763668396E633963747171747565327537636572326D716567756E75756C71326C7568713379776A637A333579796C6A657761346D676B676A7A79667768366672366A6430647A64343467686B306E78647632686E76346A356E7866777632347277646D676C6C68653070383536387367717439636B74303276326B786635616874716C3673306C746A706B636B77386774796D787478757539676372307377767A1B6CF50001 : 0x79986D7CE609A4369F6FDABAF86274863155DCADF84DD9D2CAA71B66505E8492
0x7A78766965777331713064757974676371717171707172653236776B6C343567767777776437303678773630386875636D7666616C72373539656A7766377173686A663572396161373332337A756C767A36706C68747470356D6C747163677339743033396378326430396D67713035747336336E387533356879763668396E633963747171747565327537636572326D716567756E75756C71326C7568713379776A637A333579796C6A657761346D676B676A7A79667768366672366A6430647A64343467686B306E78647632686E76346A356E7866777632347277646D676C6C68653070383536387367717439636B74303276326B786635616874716C3673306C746A706B636B77386774796D787478757539676372307377767A1B6FC50004 : 0x06639CCF024275A6B3244B5860AFA7EFF2791EF9CD91762C71A9150CB0475E85

Transaction 79986D7CE609A4369F6FDABAF86274863155DCADF84DD9D2CAA71B66505E8492 is a Sapling transaction at index 1 in block 1797365. This matches the transaction location 1B6CF50001:

• 1B6CF5 is height 1797365
• 0001 is index 1

https://zcashblockexplorer.com/transactions/79986D7CE609A4369F6FDABAF86274863155DCADF84DD9D2CAA71B66505E8492
https://zcashblockexplorer.com/blocks/1797365

Similarly for the next entry:
https://zcashblockexplorer.com/transactions/06639CCF024275A6B3244B5860AFA7EFF2791EF9CD91762C71A9150CB0475E85
https://zcashblockexplorer.com/blocks/1798085

Other arguments we might want:

• --from=startkey
• --to=endkey
• --max_keys=n

Security

The keys are listed in the database output, so our examples should remove them and not show them in the terminal.

[teor2345]

These appear to correspond to these two zecpages posts:
https://zecpages.com/z/all/6

After accounting for timezones, the zecpages server appears to be showing times that are 6-10 minutes later than the block times.

[teor2345]

The incomplete draft document is here:
https://github.com/ZcashFoundation/zebra/blob/main/book/src/user/shielded-scan.md

[mpguerra]

The incomplete draft document is here: https://github.com/ZcashFoundation/zebra/blob/main/book/src/user/shielded-scan.md

I can't see this in the zebra book yet, when does this get published?

[teor2345]

The incomplete draft document is here: main/book/src/user/shielded-scan.md

I can't see this in the zebra book yet, when does this get published?

It should be deployed every time we edit the docs, for example here is the job for the release changes:
https://github.com/ZcashFoundation/zebra/actions/runs/7175306562/job/19538368848

Which are in the book:
https://zebra.zfnd.org/user/install.html

So maybe there's some issue with deploying new files, or we need to link to the new md file from somewhere.

Let's see if PR #8095 fixes it.

[teor2345]

Yes, that works, seems like there needs to be a link to the doc for it to publish.

[mpguerra]

I think we need to expand on the instructions in the zebra book quite a bit (https://zebra.zfnd.org/user/shielded-scan.html), e.g

• Exactly how to build and install, which commands to run, how to ensure that they are running the latest
• How to add keys to be scanned, the format, etc...
• How to query the results, can possibly just point to the zebra-utils/README where the tool for reading the results is explained.
• Generally resolve the existing TODOs in the document.

This issue also includes writing a blog post using the ZecPages viewing key as a case study with the same set of instructions.

[oxarbitrage]

I think we need to expand on the instructions in the zebra book quite a bit (https://zebra.zfnd.org/user/shielded-scan.html), e.g

• Exactly how to build and install, which commands to run, how to ensure that they are running the latest
• How to add keys to be scanned, the format, etc...
• How to query the results, can possibly just point to the zebra-utils/README where the tool for reading the results is explained.
• Generally resolve the existing TODOs in the document.

This issue also includes writing a blog post using the ZecPages viewing key as a case study with the same set of instructions.

I made all the above in #8114 except for the blog post, which i think i was changed to a forum post. We can track that somewhere else.