Some DNS seeders and block explorers don't show Zebra nodes

[teor2345]

Motivation

In #1844, we deployed a few zcashd instances on testnet, to make Zebra's testnet tests more reliable. But those nodes aren't showing up in the ECC's or the Foundation's DNS seeders.

As far as we can tell, the nodes are configured correctly, they are synced to tip, their listener ports are open, and they are advertising their external IPv4 addresses. But they don't show up in the DNS of either seeder implementation.

Also, Zebra nodes don't show up in some other DNS seeders or blockchain explorers.

Known Zebra Issues

Bugs to fix

• Send the state tip height in Version messages #4875

Things to investigate

• Try pointing a seeder at a long-lived Zebra instance feat(net): return peer metadata from connect_isolated functions #4870 (comment)

• zcashd seems to ignore peers from zebrad: Replace the Seeder zcash/zcash#3881 (comment)

Other Protocol Mismatches:

• version.addr_from is typically dummy bytes, but Zebra sends the actual details: https://en.bitcoin.it/wiki/Protocol_documentation#version

List of Block Explorers

Some block explorers show up to 250 nodes.

There are about 150-200 nodes on mainnet that Zebra instances will connect to.
(Recent Zebra versions are limited to 75 outbound and 50 inbound connections, but this is configurable.)

Zebra might connect to fewer nodes because it has strict latency limits.

Zebra Nodes

This block explorer shows about 65 nodes, but it includes Zebra:

• https://explorer.zecmate.com/network

No Zebra Nodes

This block explorer shows about 250 nodes:

• https://blockchair.com/zcash/nodes (click "Load More" to see all user agents)

These block explorers show connected nodes only:

• https://zcashblockexplorer.com/nodes

Currently down:

• https://explorer.testnet.z.cash/

Completed

Make Zebra have long-term connections to peers:

• Stop closing connections on unexpected messages, Credit: Equilibrium #3120

DNS seeder bugs:

• work out why some DNS seeders return peers with outdated network protocol versions
  • this is a bug in the Foundation's DNS seeder: Enforce the minimum protocol version dnsseeder#10

Metrics:

• Add connection user-agent metrics to Zebra #2871

Optional

We could investigate why DNS seeders ignore zebrad nodes:

Infrastructure:

• try running zcashd on a different network, to see if Google is blocking cryptocurrency traffic
• try to run a Zebra node with the initial testnet addresses set to some of those zcashd nodes, to make sure they respond to requests correctly

Code Review:

• check the zcashd version of our nodes, and the versions accepted by the DNS seeders

[str4d]

This is the list of conditions enforced by zcash-seeder on peers in order for their IPs to be given out:

• The peer's port must be 8233 on mainnet or 18233 on testnet (DNS only gives out IPs, so the port needs to be predictable).
• The NODE_NETWORK service bit must be set.
• The peer's IP must be routable.
• The peer's client version must be at least REQUIRE_VERSION (currently 170015 in the repo, 170014 on my own seeder).
• The peer's block height must be at least 0 (we could bump this to require peers to be closer to the tip, but don't currently do so).
• The peer must be reliably reachable (this is implemented by tracking the success or failure of each connection attempt, with an exponential decay of each attempt's weight).

However, before we get to the above checks, the seeder needs to be able to connect to the peer and perform a very minimal handshake and protocol query:

• version / verack
• getaddr / addr

[teor2345]

I'm pretty sure our mainnet instances satisfy all these criteria.

But they're not configured with their external IP, and Zebra doesn't have external IP auto-detection. So they'll be advertising 0.0.0.0:(default). That might be the issue here. (We'd need to implement #1893, or make changes to our deployment configs.)

It's also possible that Google Cloud is filtering cryptocurrency traffic.

[teor2345]

We could test by configuring a publicly routable Zebra instance with the correct external IP address.
And ideally outside Google cloud, to eliminate that possibility.

[mpguerra]

Could this be related to #794 at all?

[conradoplg]

Could this be related to #794 at all?

I think it's not, the current value seems to follow the rules

zebra/zebra-network/src/constants.rs

Line 112 in 8120e8a

pub const USER_AGENT: &str = "/Zebra:1.0.0-alpha.18/";

But we can double check.

[mpguerra]

#2380 might help us diagnose this one

[teor2345]

We could add user-agent metrics to Zebra.
It's a very small change, just copy and tweak the node version metrics.

This would show us if DNS seeders or block explorers are connecting to Zebra.

[teor2345]

I spoke to someone who runs a Zebra mainnet node with high uptime.
It looks like their setup should work on DNS seeders and block explorers.

According to the rules posted above: #1904 (comment)

So it seems likely that Zebra is implementing some undocumented part of the network protocol differently.
And that's enough to get it rejected from seeders and block explorers.

We might have to do low-level network diagnostics, or look at the detailed logs from a DNS seeder.

[conradoplg]

Zecmate is showing Zebra nodes: https://explorer.zecmate.com/network

[mpguerra]

Updated Blockchair/Blockchair.Support#527 to reference zebra nodes seen in zecmate

[teor2345]

This is blocked by an investigation of how block explorers handle the Zcash network protocol.

[teor2345]

In zcash/zcash#3881 (comment) , we found out that zcashd seems to ignore peers from zebrad.

We should try to find out why.

[softminus]

In zcash/zcash#3881 (comment) , we found out that zcashd seems to ignore peers from zebrad.

To clarify, this was not from zebrad, this was from https://github.com/superbaud/zcash-cotyledon/blob/main/src/main.rs; which only answers Peers requests.

@str4d said here that

zcashd does not anticipate the existence of network peers that only respond to getaddr messages, because that is not how the seeders interact with the network. If such a peer type were desired, it would need a corresponding service bit (to indicate to peers that it does not serve as a regular peer).

so I don't think this necessarily reflects a problem with zebrad.

[softminus]

I ran rm ~/.zcash/peers.dat; ./src/zcashd -dnsseed=0 -seednode=127.0.0.1 -port=12345 on a machine with a running zebrad (not fully synced to chaintip, if that matters) and the zcashd was able to get peers from zebrad.

[softminus]

https://github.com/eqlabs/ziggurat/blob/main/src/tests/conformance/peering.rs#L20-L48 might also be related/helpful

[teor2345]

In zcash/zcash#3881 (comment) , we found out that zcashd seems to ignore peers from zebrad.

To clarify, this was not from zebrad, this was from https://github.com/superbaud/zcash-cotyledon/blob/main/src/main.rs; which only answers Peers requests.

It's still using a zebra-network stack, which is one possible cause of the DNS seeder issues.

Thanks for letting us know that zebrad's Peers requests work for zcashd.

[teor2345]

This seems to be fixed for now, if we find issues with a specific block explorer or seeder, we can open another ticket with those details.