Deploy more Zebra or zcashd instances on testnet

[teor2345]

Motivation

The Zebra sync tests currently run on mainnet and testnet. Sometimes the testnet nodes are slow or unreliable, so the sync tests are unreliable.

Solution

Deploy about 10 Zebra or zcashd instances on testnet, to increase the average speed and reduce the average latency of testnet.

Open Questions

Right now, we just need a few more Zebra or zcashd instances to stabilise the test network. We can deal with distribution, mining, and network speed later.

Should the zcashd instances be configured to mine blocks?

• Maybe a few should mine on 1 core each

Where should they be located? How fast should they be?

• Reasonably fast
• Distributed across multiple DCs
• In reasonably central internet locations, so Zebra developers have low latency for quick sync tests
• Near our CI and CD runners?

Related Issues

Testnet sync fixes: Issue #1207 PR #1223
Tracking issue #745

[dconnolly]

Maybe this work will scale out / let us deploy zebra nodes on testnet too

[teor2345]

In the most recent team sync, we decided to deploy a few nodes to mainnet and testnet for testing purposes.

We can do this as soon as the hang PR in #1707 is merged.

[yaahc]

From the discord:

quick thoughts

• there is a zcashd Dockerfile somewhere in github.com/zcash repos (https://github.com/zcash/zcash/tree/master/contrib/docker)
• Can copy the broad strokes of the cd.yml workflow, but we may make another google cloud project to put those in, since they are not technically Zebra(edited)
• Our zebrad Dockerfiles have ARG's and stuff to allow mainnet/testnet and other config options, I am not familiar how configurable the existing zcashd Dockerfile is or zcashd period (exploration time! )
  I think all Zfnd engineers have a lot of powers in our gcloud but we can get there when we need to

[teor2345]

• Can copy the broad strokes of the cd.yml workflow, but we may make another google cloud project to put those in, since they are not technically Zebra(edited)

Yeah, we might want to keep the following deployments separate:

• mainnet and testnet
• per-merge-to-main and manual deploys
• Zebra and zcashd

That way, it's easier for everyone to see what we have deployed, and to shut down things that aren't working.