testnet-sanity panic: solana-jsonrpchttp.worker3 panic: Address already in use

[mvines]

#960 appears to have caused an issue with testnet-sanity. See https://buildkite.com/solana-labs/testnet-sanity/builds/871#33874f3a-91d2-4ccf-b033-522ca0883939

INFO 2018-08-18T22:28:42.715169657Z: solana::fullnode: creating bank...
--
  | INFO 2018-08-18T22:28:44.214224018Z: solana::fullnode: processing ledger...
  | INFO 2018-08-18T22:28:49.156724865Z: solana::fullnode: processed 4321 ledger...
  | INFO 2018-08-18T22:28:49.156766150Z: solana::fullnode: creating networking stack...
  | INFO 2018-08-18T22:28:49.156777967Z: solana::fullnode: starting... local gossip address: 0.0.0.0:9001 (advertising 10.138.15.237:9001)
  | thread 'thread 'solana-jsonrpchttp.worker3' panicked at '' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 98, kind: AddrInUse, message: "Address already in use" }Server initialization awaits local address.: "SendError(..)"', ', libcore/result.rslibcore/result.rs::945945::55

In the mean time, I've set

NO_VALIDATOR_SANITY=1

to keep us going. Ref: https://github.com/solana-labs/solana/blob/master/ci/testnet-sanity.sh#L54

[CriesofCarrots]

@mvines , just to make sure I understand properly, testnet-sanity ran fine with #960 incorporated four times (builds 863, 865, 867, 869), correct?

So far I've only been able to reproduce the panic by binding to the RPC port with another application before starting a fullnode. Leader and validators on the same machine just yoink the port from each other.

I wonder if this testnet-sanity build might have been running into something at the end of the previous testnet-deploy build, since they started 2min apart instead of the usual 5min. Would you be completely opposed to re-enabling the validator sanity test for half a day or so to see how often the issue recurs in your scheduled builds?

That said, I will add in a non-panicky way to handle an RPC port conflict regardless...

[mvines]

Yep, no problem. I've removed the NO_VALIDATOR_SANITY=1 again.
There's a couple other issues in sanity though (ledger verification and wallet airdrops are not 100%) so the noise level is a little high right now :-/

You're right though, this issue doesn't seem to reproduce 100% of the time. I've actually only seen it occur once

[mvines]

Happened again at https://buildkite.com/solana-labs/testnet-sanity/builds/959#7f2829f1-7ea2-41bc-9dfe-558ac3124295