polkadot 0.9.37 appears to break --no-private-ipv4

[briancolecoinmetrics]

Problem

We just deployed 0.9.37, and our hosting provider promptly emailed to ask why we were running port scans, with logs showing the just-upgraded machine hitting ports 30333-30335 on RFC1918 addresses, even though we explicitly run polkadot with --no-private-ipv4.

General Info

• It would help if you submit info about the system you are running, e.g.: operating system, kernel version, amount of available memory and swap, etc.
  We built with https://github.com/coinmetrics/fullnode/blob/master/fullnodes/polkadot/polkadot-0.9.37.nix to produce container image https://hub.docker.com/layers/coinmetrics/polkadot/0.9.37/images/sha256-9680864e184e4e495f0275d50a2ec4d49388b260aab66fa6559add99eed4b265?context=explore which is run in docker on Ubuntu. (I can provide more details if it's relevant)
• Logs could be very helpful. If possible, submit the whole log. Please format it as code blocks.

2023-01-18 20:06:02 Parity Polkadot
2023-01-18 20:06:02 ✌️ version 0.9.37-unknown
2023-01-18 20:06:02 ❤️ by Parity Technologies <[email protected]>, 2017-2023
2023-01-18 20:06:02 📋 Chain specification: Polkadot
2023-01-18 20:06:02 🏷 Node name: coinmetrics
2023-01-18 20:06:02 👤 Role: FULL
2023-01-18 20:06:02 💾 Database: RocksDb at /opt/data/chains/polkadot/db/full
2023-01-18 20:06:02 ⛓ Native runtime: polkadot-9370 (parity-polkadot-0.tx20.au0)
2023-01-18 20:06:26 🏷 Local node identity is:REDACTED
2023-01-18 20:06:26 🔍 Discovered new external address for our node: /ip4/REDACTED/tcp/30333/p2p/REDACTED
2023-01-18 20:06:26 💻 Operating system: linux
2023-01-18 20:06:26 💻 CPU architecture: x86_64
2023-01-18 20:06:26 💻 Target environment: gnu
2023-01-18 20:06:26 💻 CPU: AMD EPYC 7502P 32-Core Processor
2023-01-18 20:06:26 💻 CPU cores: 32
2023-01-18 20:06:26 💻 Memory: 128755MB
2023-01-18 20:06:26 💻 Kernel: 5.4.0-72-generic
2023-01-18 20:06:26 💻 Linux distribution: Ubuntu 20.04.5 LTS
2023-01-18 20:06:26 💻 Virtual machine: no
2023-01-18 20:06:26 📦 Highest known block at #13860213
2023-01-18 20:06:26 〽️ Prometheus exporter started at 0.0.0.0:9615
2023-01-18 20:06:26 Running JSON-RPC HTTP server: addr=0.0.0.0:9933, allowed origins=["*"]
2023-01-18 20:06:26 Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=["*"]
2023-01-18 20:06:26 🏁 CPU score: 724.75 MiBs
2023-01-18 20:06:26 🏁 Memory score: 15.38 GiBs
2023-01-18 20:06:26 🏁 Disk score (seq. writes): 646.38 MiBs
2023-01-18 20:06:26 🏁 Disk score (rand. writes): 157.47 MiBs
2023-01-18 20:06:26 ✨ Imported #13860214 (0x2353…9845)
2023-01-18 20:06:26 ✨ Imported #13860215 (0x2d02…8e51)
2023-01-18 20:06:26 ✨ Imported #13860216 (0x1e6a…5f38)
2023-01-18 20:06:26 ✨ Imported #13860217 (0x2e42…f567)
2023-01-18 20:06:26 ✨ Imported #13860218 (0x74de…eb24)
2023-01-18 20:06:26 🔍 Discovered new external address for our node: /ip4/REDACTED/tcp/30333/ws/p2p/REDACTED
2023-01-18 20:06:27 Accepting new connection 1/100
2023-01-18 20:06:31 ✨ Imported #13860219 (0xd340…5e1a)
2023-01-18 20:06:31 💤 Idle (10 peers), best: #13860219 (0xd340…5e1a), finalized #13860211 (0x12cd…bef5), ⬇ 131.8kiB/s ⬆ 36.0kiB/s
2023-01-18 20:06:36 💤 Idle (31 peers), best: #13860219 (0xd340…5e1a), finalized #13860216 (0x1e6a…5f38), ⬇ 24.7kiB/s ⬆ 10.4kiB/s
2023-01-18 20:06:36 ✨ Imported #13860220 (0x222f…102e)

(after that it goes into the usual Idle/Import loop)

• Describe the role your node plays, e.g. validator, full node or light client.
  Log says "full" and that looks right.
• Any command-line options were passed?

    command: >-
      --base-path /opt/data
      --chain polkadot
      --pruning archive
      --public-addr /ip4/$HOSTIP/tcp/30333
      --no-private-ipv4
      --rpc-external
      --ws-external
      --prometheus-external
      --rpc-cors all
      --name coinmetrics
      --disable-log-color
      --execution native 
      --wasm-execution Compiled

(where $HOSTIP is templated in on each server)

[bkchr]

CC @paritytech/networking

[stakeworld]

I can confirm, got an abuse report where i was "port scanning" al kind of 10.x.x.x and 192.168.x.x ranges, after reverting to the old binary no more complaints. Maybe related to libp2p update #6500 ?

[bkchr]

Yeah, I also suspect the update. We are already looking into it!

[bkchr]

Should be fixed by: paritytech/substrate#13185

[briancolecoinmetrics]

Any idea when a release with the fix might be available? Considering that 0.9.37 was labeled

you should upgrade in a timely manner

we'd really like to update, but we can't run a release that gets our servers flagged for abuse.

[bkchr]

It should come with 0.9.38. There is no real need to upgrade to 0.9.37. So staying on 0.9.36 should be safe!

[ksmnetwork]

Iptables can be in use here for rfc1918 ranges:

iptables -I OUTPUT -o <NIC> -d 172.16.0.0/12 -j REJECT
iptables -I OUTPUT -o <NIC> -d 10.0.0.0/8 -j REJECT
iptables -I OUTPUT -o <NIC> -d 192.168.0.0/16 -j REJECT
iptables -I OUTPUT -o <NIC> -d 100.64.0.0/10 -j REJECT