Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

v1.11.3-beta stuck on mainnet - 3x nodes stopped processing blocks #8818

Closed
XertroV opened this issue Jun 6, 2018 · 4 comments
Closed

v1.11.3-beta stuck on mainnet - 3x nodes stopped processing blocks #8818

XertroV opened this issue Jun 6, 2018 · 4 comments
Labels
F3-annoyance 💩 The client behaves within expectations, however this “expected behaviour” itself is at issue. M4-core ⛓ Core client code / Rust. P2-asap 🌊 No need to stop dead in your tracks, however issue should be addressed as soon as possible.
Milestone
2.3

Comments

@XertroV
Copy link
Contributor

XertroV commented Jun 6, 2018 •
edited
Loading

I'm running:

  • Which Parity version?: 1.11.3
  • Which operating system?: Linux
  • How installed?: binaries
  • Are you fully synchronized?: yes (well, was)
  • Which network are you connected to?: ethereum
  • Did you try to restart the node?: yes

Upgraded to v1.11.3 to get around some other issues

3x archive nodes have stopped processing new blocks / or are processing them EXTREMELY slowly (minutes per block)

Things noticed:

  • 1 cpu pinned to 100%
  • Ram usage low (like 5GB out of 60GB w 16GB cache)
  • Restarting node gets past the blockage This returns after some time.
  • It does seem to recover, but it takes like 30 minutes -- see logs right at the bottom of my EU node (which I didn't restart)

Logs

EU

Jun 06 08:05:09 eth-eu-node-01 parity[26827]: 2018-06-06 08:05:09 UTC Imported #5741014 0x45c9…87ad (214 txs, 7.99 Mgas, 773.33 ms, 34.12 KiB)
Jun 06 08:05:36 eth-eu-node-01 parity[26827]: 2018-06-06 08:05:36 UTC Local tx 0x671e9783a1231db42c31d9e1bc3870db1f654c260ef497889290d53062d01275 below minimal gas price accepted
Jun 06 08:05:36 eth-eu-node-01 parity[26827]: 2018-06-06 08:05:36 UTC Local tx 0xbe5e6eda0e23423386883690e60a08390eca8509aa62f17d95a641d12cf56ec0 below minimal gas price accepted
Jun 06 08:05:38 eth-eu-node-01 parity[26827]: 2018-06-06 08:05:38 UTC Imported #5741016 0xa93d…9c83 (135 txs, 7.98 Mgas, 567.20 ms, 23.62 KiB) + another 1 block(s) containing 192 tx(s)
Jun 06 08:05:38 eth-eu-node-01 parity[26827]: 2018-06-06 08:05:38 UTC Local tx 0xb36c0503d7358fe6feadfd9d7b4c493781a0fe8798f93ef900dc4a29e9b45e44 below minimal gas price accepted
Jun 06 08:05:38 eth-eu-node-01 parity[26827]: 2018-06-06 08:05:38 UTC Local tx 0x133bccddffa4e9fc01f62591485dd70d6f62e8f91e7b8cfc2faa905be47bbfc5 below minimal gas price accepted
Jun 06 08:06:04 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:04 UTC   15/50 peers      4 MiB chain    3 MiB db  0 bytes queue   22 KiB sync  RPC:  0 conn,  4 req/s, 1091730 µs
Jun 06 08:06:10 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:10 UTC Local tx 0xb5d4742b6f349cf9ce36316814bec7260b93f34bc928b9c0fbda2f420d352df8 below minimal gas price accepted
Jun 06 08:06:10 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:10 UTC Local tx 0x7c7b689f56998f71b1f8a45ac63fccd3bca3343a8f73e4cacdd2b4f01527b50c below minimal gas price accepted
Jun 06 08:06:22 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:22 UTC Imported #5741017 0x5800…d78a (161 txs, 7.99 Mgas, 432.75 ms, 21.51 KiB)
Jun 06 08:06:36 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:36 UTC Imported #5741017 0x0e7a…6f03 (173 txs, 7.99 Mgas, 301.68 ms, 24.36 KiB)
Jun 06 08:06:40 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:40 UTC Local tx 0x74060b200e5f87d1c64284ab2ed6bbc90399ffd4df9e61d8e038b5da31d830c1 below minimal gas price accepted
Jun 06 08:06:40 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:40 UTC Local tx 0x5aa8f27fa33e2ae1d25ffc2ba4dfcf9dc09a9a33ff3fac699849f8b5df9e4e54 below minimal gas price accepted
Jun 06 08:06:47 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:47 UTC    9/50 peers      4 MiB chain    3 MiB db  0 bytes queue   22 KiB sync  RPC:  0 conn,  4 req/s, 24557777 µs
Jun 06 08:10:51 eth-eu-node-01 parity[26827]: 2018-06-06 08:10:51 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    4 tx/s   0 Mgas/s      0+    4 Qed  #5741026    1/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  3 req/s, 129069939 µs
Jun 06 08:13:16 eth-eu-node-01 parity[26827]: 2018-06-06 08:13:16 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+    4 Qed  #5741026    2/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  4 req/s, 20757182 µs
Jun 06 08:14:54 eth-eu-node-01 parity[26827]: 2018-06-06 08:14:54 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+    4 Qed  #5741026    2/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  4 req/s, 24302027 µs
Jun 06 08:22:19 eth-eu-node-01 parity[26827]: 2018-06-06 08:22:19 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+    4 Qed  #5741026    1/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  3 req/s, 196405603 µs
Jun 06 08:22:41 eth-eu-node-01 parity[26827]: 2018-06-06 08:22:41 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+    4 Qed  #5741026    1/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  4 req/s, 189889122 µs
Jun 06 08:26:23 eth-eu-node-01 parity[26827]: 2018-06-06 08:26:23 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   35 Qed  #5741057    2/50 peers      4 MiB chain    4 MiB db    5 MiB queue   72 KiB sync  RPC:  0 conn,  4 req/s, 54668340 µs
Jun 06 08:26:49 eth-eu-node-01 parity[26827]: 2018-06-06 08:26:49 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   35 Qed  #5741057    1/50 peers      4 MiB chain    4 MiB db    5 MiB queue   51 KiB sync  RPC:  0 conn,  4 req/s, 64163459 µs

SYD

Jun 06 08:14:40 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:14:40 UTC Local tx 0x1578992c165008787939c519ff66fbdd9435a26bc2fbcaa16d6af7b4035a135c below minimal gas price accepted
Jun 06 08:14:50 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:14:50 UTC Imported #5741044 0x8ef8…4b47 (157 txs, 8.00 Mgas, 526.57 ms, 26.04 KiB)
Jun 06 08:14:58 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:14:58 UTC Imported #5741044 0xa603…0e69 (152 txs, 7.98 Mgas, 345.35 ms, 25.49 KiB)
Jun 06 08:15:07 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:15:07 UTC Imported #5741045 0xe16f…257f (177 txs, 7.99 Mgas, 592.54 ms, 37.50 KiB)
Jun 06 08:15:31 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:15:31 UTC Local tx 0x923b4e2feb151fe5e82202ba2cbfb6aae6e52cc3b1b362a900f66f35b6c514c3 below minimal gas price accepted
Jun 06 08:15:33 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:15:33 UTC Local tx 0x033baeb73716a59472401aa63d028c43d4b53d807b24547c351d32eb9388f97c below minimal gas price accepted
Jun 06 08:15:36 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:15:36 UTC Imported #5741046 0x5e2e…e46d (264 txs, 7.99 Mgas, 672.27 ms, 32.54 KiB)
Jun 06 08:15:44 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:15:44 UTC Imported #5741042 0xb263…0d93 (157 txs, 7.98 Mgas, 393.35 ms, 25.67 KiB)
Jun 06 08:16:25 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:16:25 UTC Local tx 0xea915df0b49b3dc1f05a9400b892242ba92edd73e993d20c701992b52856eb31 below minimal gas price accepted
Jun 06 08:16:25 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:16:25 UTC Local tx 0x32b3c22b8c60c87fb90f8ad1de746da58bc892a7ef60f0a8318fcd48dd626580 below minimal gas price accepted
Jun 06 08:19:13 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:19:13 UTC Syncing #5741053 0xcdd7…e19e     0 blk/s   10 tx/s   0 Mgas/s      0+   13 Qed  #5741066    1/50 peers     40 MiB chain    7 MiB db    1 MiB queue   23 KiB sync  RPC:  0 conn,  8 req/s, 2331067 µs
Jun 06 08:19:27 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:19:27 UTC Local tx 0x33f02d87d3983b3774b8cc5d5ca054f9fb4f39f72ebc45f8b66ccefa338295de below minimal gas price accepted
Jun 06 08:19:27 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:19:27 UTC Local tx 0x6e13d3352398f1ea424fc7beed2afef42b72731373525e6b569990dc57cf7333 below minimal gas price accepted
Jun 06 08:23:24 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:23:24 UTC Syncing #5741053 0xcdd7…e19e     0 blk/s    0 tx/s   0 Mgas/s      0+   28 Qed  #5741077    1/50 peers     40 MiB chain    7 MiB db    3 MiB queue   23 KiB sync  RPC:  0 conn,  4 req/s, 79869053 µs
Jun 06 08:24:16 eth-aws-syd-node-02 parity[4217]: 2018-06-06 08:24:16 UTC Syncing #5741053 0xcdd7…e19e     0 blk/s    0 tx/s   0 Mgas/s      0+   29 Qed  #5741079    1/50 peers     40 MiB chain    7 MiB db    3 MiB queue   23 KiB sync  RPC:  0 conn,  4 req/s, 92855225 µs

US

Jun 06 08:13:42 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:13:42 UTC Imported #5741041 0x70a2…1d7e (174 txs, 7.99 Mgas, 576.79 ms, 27.64 KiB)
Jun 06 08:14:14 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:14:14 UTC Imported #5741042 0xb2ea…2fe2 (238 txs, 7.97 Mgas, 164.88 ms, 29.64 KiB)
Jun 06 08:14:22 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:14:22 UTC Imported #5741043 0x3dce…4817 (178 txs, 7.81 Mgas, 401.45 ms, 33.02 KiB)
Jun 06 08:14:42 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:14:42 UTC Imported #5741044 0x8ef8…4b47 (157 txs, 8.00 Mgas, 410.89 ms, 26.04 KiB)
Jun 06 08:15:03 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:15:03 UTC Imported #5741044 0xa603…0e69 (152 txs, 7.98 Mgas, 343.93 ms, 25.49 KiB)
Jun 06 08:15:29 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:15:29 UTC Imported #5741046 0x5e2e…e46d (264 txs, 7.99 Mgas, 478.04 ms, 32.54 KiB)
Jun 06 08:15:50 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:15:50 UTC Imported #5741048 0xbbce…700d (153 txs, 8.00 Mgas, 1113.00 ms, 22.82 KiB) + another 1 block(s) containing 166 tx(s)
Jun 06 08:16:13 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:16:13 UTC Imported #5741049 0xa62b…304a (137 txs, 7.99 Mgas, 1477.94 ms, 21.98 KiB)
Jun 06 08:16:35 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:16:35 UTC Imported #5741050 0xcb55…1a09 (281 txs, 7.98 Mgas, 219.77 ms, 33.92 KiB)
Jun 06 08:16:47 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:16:47 UTC    2/50 peers      5 MiB chain    5 MiB db  0 bytes queue   32 KiB sync  RPC:  0 conn,  2 req/s, 97909904 µs
Jun 06 08:18:26 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:18:26 UTC Syncing #5741055 0xf36d…ca6d     0 blk/s    6 tx/s   0 Mgas/s      0+    6 Qed  #5741061    1/50 peers      6 MiB chain    5 MiB db  682 KiB queue   32 KiB sync  RPC:  0 conn,  4 req/s, 2313952 µs
Jun 06 08:18:41 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:18:41 UTC Syncing #5741055 0xf36d…ca6d     0 blk/s    0 tx/s   0 Mgas/s      0+    6 Qed  #5741061    1/50 peers      6 MiB chain    5 MiB db  682 KiB queue   32 KiB sync  RPC:  0 conn,  6 req/s, 217 µs
Jun 06 08:20:36 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:20:36 UTC Syncing #5741055 0xf36d…ca6d     0 blk/s    0 tx/s   0 Mgas/s      0+    6 Qed  #5741061    1/50 peers      6 MiB chain    5 MiB db  682 KiB queue   32 KiB sync  RPC:  0 conn,  5 req/s, 14973898 µs
Jun 06 08:28:33 eth-aws-nv-node-03 parity[10807]: 2018-06-06 08:28:33 UTC Syncing #5741055 0xf36d…ca6d     0 blk/s    0 tx/s   0 Mgas/s      0+    6 Qed  #5741061    1/50 peers      8 MiB chain    5 MiB db  682 KiB queue   32 KiB sync  RPC:  0 conn,  5 req/s, 23885597 µs

Config (same for all 3)

[rpc]
apis = ["web3", "eth", "net", "rpc"]
hosts = ["*"]
cors = ["*"]
port = 38545
interface = "all"
server_threads = 15
processing_threads = 15

[ipfs]
interface = "0.0.0.0"

[parity]
auto_update = "all"
identity = "securevote-eth-aws-nv-node-03"
release_track = "beta"
no_persistent_txqueue = true

[network]
warp = false
min_peers = 50
max_peers = 500

[footprint]
cache_size_db = 14000
cache_size_queue = 4000
cache_size_state = 4000
cache_size_blocks = 4000
pruning = "archive"
cache_size = 16384

[websockets]
disable = false
port = 8546
interface = "local"
origins = ["*"]
apis = ["web3", "eth", "net", "rpc"]
hosts = ["*"]

[secretstore]
disable = true

[dapps]
disable = true

[ui]
disable = true

[mining]
tx_queue_size = 42949672

EU node - no restart - 30 min to recover

Jun 06 08:06:36 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:36 UTC Imported #5741017 0x0e7a…6f03 (173 txs, 7.99 Mgas, 301.68 ms, 24.36 KiB)
Jun 06 08:06:40 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:40 UTC Local tx 0x74060b200e5f87d1c64284ab2ed6bbc90399ffd4df9e61d8e038b5da31d830c1 below minimal gas price accepted
Jun 06 08:06:40 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:40 UTC Local tx 0x5aa8f27fa33e2ae1d25ffc2ba4dfcf9dc09a9a33ff3fac699849f8b5df9e4e54 below minimal gas price accepted
Jun 06 08:06:47 eth-eu-node-01 parity[26827]: 2018-06-06 08:06:47 UTC    9/50 peers      4 MiB chain    3 MiB db  0 bytes queue   22 KiB sync  RPC:  0 conn,  4 req/s, 24557777 µs
Jun 06 08:10:51 eth-eu-node-01 parity[26827]: 2018-06-06 08:10:51 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    4 tx/s   0 Mgas/s      0+    4 Qed  #5741026    1/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  3 req/s, 129069939 µs
Jun 06 08:13:16 eth-eu-node-01 parity[26827]: 2018-06-06 08:13:16 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+    4 Qed  #5741026    2/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  4 req/s, 20757182 µs
Jun 06 08:14:54 eth-eu-node-01 parity[26827]: 2018-06-06 08:14:54 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+    4 Qed  #5741026    2/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  4 req/s, 24302027 µs
Jun 06 08:22:19 eth-eu-node-01 parity[26827]: 2018-06-06 08:22:19 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+    4 Qed  #5741026    1/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  3 req/s, 196405603 µs
Jun 06 08:22:41 eth-eu-node-01 parity[26827]: 2018-06-06 08:22:41 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+    4 Qed  #5741026    1/50 peers      4 MiB chain    4 MiB db  669 KiB queue   23 KiB sync  RPC:  0 conn,  4 req/s, 189889122 µs
Jun 06 08:26:23 eth-eu-node-01 parity[26827]: 2018-06-06 08:26:23 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   35 Qed  #5741057    2/50 peers      4 MiB chain    4 MiB db    5 MiB queue   72 KiB sync  RPC:  0 conn,  4 req/s, 54668340 µs
Jun 06 08:26:49 eth-eu-node-01 parity[26827]: 2018-06-06 08:26:49 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   35 Qed  #5741057    1/50 peers      4 MiB chain    4 MiB db    5 MiB queue   51 KiB sync  RPC:  0 conn,  4 req/s, 64163459 µs
Jun 06 08:31:24 eth-eu-node-01 parity[26827]: 2018-06-06 08:31:24 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   35 Qed  #5741057    1/50 peers      4 MiB chain    4 MiB db    5 MiB queue   51 KiB sync  RPC:  0 conn,  5 req/s, 65811129 µs
Jun 06 08:32:05 eth-eu-node-01 parity[26827]: 2018-06-06 08:32:05 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   35 Qed  #5741057    1/50 peers      4 MiB chain    4 MiB db    5 MiB queue   51 KiB sync  RPC:  0 conn,  4 req/s, 64655081 µs
Jun 06 08:35:09 eth-eu-node-01 parity[26827]: 2018-06-06 08:35:09 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   35 Qed  #5741057    0/50 peers      4 MiB chain    4 MiB db    5 MiB queue   51 KiB sync  RPC:  0 conn,  3 req/s, 60339808 µs
Jun 06 08:41:30 eth-eu-node-01 parity[26827]: 2018-06-06 08:41:30 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   98 Qed  #5741120    1/50 peers      4 MiB chain    4 MiB db   12 MiB queue   51 KiB sync  RPC:  0 conn,  4 req/s, 173380386 µs
Jun 06 08:41:30 eth-eu-node-01 parity[26827]: 2018-06-06 08:41:30 UTC Syncing #5741022 0x81cf…61ed     0 blk/s    0 tx/s   0 Mgas/s      0+   94 Qed  #5741120    1/50 peers      4 MiB chain    4 MiB db   12 MiB queue   51 KiB sync  RPC:  0 conn,  4 req/s, 173113810 µs
Jun 06 08:41:47 eth-eu-node-01 parity[26827]: 2018-06-06 08:41:47 UTC Syncing #5741033 0xa94c…d17f     1 blk/s  301 tx/s  13 Mgas/s      0+   86 Qed  #5741120    1/50 peers      5 MiB chain    5 MiB db   11 MiB queue   51 KiB sync  RPC:  0 conn,  3 req/s, 119674746 µs
Jun 06 08:41:47 eth-eu-node-01 parity[26827]: 2018-06-06 08:41:47 UTC Syncing #5741054 0x20ce…3eaf     1 blk/s  336 tx/s  14 Mgas/s      0+   66 Qed  #5741120    0/50 peers      6 MiB chain    7 MiB db    8 MiB queue   51 KiB sync  RPC:  0 conn,  3 req/s, 119674746 µs
Jun 06 08:41:58 eth-eu-node-01 parity[26827]: 2018-06-06 08:41:58 UTC Syncing #5741076 0xcfd4…bd73     2 blk/s  404 tx/s  18 Mgas/s      0+   42 Qed  #5741120    1/50 peers      7 MiB chain    9 MiB db    5 MiB queue   51 KiB sync  RPC:  0 conn,  4 req/s, 119674746 µs
Jun 06 08:42:21 eth-eu-node-01 parity[26827]: 2018-06-06 08:42:21 UTC Syncing #5741102 0x57af…5144     3 blk/s  519 tx/s  21 Mgas/s      0+   14 Qed  #5741120    1/50 peers      8 MiB chain   11 MiB db    1 MiB queue   51 KiB sync  RPC:  0 conn,  4 req/s, 119674746 µs
Jun 06 08:42:30 eth-eu-node-01 parity[26827]: 2018-06-06 08:42:30 UTC Imported #5741120 0xa2be…07ca (62 txs, 7.97 Mgas, 323.85 ms, 29.22 KiB) + another 1 block(s) containing 23 tx(s)
Jun 06 08:42:41 eth-eu-node-01 parity[26827]: 2018-06-06 08:42:41 UTC Syncing #5741120 0xa2be…07ca     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #5741120    2/50 peers     10 MiB chain   12 MiB db  0 bytes queue   51 KiB sync  RPC:  0 conn, 11 req/s, 202 µs
Jun 06 08:43:06 eth-eu-node-01 parity[26827]: 2018-06-06 08:43:06 UTC Syncing #5741143 0xfaec…7a5f     2 blk/s  371 tx/s  18 Mgas/s      0+    7 Qed  #5741151    1/50 peers     11 MiB chain   13 MiB db  968 KiB queue   51 KiB sync  RPC:  0 conn, 22 req/s, 793 µs
Jun 06 08:43:06 eth-eu-node-01 parity[26827]: 2018-06-06 08:43:06 UTC Imported #5741151 0xf904…907d (237 txs, 8.00 Mgas, 1505.57 ms, 38.40 KiB) + another 2 block(s) containing 298 tx(s)
Jun 06 08:43:18 eth-eu-node-01 parity[26827]: 2018-06-06 08:43:18 UTC Imported #5741153 0x8ead…bbcc (144 txs, 7.98 Mgas, 446.88 ms, 25.43 KiB)
Jun 06 08:43:35 eth-eu-node-01 parity[26827]: 2018-06-06 08:43:35 UTC Syncing #5741153 0x8ead…bbcc     0 blk/s    7 tx/s   0 Mgas/s      0+    0 Qed  #5741153    5/50 peers     11 MiB chain   14 MiB db  0 bytes queue   51 KiB sync  RPC:  0 conn, 10 req/s, 207 µs
Jun 06 08:43:41 eth-eu-node-01 parity[26827]: 2018-06-06 08:43:41 UTC Syncing #5741153 0x8ead…bbcc     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #5741153    3/50 peers     11 MiB chain   14 MiB db  0 bytes queue   53 KiB sync  RPC:  0 conn,  8 req/s, 7107 µs
Jun 06 08:43:46 eth-eu-node-01 parity[26827]: 2018-06-06 08:43:46 UTC Syncing #5741153 0x8ead…bbcc     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #5741153    3/50 peers     11 MiB chain   14 MiB db  0 bytes queue   53 KiB sync  RPC:  0 conn,  9 req/s, 160 µs
Jun 06 08:43:56 eth-eu-node-01 parity[26827]: 2018-06-06 08:43:56 UTC Syncing #5741153 0x8ead…bbcc     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed  #5741155    6/50 peers     11 MiB chain   14 MiB db  0 bytes queue   51 KiB sync  RPC:  0 conn,  8 req/s, 136 µs
Jun 06 08:44:10 eth-eu-node-01 parity[26827]: 2018-06-06 08:44:10 UTC Syncing #5741157 0x616e…d1d1     0 blk/s   71 tx/s   3 Mgas/s      0+    0 Qed  #5741151    3/50 peers     11 MiB chain   14 MiB db  0 bytes queue   51 KiB sync  RPC:  0 conn,  7 req/s, 2916261 µs
Jun 06 08:44:11 eth-eu-node-01 parity[26827]: 2018-06-06 08:44:11 UTC Local tx 0x290899cd3956d4951b53b2c97bd466ff26b346de4ff815fe84a9151acfef1f0c below minimal gas price accepted
Jun 06 08:44:11 eth-eu-node-01 parity[26827]: 2018-06-06 08:44:11 UTC Local tx 0x420eaba30f85ca563dcc421c22f67d3ae1d7e620d80c1dbec0c23deb62da1e79 below minimal gas price accepted
Jun 06 08:44:19 eth-eu-node-01 parity[26827]: 2018-06-06 08:44:19 UTC Local tx 0xba1f3f711269700e53bdc1532b25823521d0a68f45ccea7727367b04ce4b0b12 below minimal gas price accepted
Jun 06 08:44:19 eth-eu-node-01 parity[26827]: 2018-06-06 08:44:19 UTC Local tx 0x6c99bd8a3383ed6bf2150a0006b1fbdf909f35abf62ef12480a4f356d4a9d5ef below minimal gas price accepted

Btw - all those transactions you see there, a sample is here: https://pastebin.com/Se2K7nsH

They all seem to be going to 0xba7435a4b4c747e0101780073eeda872a69bdcd4 (some airdrop token)
@Tbaut Tbaut added F3-annoyance 💩 The client behaves within expectations, however this “expected behaviour” itself is at issue. M4-core ⛓ Core client code / Rust. labels Jun 6, 2018
@Tbaut Tbaut added this to the 1.12 milestone Jun 6, 2018
@tomusdrw
Copy link
Collaborator

tomusdrw commented Jun 6, 2018

@XertroV

  1. It seems that you are experiencing Massive increase in CPU usage with 1.11.1 #8696 that causes peers to drop to 0-1, that prevents the node from syncing. As a workaround please downgrade to stable version of Parity (currently 1.10.x, which I believe you got auto-upgraded from).

  2. Regarding transactions, it seems that they originate from an address that you have a private key for imported in your node. Such transactions are identified as local and accepted even if they are below gas price.
    It seems that your node is exposed to outside world, so anyone can send a transaction using it. Is that expected?

@XertroV
Copy link
Contributor Author

XertroV commented Jun 6, 2018

@tomusdrw - cool, thanks for the looking over things

As a workaround please downgrade to stable version of Parity (currently 1.10.x, which I believe you got auto-upgraded from).

Actually just upgraded to 1.11.3 to get around #8814 which I think was caused by a large tx backlog. It seems like no_persistent_txqueue = true fixed that (maybe) though.

Will try downgrading and swapping back to stable stream. The only thing I care about is reliability of historical access (not so great today) and having the node be responsive to RPC requests.

Regarding transactions, it seems that they originate from an address that you have a private key for imported in your node.

> rpc.modules
{
  eth: "1.0",
  net: "1.0",
  rpc: "1.0",
  web3: "1.0"
}
> eth.accounts
[]

I wouldn't have thought there'd be a way to do that given the above..

Re the txs though (since they seem to be heavily correlated with my problems today)

  • Is there a way to avoid relaying zero-gas-price transactions? (not sure if the mining settings will help here)
  • Is there anything obvious that I should prevent people doing?
  • Any way to look into this more? I can't find a way to turn on much logging or even get a debug.log type file

It seems that your node is exposed to outside world, so anyone can send a transaction using it. Is that expected?

Yes. Essentially we need random historical access to token balances. Although our needs are very specific I don't think there's an alternative to archive mode, AFAIK (that goes for geth and parity). If there is please let me know. Running archive nodes is expensive (esp with a 1.2 Tb database that needs to be on an SSD)

@tomusdrw
Copy link
Collaborator

tomusdrw commented Jun 6, 2018

Oh, so actually transactions received over eth_sendRawTransaction are treated as local (even though we don't have a private key for them).
That was the case for 1.10.x I believe, we just didn't print info message for that.

I've created a separate issue for that: #8820

@XertroV XertroV mentioned this issue Jun 7, 2018
Closed
@5chdn 5chdn added the P2-asap 🌊 No need to stop dead in your tracks, however issue should be addressed as soon as possible. label Jun 12, 2018
@5chdn 5chdn modified the milestones: 2.0, 2.1 Jul 17, 2018
@5chdn 5chdn modified the milestones: 2.1, 2.2 Sep 11, 2018
@5chdn 5chdn modified the milestones: 2.2, 2.3 Oct 29, 2018
@Tbaut
Copy link
Contributor

Tbaut commented Nov 14, 2018

followed-up #8820

@Tbaut Tbaut closed this as completed Nov 14, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
F3-annoyance 💩 The client behaves within expectations, however this “expected behaviour” itself is at issue. M4-core ⛓ Core client code / Rust. P2-asap 🌊 No need to stop dead in your tracks, however issue should be addressed as soon as possible.
Projects
None yet
Milestone
2.3
Development

No branches or pull requests
4 participants
@tomusdrw @XertroV @5chdn @Tbaut