Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

Parity taking days to (fail to) sync on warp-barrier #8531

Closed
LDMFD opened this issue May 2, 2018 · 40 comments
Closed

Parity taking days to (fail to) sync on warp-barrier #8531

LDMFD opened this issue May 2, 2018 · 40 comments
Labels
F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. P5-sometimesoon 🌲 Issue is worth doing soon.
Milestone

Comments

@LDMFD
Copy link

LDMFD commented May 2, 2018

I'm running:

  • Which Parity version?: v1.10.2-beta-f4ae813-20180423/x86_64-linux-gnu/rustc1.25.0
  • Which operating system?: Ubuntu 16.06 (recently installed)
  • How installed?: sudo bash <(curl https://get.parity.io -kL)
  • Are you fully synchronized?: no
  • Which network are you connected to?: ethereum (default)
  • Did you try to restart the node?: yes

Parity taking days to (fail to) sync despite multiple restarts as follows:

  1. Initial run started with canonical parity - cancelled after a couple of days, deleted ~/.local/share/io.parity.ethereum

  2. parity --mode active --tracing off --pruning fast --db-compaction ssd --cache-size 1024 - again cancelled, deleted parity dir and then:

  3. parity --db-compaction ssd --cache-size 1024 --warp-barrier 5520000

This last execution was left running for 48 hours and the last line of the log at that point:

2018-05-02 15:52:13 Syncing #4179651 a017…7492 5 blk/s 434 tx/s 28 Mgas/s 0+ 453 Qed #4180107 25/25 peers 66 MiB chain 393 MiB db 43 MiB queue 13 MiB sync RPC: 0 conn, 0 req/s, 44057 µs

My system specs are as follows:

CPU: Intel i7 6700K
GPU: GeForce GTX 1060 6GB
RAM: 16GB DDR4 SDRAM
HD: 128GB M.2 SSD
Internet: 950 Mb/s down, 120 Mb/s up, fiber
Clock is synced. 8GB swapfile is 152M used.

StackExchange discussion here, https://ethereum.stackexchange.com/questions/46950/parity-sync-time-april-2018-many-days-normal

Raising a bug as requested by Tbaut. I would be happy to help with diagnosis and testing.

Thank you,

Darren

@LDMFD
Copy link
Author

LDMFD commented May 2, 2018

Having deleted the data dir and restarted with only parity --warp-barrier 5520000, I'm going to copy here some of the output. Perhaps there will be something obviously wrong here:

darren@dubuntu:~$ parity --warp-barrier 5520000
2018-05-02 16:15:17  Starting Parity/v1.10.2-beta-f4ae813-20180423/x86_64-linux-gnu/rustc1.25.0
2018-05-02 16:15:17  Keys path /home/darren/.local/share/io.parity.ethereum/keys/Foundation
2018-05-02 16:15:17  DB path /home/darren/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
2018-05-02 16:15:17  Path to dapps /home/darren/.local/share/io.parity.ethereum/dapps
2018-05-02 16:15:17  State DB configuration: fast
2018-05-02 16:15:17  Operating mode: active
2018-05-02 16:15:17  Configured for Foundation using Ethash engine
2018-05-02 16:15:17  Updated conversion rate to Ξ1 = US$674.47 (7060217.5 wei/gas)
2018-05-02 16:15:22  Public node URL: enode://1fc338abaac538bbac456806d14515e5478712425f6586b0c384c7d13a83a127d16bf5bfac846f1fb76e69686fb5946eb5a5214d8bd0cb42b5a89350bfac421b@192.168.0.125:30303
2018-05-02 16:15:37  Syncing snapshot 0/1315        #0    6/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 16:15:42  Syncing snapshot 0/1315        #0    6/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 16:15:52  Syncing snapshot 6/1315        #0    7/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 16:15:57  Syncing snapshot 12/1315        #0    7/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
2018-05-02 16:28:12  Syncing snapshot 51/1315        #0   16/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
2018-05-02 17:45:37  Syncing snapshot 51/1315        #0   26/50 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
2018-05-02 19:43:28  Syncing snapshot 51/1315        #0   26/50 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
2018-05-02 22:58:07  Syncing snapshot 51/1315        #0   26/50 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:58:17  Syncing snapshot 51/1315        #0   26/50 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:58:22  Syncing snapshot 51/1315        #0   26/50 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:58:32  Syncing snapshot 51/1315        #0   26/50 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:58:42  Imported #254 47e6…0bb6 (0 txs, 0.00 Mgas, 0.12 ms, 1.05 KiB)
2018-05-02 22:58:42  Updated conversion rate to Ξ1 = US$705.51 (6749592.5 wei/gas)
2018-05-02 22:58:52  Imported #255 c631…3ff9 (0 txs, 0.00 Mgas, 0.34 ms, 0.53 KiB)
2018-05-02 22:59:02  Syncing   #12066 f893…1d37   393 blk/s    0 tx/s   0 Mgas/s  15639+    0 Qed    #27705   23/25 peers   0 bytes chain 56 MiB db 25 MiB queue 6 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:59:12  Syncing   #18179 c2f6…647a   600 blk/s    0 tx/s   0 Mgas/s  17197+ 5370 Qed    #40786   25/25 peers   3 MiB chain 73 MiB db 37 MiB queue 7 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:59:22  Syncing   #24317 31ce…7211   624 blk/s    0 tx/s   0 Mgas/s      0+21558 Qed    #45877   25/25 peers   6 MiB chain 73 MiB db 35 MiB queue 19 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:59:28  Syncing   #26329 46b0…4fc2   315 blk/s    0 tx/s   0 Mgas/s      0+34267 Qed    #60598   25/25 peers   5 MiB chain 73 MiB db 64 MiB queue 8 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:59:35  Syncing   #28419 83f3…e708   306 blk/s    0 tx/s   0 Mgas/s      0+32179 Qed    #60598   25/25 peers   6 MiB chain 73 MiB db 60 MiB queue 8 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:59:42  Syncing   #36826 1acb…d89d  1134 blk/s    0 tx/s   0 Mgas/s      0+23771 Qed    #60598   25/25 peers   4 MiB chain 73 MiB db 46 MiB queue 8 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:59:53  Syncing   #44362 6a31…10c3   725 blk/s    0 tx/s   0 Mgas/s      0+22077 Qed    #66440   24/25 peers   5 MiB chain 73 MiB db 46 MiB queue 11 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 22:59:53  Syncing   #44366 abf2…ca2b     0 blk/s    0 tx/s   0 Mgas/s      0+22073 Qed    #66440   24/25 peers   5 MiB chain 73 MiB db 46 MiB queue 11 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-02 23:00:07  Syncing   #48820 9823…293a   319 blk/s   97 tx/s   2 Mgas/s      0+
...
2018-05-03 02:06:00  Syncing #1660440 7cbb…9818    70 blk/s  332 tx/s  12 Mgas/s      0+ 5100 Qed  #1665543   26/50 peers   6 MiB chain 108 MiB db 41 MiB queue 9 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
2018-05-03 14:33:52    25/25 peers   4 MiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
... 4 days later
...
2018-05-07 17:10:43  Syncing #4592611 c5f3…a764     2 blk/s  186 tx/s   8 Mgas/s      0+  540 Qed  #4593155   25/25 peers   5 MiB chain 257 MiB db 40 MiB queue 58 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:10:53  Syncing #4592623 eb06…01c4     1 blk/s   51 tx/s   2 Mgas/s      0+  528 Qed  #4593155   25/25 peers   5 MiB chain 257 MiB db 40 MiB queue 58 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:11:03  Syncing #4592652 861d…61f0     2 blk/s  242 tx/s  14 Mgas/s      0+  534 Qed  #4593189   25/25 peers   5 MiB chain 257 MiB db 40 MiB queue 72 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:11:13  Syncing #4592707 66ce…c191     5 blk/s  439 tx/s  22 Mgas/s      0+ 1939 Qed  #4594650   25/25 peers   5 MiB chain 257 MiB db 155 MiB queue 59 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:11:23  Syncing #4592734 6ec7…9af8     2 blk/s  262 tx/s  13 Mgas/s      0+ 1915 Qed  #4594650   25/25 peers   5 MiB chain 257 MiB db 152 MiB queue 59 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:11:33  Syncing #4592785 d569…187f     5 blk/s  390 tx/s  21 Mgas/s      0+ 1863 Qed  #4594650   25/25 peers   5 MiB chain 257 MiB db 148 MiB queue 59 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:11:43  Syncing #4592832 ece6…2cf7     4 blk/s  329 tx/s  17 Mgas/s      0+ 1815 Qed  #4594650   25/25 peers   6 MiB chain 258 MiB db 145 MiB queue 59 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
^C2018-05-07 17:11:49  Finishing work, please wait...
darren@dubuntu:~$ parity
2018-05-07 17:12:38  Starting Parity/v1.10.2-beta-f4ae813-20180423/x86_64-linux-gnu/rustc1.25.0
2018-05-07 17:12:38  Keys path /home/darren/.local/share/io.parity.ethereum/keys/Foundation
2018-05-07 17:12:38  DB path /home/darren/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
2018-05-07 17:12:38  Path to dapps /home/darren/.local/share/io.parity.ethereum/dapps
2018-05-07 17:12:38  State DB configuration: fast
2018-05-07 17:12:38  Operating mode: active
2018-05-07 17:12:38  Configured for Foundation using Ethash engine
2018-05-07 17:12:38  Updated conversion rate to Ξ1 = US$760.96 (6257759 wei/gas)
2018-05-07 17:12:44  Public node URL: enode://7e2acbd21a0761079e622d06620c12fe775ee72dcfcb6eab3bd0d78cafff5090a07cf99d14942196943ca23cbf0e247aa53d80d0cebc6451c059c5eec8464999@192.168.0.125:30303
2018-05-07 17:12:49  Syncing snapshot 2/1326  #4592843   23/25 peers   91 KiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:12:59  Syncing snapshot 8/1326  #4592843   25/25 peers   91 KiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
^C2018-05-07 17:13:01  Finishing work, please wait...
^[[Adarren@dubuntu^C
darren@dubuntu:~$ parity --warp-barrier 5520000
2018-05-07 17:13:23  Starting Parity/v1.10.2-beta-f4ae813-20180423/x86_64-linux-gnu/rustc1.25.0
2018-05-07 17:13:23  Keys path /home/darren/.local/share/io.parity.ethereum/keys/Foundation
2018-05-07 17:13:23  DB path /home/darren/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
2018-05-07 17:13:23  Path to dapps /home/darren/.local/share/io.parity.ethereum/dapps
2018-05-07 17:13:23  State DB configuration: fast
2018-05-07 17:13:23  Operating mode: active
2018-05-07 17:13:23  Configured for Foundation using Ethash engine
2018-05-07 17:13:23  Updated conversion rate to Ξ1 = US$761.47 (6253569 wei/gas)
2018-05-07 17:13:28  Public node URL: enode://7e2acbd21a0761079e622d06620c12fe775ee72dcfcb6eab3bd0d78cafff5090a07cf99d14942196943ca23cbf0e247aa53d80d0cebc6451c059c5eec8464999@192.168.0.125:30303
2018-05-07 17:13:33  Syncing snapshot 0/1326  #4592843   22/25 peers   91 KiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:13:43  Syncing snapshot 7/1326  #4592843   24/25 peers   91 KiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
2018-05-07 17:39:33  Syncing snapshot 286/1326  #4592843   24/25 peers   605 KiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
2018-05-07 17:44:18  Syncing snapshot 323/1326  #4592843   25/25 peers   605 KiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:44:28  Syncing snapshot 324/1326  #4592843   25/25 peers   605 KiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:44:38  Syncing snapshot 324/1326  #4592843   24/25 peers   605 KiB chain 65 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:45:49  Syncing #4592847 da0b…438d     0 blk/s    1 tx/s   0 Mgas/s   1151+  731 Qed  #4594750   14/25 peers   627 KiB chain 66 MiB db 98 MiB queue 13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:46:13  Syncing #4592847 da0b…438d     0 blk/s    0 tx/s   0 Mgas/s      0+ 1903 Qed  #4594750   19/25 peers   627 KiB chain 66 MiB db 154 MiB queue 13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 17:46:43  Syncing #4592847 da0b…438d     0 blk/s    0 tx/s   0 Mgas/s      0+ 1899 Qed  #4594750   21/25 peers   627 KiB chain 66 MiB db 153 MiB queue 13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
2018-05-07 23:21:04  Syncing #4638378 9e67…8ecc     1 blk/s  164 tx/s   7 Mgas/s      0+  660 Qed  #4639039   25/25 peers   5 MiB chain 105 MiB db 72 MiB queue 16 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 23:21:14  Syncing #4638390 73c7…480a     1 blk/s  149 tx/s   6 Mgas/s      0+  648 Qed  #4639039   25/25 peers   5 MiB chain 105 MiB db 70 MiB queue 16 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-07 23:21:25  Syncing #4638398 9b2d…04bc     0 blk/s   78 tx/s   4 Mgas/s      0+  640 Qed  #4639039   25/25 peers   5 MiB chain 105 MiB db 69 MiB queue 16 MiB sync  RPC:  0 conn,  0 req/s,   0 µs

@Tbaut Tbaut added F2-bug 🐞 The client fails to follow expected behavior. P5-sometimesoon 🌲 Issue is worth doing soon. M4-core ⛓ Core client code / Rust. labels May 3, 2018
@Tbaut Tbaut added this to the 1.12 milestone May 3, 2018
@Tbaut
Copy link
Contributor

Tbaut commented May 3, 2018

cc: @dvdplm, @ngotchac and @tomusdrw

@JSterling8
Copy link

JSterling8 commented May 6, 2018

Same here. Seems to start with warp sync, then switches to full sync.

  • Which Parity version?: v1.10.2-beta-f4ae813-20180423/x86_64-linux-gnu/rustc1.25.0
  • Which operating system?: Ubuntu 16.06
  • How installed?: Downloaded .deb from GitHub -> Ubuntu Software Manager installed
  • Are you fully synchronized?: no
  • Which network are you connected to?: ethereum (default)
  • Did you try to restart the node?: yes

My system specs are as follows:

CPU: Intel® Core™ i7-7500U CPU @ 2.70GHz × 4
GPU: N/A
RAM: 16GB
HD: 512GB SSD
Internet: 50 Mb/s down, 20 Mb/s up, fiber

Output:

➜  ~ git:(master) ✗ parity --db-compaction ssd --cache-size 1024 --warp-barrier 5520000 ui
2018-05-06 16:52:35  Parity browser interface is deprecated. It's going to be removed in the next version, use standalone Parity UI instead.
2018-05-06 16:52:35  Standalone Parity UI: https://github.com/Parity-JS/shell/releases
2018-05-06 16:52:35  Starting Parity/v1.10.2-beta-f4ae813-20180423/x86_64-linux-gnu/rustc1.25.0
2018-05-06 16:52:35  Keys path /home/jsterling8/.local/share/io.parity.ethereum/keys/Foundation
2018-05-06 16:52:35  DB path /home/jsterling8/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
2018-05-06 16:52:35  Path to dapps /home/jsterling8/.local/share/io.parity.ethereum/dapps
2018-05-06 16:52:35  State DB configuration: fast
2018-05-06 16:52:35  Operating mode: active
2018-05-06 16:52:35  Configured for Foundation using Ethash engine

Open: http://127.0.0.1:8180/#/auth?token=<REDACTED>
to authorize your browser.
Or use the generated token:
<REDACTED>
2018-05-06 16:52:36  Public node URL: enode://3108a412f3b3f8c2001802ca22aa8c81aa8a3f22536d5839379c44354a06d23fd82af268ca2067c1bce5e553890c95da514cc03e6063e5c5c2d10412db5847e8@192.168.1.155:30303
Created new window in existing browser session.
[657:670:0506/165236.412743:ERROR:browser_gpu_channel_host_factory.cc(119)] Failed to launch GPU process.
2018-05-06 16:52:37  Updated conversion rate to Ξ1 = US$774.46 (6148677 wei/gas)
2018-05-06 16:53:01  Syncing snapshot 0/1337        #0    2/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  1 conn,  1 req/s,  51 µs
2018-05-06 16:53:11  Syncing snapshot 2/1337        #0    3/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  1 conn,  1 req/s,  51 µs
...
2018-05-06 16:55:41  Syncing snapshot 21/1337        #0    7/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  1 conn,  2 req/s, 198 µs
2018-05-06 16:55:51  Syncing       #0 d4e5…8fa3     0 blk/s    0 tx/s   0 Mgas/s    123+    0 Qed      #127    4/25 peers   8 KiB chain 3 MiB db 234 KiB queue 73 KiB sync  RPC:  1 conn,  1 req/s, 229 µs
2018-05-06 16:55:56  Syncing    #1524 3d94…7253   304 blk/s    0 tx/s   0 Mgas/s      2+    0 Qed     #1527    6/25 peers   2 MiB chain 11 MiB db 4 KiB queue 419 KiB sync  RPC:  1 conn,  4 req/s, 198 µs
2018-05-06 16:56:06  Syncing    #1781 2f92…6a9a    25 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed     #1781    7/25 peers   2 MiB chain 11 MiB db 0 bytes queue 5 MiB sync  RPC:  1 conn,  3 req/s,  80 µs
2018-05-06 16:56:16  Syncing    #1781 2f92…6a9a     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed     #1781    7/25 peers   2 MiB chain 11 MiB db 0 bytes queue 9 MiB sync  RPC:  1 conn,  1 req/s,  80 µs

@JSterling8
Copy link

CTRL+C'ing then rerunning with the same command appears to be working now. At snapshot 365 of 1337 at present. Will update if it completes or switches back to full syncing again.

@JSterling8
Copy link

Just got a full sync. Total time about 2.5 hours. @cachvico - Instead of deleting ~/.local/share/io.parity.ethereum and restarting, simply CTRL+C then run again with the same params. I used: parity --db-compaction ssd --cache-size 1024 --warp-barrier 5520000

@LDMFD
Copy link
Author

LDMFD commented May 7, 2018

I have restarted first with parity (accidentally) then parity --warp-barrier 5520000. I've updated my log (in the second post in this thread) with the complete output appended.

@JSterling8
Copy link

@cachvico - Does it get further for you now after the restart? Mine didn't complete the first time, but completed on the first restart.

@LDMFD
Copy link
Author

LDMFD commented May 8, 2018

@JSterling8 not really - still only up to #4638589

I've updated my log at #8531 (comment)

@ngotchac
Copy link
Contributor

ngotchac commented May 8, 2018

This issue should be fixed with #8543 (and improved with #8544) 🙂

@LDMFD
Copy link
Author

LDMFD commented May 8, 2018

Great!

Which branch should I build from to pick up the fix? (Or, when's the next release that includes it?)

Thanks

@ngotchac
Copy link
Contributor

ngotchac commented May 8, 2018

@cachvico You can try ng-warp-resume

@LDMFD
Copy link
Author

LDMFD commented May 8, 2018

And we're off! Fingers crossed

./target/release/parity --db-compaction ssd --cache-size 1024 --warp-barrier 5520000

@LDMFD
Copy link
Author

LDMFD commented May 8, 2018

I deleted ~/.local/share/io.parity.ethereum, built from source using ~/.cargo/bin/cargo build --release on branch ng-warp-resume, and invoked ./target/release/parity --db-compaction ssd --cache-size 1024 --warp-barrier 5520000.

Observe that I still seemed to need to hit Ctrl-C and restart, in order for sync from snapshots to work:

$ ./target/release/parity  --db-compaction ssd --cache-size 1024 --warp-barrier 5520000
2018-05-08 12:01:20  Starting Parity/v1.12.0-unstable-ce2bce412-20180507/x86_64-linux-gnu/rustc1.25.0
2018-05-08 12:01:20  Keys path /home/darren/.local/share/io.parity.ethereum/keys/Foundation
2018-05-08 12:01:20  DB path /home/darren/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
2018-05-08 12:01:20  Path to dapps /home/darren/.local/share/io.parity.ethereum/dapps
2018-05-08 12:01:20  State DB configuration: fast
2018-05-08 12:01:20  Operating mode: active
2018-05-08 12:01:20  Configured for Foundation using Ethash engine
2018-05-08 12:01:21  Sending warning alert CloseNotify
2018-05-08 12:01:21  Updated conversion rate to Ξ1 = US$733.34 (6493447.5 wei/gas)
2018-05-08 12:01:26  Public node URL: enode://d270659d99405b9dffc3d397d8e97a4dc84759f2b1beea0af61a809711d21bb059d3f893bb0017c31c1a42b8d01d49625548f71d3b9e3794b7aebf4792dde7d4@192.168.0.125:30303
2018-05-08 12:01:31  Syncing       #0 0xd4e5…8fa3     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed        #0    4/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:01:41  Syncing       #0 0xd4e5…8fa3     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed        #0    3/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:01:51  Syncing       #0 0xd4e5…8fa3     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed        #0    3/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...
... (skip 30 mins)
...
2018-05-08 12:35:21  Syncing       #0 0xd4e5…8fa3     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed        #0   25/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:35:26  Syncing       #0 0xd4e5…8fa3     0 blk/s    0 tx/s   0 Mgas/s      0+    0 Qed        #0   25/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
^C2018-05-08 12:35:31  Finishing work, please wait...
$ 
$ ./target/release/parity  --db-compaction ssd --cache-size 1024 --warp-barrier 5520000
2018-05-08 12:35:33  Starting Parity/v1.12.0-unstable-ce2bce412-20180507/x86_64-linux-gnu/rustc1.25.0
2018-05-08 12:35:33  Keys path /home/darren/.local/share/io.parity.ethereum/keys/Foundation
2018-05-08 12:35:33  DB path /home/darren/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
2018-05-08 12:35:33  Path to dapps /home/darren/.local/share/io.parity.ethereum/dapps
2018-05-08 12:35:33  State DB configuration: fast
2018-05-08 12:35:33  Operating mode: active
2018-05-08 12:35:33  Configured for Foundation using Ethash engine
2018-05-08 12:35:34  Sending warning alert CloseNotify
2018-05-08 12:35:34  Updated conversion rate to Ξ1 = US$733.80 (6489376.5 wei/gas)
2018-05-08 12:35:39  Public node URL: enode://d270659d99405b9dffc3d397d8e97a4dc84759f2b1beea0af61a809711d21bb059d3f893bb0017c31c1a42b8d01d49625548f71d3b9e3794b7aebf4792dde7d4@192.168.0.125:30303
2018-05-08 12:35:49  Syncing snapshot 0/1344        #0    5/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:35:59  Syncing snapshot 0/1344        #0   12/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:36:04  Syncing snapshot 0/1344        #0   12/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:36:14  Syncing snapshot 0/1344        #0   16/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:36:19  Syncing snapshot 1/1344        #0   20/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:36:29  Syncing snapshot 3/1344        #0   22/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:36:34  Syncing snapshot 4/1344        #0   20/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
2018-05-08 12:36:44  Syncing snapshot 4/1344        #0   19/25 peers   8 KiB chain 3 MiB db 0 bytes queue 10 KiB sync  RPC:  0 conn,  0 req/s,   0 µs
...

@LDMFD
Copy link
Author

LDMFD commented May 8, 2018

In fact it seems to choose randomly between snapshot sync and regular sync when restarting after ctrl-c

@ngotchac
Copy link
Contributor

ngotchac commented May 8, 2018

@cachvico Actually the logs saying "Syncing #0" is quite misleading because it could be that the node is just waiting for valid peers to start the snapshot. So the issue here is that it can't find one.
I could see two reasons :

  1. The peer is blocking the connection or not answering the requests
  2. There are already too many other peers connected (and that's something we could/should fix)

Just to be sure, if you could provide -l sync,warp,snapshot,network=trace logs it would be great!

@ngotchac
Copy link
Contributor

ngotchac commented May 8, 2018

(The issue on the mainnet is that currently there seem to be only one peer serving snapshots)

@LDMFD
Copy link
Author

LDMFD commented May 8, 2018

Obviously the $ ./target/release/parity --db-compaction ssd --cache-size 1024 --warp-barrier 5520000 -l sync,warp,snapshot,network=trace command generates gobs of logs - how many minutes-worth would you like me to send?

@ngotchac
Copy link
Contributor

ngotchac commented May 8, 2018

Yeah sure, just the first minute should be fine.

@LDMFD
Copy link
Author

LDMFD commented May 8, 2018

Here we go:

$ ./target/release/parity --db-compaction ssd --cache-size 1024 --warp-barrier 5520000 -l sync,warp,snapshot,network=trace > ../output.txt 2>&1`

output.txt

@ngotchac
Copy link
Contributor

ngotchac commented May 8, 2018

So reading from the logs, it seems that the only peer that could provide a snapshot (0x1d1f7...) closes the TCP connection as soon as a Handshake is initialized (tried 3 times in your run).
I'm not sure why that is, it could be that it already has too many open connections (cc @arkpar ?).
A potential solution would be to delete your ~/.local/share/io.parity.ethereum/chains/ethereum/network folder, and also try running with --no-discovery

@LDMFD
Copy link
Author

LDMFD commented May 8, 2018

I deleted my network folder and restarted with --no-discover.

I've been trying to sync with Parity for 10 days now.. should I restart without --warp-barrier if it has a greater chance of success?

@LDMFD
Copy link
Author

LDMFD commented May 9, 2018

Restarted without warp, in branch ng-warp-resume, and after 8 hours, finally synced!

\o/

@ngotchac
Copy link
Contributor

ngotchac commented May 9, 2018

Daily update : we actually found 2 bugs you might have run into:

  1. The remote node would disconnect while your node was fetching the snapshot manifest, so your node would be stuck indefinitely. This is now fixed in ng-warp-resume Resumable warp-sync / Seed downloaded snapshots #8544
  2. The remote node would disconnect every 3.5 seconds. The fix is here Fix packet count when talking with PAR2 peers #8555 or you can fix it manually by running --no-serve-light

With those 2 fixes you should be able to sync in 1h30 now.

@5chdn
Copy link
Contributor

5chdn commented May 9, 2018

The 2nd one should not be on 1.10 branch.

@folsen
Copy link
Contributor

folsen commented May 21, 2018

Closing this issue as the latest PRs should have addressed it. Reopen if it persists.

@folsen folsen closed this as completed May 21, 2018
@LDMFD
Copy link
Author

LDMFD commented May 23, 2018

I've been using Parity since my successful sync two weeks ago, but didn't have too much luck on a clean restart today.

Can I confirm the fixes are merged into master? I deleted ~/.local/share/io.parity.ethereum, switched to master, pulled, built and restarted with ./target/release/parity --db-compaction ssd --cache-size 1024 --warp-barrier 5650000, but it's not looking too promising - I've attached a log file showing various restarts over a couple of hours.

Thanks a lot

output.txt

@folsen
Copy link
Contributor

folsen commented May 23, 2018

@cachvico The first two times you're shutting your node down while it's trying to download a snaphot? And it's actually successfully picking up where it left off. The last time I think you're being hit by a bug that's only on master that's a deadlock issue, it's being fixed right now (in general building and running from master is not always the most reliable).

@LDMFD
Copy link
Author

LDMFD commented May 23, 2018

Thanks @folsen - which is the branch to build from to pick up all the fixes referenced in this issue?

@folsen
Copy link
Contributor

folsen commented May 23, 2018

@cachvico I don't know, it depends on what you're trying to do, are you trying to develop something and work on a fix for something? Otherwise I'd just recommend to download the latest release.

@LDMFD
Copy link
Author

LDMFD commented May 23, 2018

I'm simply trying to run the Parity client (e.g. for blockchain querying via web3).

Sorry can you confirm which release version to use? My build is on 1.12.0-unstable, but the latest release on the website is 1.10.4. Given all the problems I had earlier (hence this Issue thread), I don't want to try again with the wrong version..

@folsen
Copy link
Contributor

folsen commented May 23, 2018

Try 1.11.1 from here https://github.com/paritytech/parity/releases

@LDMFD
Copy link
Author

LDMFD commented May 23, 2018

Built 1.11.1 from source and it's running, however not sure it's working. Output attached - first run from clean build after 2 hours. (update: 8 hours and still no apparent progress; ~/.local dir is 2.4M)

Command used was ./target/release/parity --db-compaction ssd --cache-size 1024 --warp-barrier 5650000

Couple of more minor observations:

  • Is there a newcomer guide to all the columns in the standard output? Half of the chit-chat in this bug report has unfortunately been due to me not having a clue what's going on - I see a bunch of zeros but have basically no idea what's happening (in particular whether the fast sync / warp barrier etc. stuff is working).
  • Is there an "uninstall" guide anywhere? My first install was done with the one-line rustup command (for Ubuntu). Since then, I have no idea what's in my system and needs to be deleted to be able to uninstall / clean install a new version. (That's why I built 1.11.1 from source). Is /usr/bin/parity the only file installed, plus ~/.local/share/io.parity.ethereum? (Even if it's a quick rustup command, that would be useful to provide for those who don't know Rust/rustup)
  • Most importantly - is the attached output showing normal state?

Many thanks for all the follow-ups.

output.txt

@Tbaut
Copy link
Contributor

Tbaut commented May 24, 2018

@cachvico thanks for your feedback. A newcomer guide isn't available right now but a work to make our FAQ more beginner friendly has already started.

For such cases as yours, please join our Gitter channel to get to understand how things work under the hood, not only from the parity team but also from the community and more importantly help you resolve your problems.

@LDMFD
Copy link
Author

LDMFD commented May 24, 2018

Sorry what does "cases like mine" mean?

Please note that since opening this Github Issue (at your request @Tbaut on StackOverflow) I still don't have a functioning Parity client despite following instructions and due diligence with clean builds, etc.

Ever so slightly frustrating to be given the brush-off here.

@folsen
Copy link
Contributor

folsen commented May 24, 2018

@cachvico A case like yours would be where someone despite (your) best efforts can't get parity running properly, we've had this back-and-forth in this issue for over a month. Most people don't have this experience, and having one message back and forth per day for another month is not the most effective way to deal with it. If you jump into chat with us, we can probably get it resolved faster.

Edit: Also, the original thing that @Tbaut asked you to raise a bug about in the StackExchange thread was that --warp-barrier was falling back to normal sync, which is now fixed.

@LDMFD
Copy link
Author

LDMFD commented May 24, 2018

To be clear, I was running the fix set up for me by yourselves, for 2 weeks of that month. You then (reasonably) closed the bug but I was stranded on a dev branch so now all I need to do is build from the fixed release.

Except the fixed release given is not working, so back to square 1 (in fact a step backward because now I don't have a functioning node at all, which I did before).

I can now be reached on the Gitter channel and will post a message in due course..

@folsen
Copy link
Contributor

folsen commented May 25, 2018

@cachvico Yeah I feel you, you've been hit by a cacophony of different issues, all of them should have been fixed, but I think you may just need some patience until we cut a 1.11.2 release to get all of them. Building from master you hit another issue and we don't recommend running a node that is built from master. You'd probably have more luck going back to 1.9.x or 1.10.x or some stable release than trying to rely on beta/alpha builds. 1.11.1 should work for you, but for some reason your sync got stuck, there can be a ton of reasons for this, thus it's easier to debug on chat. Hopefully it'll be resolved shortly.

@LDMFD
Copy link
Author

LDMFD commented May 25, 2018

Tbaut verified my above results in chat, confirmed fast sync from clean start is currently unavailable, and advised me to keep retrying.

I'm trying --no-warp, which sounds like it will beat the hare at this rate.

@Tbaut
Copy link
Contributor

Tbaut commented May 28, 2018

@cachvico It's related to #8710 to be patched in 1.11.2

@LDMFD
Copy link
Author

LDMFD commented May 28, 2018

OK. I'm back on geth in the meantime, completed a full sync in about two days.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. P5-sometimesoon 🌲 Issue is worth doing soon.
Projects
None yet
Development

No branches or pull requests

6 participants