eth: make fast sync great again

[fjl]

This is a tracking issue for fixing various things that make fast sync slow right now:

• merge concurrent header downloads eth/downloader: concurrent header downloads #2315
• fix frequent rollbacks (possibly caused by tx issue eth: skip transaction handling during fast sync #2574, maybe wontfix)
• auto-disable fast sync less aggressively
• disable tx processing/relay, pending state during fast sync eth: skip transaction handling during fast sync #2574
• don't give up so early if state download is interrupted eth/downloader: interrupted state download causes sync to restart from block 0 #2469 eth/downloader: make fast sync resilient to critical section fails #2647
• pull state trie concurrently with blockchain eth/downloader, trie: pull head state concurrently with chain #2627
• dynamically adapt to various network conditions eth/downloader: adaptive quality of service tuning #2630

[taoeffect]

I'm seeing Synchronisation failed: no peers to keep download active, but I believe fast-sync won't help as I've already started (a long time ago) to download the chain? I just need to catch up but it keeps stalling, and that doesn't make sense unless (1) there's a bug, or (2) there's censorship.

[ghost]

@taoeffect Are you in an Amazon data center? I'm interested in the (2) that you mention

[taoeffect]

No, residential.

[ghost]

OK. I don't think Amazon are traffic shaping. Just slightly paranoid. Good to be able to discount that possibility.

I wonder are there a lot of nodes out there that aren't up-to-date and it's difficult to find the correct blocks required? I don't know very much about this to be honest. Maybe I'm completely wrong about this.

[taoeffect]

Traffic shaping can still exist on a larger scale (i.e. at mega-ISP level).

[ghost]

Interesting. Well good luck with your sync. I'm doing OK now. I have my Ubuntu 16.04 EC2 instance sync'd. I'm sync'ing geth on El Capitan right now - it seems to be working fine. But I'm worried that if I turn my Mac off and try to re-sync, it won't work. This happened to me last week. It's very frustrating.

[fluidvoice]

Same problem here. Been trying to sync for days at home. Is there any workaround? First time it crapped out around 800K blocks this time ~1.1M and seems to die around block #192 every time. I'm running Mint 17.2 (Ubuntu 14.04).

I0520 20:20:04.856909 core/blockchain.go:751] imported 39 receipt(s) (0 ignored) in 26.745337ms. #1105498 [36f4ec37… / ebe93933…]
I0520 20:20:06.865184 eth/downloader/downloader.go:1130] Rolled back 767 headers (LH: 1105650->1104883, FB: 1105498->1104883, LB: 0->0)
I0520 20:20:08.207837 core/blockchain.go:751] imported 4 receipt(s) (0 ignored) in 4.122744ms. #1105502 [54c9a3c4… / dbf47623…]
I0520 20:20:08.646643 core/blockchain.go:751] imported 148 receipt(s) (0 ignored) in 123.650486ms. #1105650 [7eea44d2… / 9709304f…]
I0520 20:20:34.383578 eth/downloader/downloader.go:998] Peer c955d4c87e933322 [blocks 0.00/s, receipts 0.00/s, states 0.00/s, lacking 0]: empty head header set
I0520 20:21:13.322653 core/headerchain.go:294] imported 192 header(s) (0 ignored) in 3.144793158s. #192 [fd337a68… / fafce6c5…]
I0520 20:21:15.285444 core/blockchain.go:959] imported 192 block(s) (0 queued 0 ignored) including 0 txs in 1.904347632s. #192 [fd337a68 / fafce6c5]
I0520 20:21:18.074450 core/headerchain.go:294] imported 0 header(s) (192 ignored) in 5.877154ms. #384 [967642fd… / d3d5d5c1…]
I0520 20:21:18.074585 eth/downloader/queue.go:336] Header #193 [967642fd] broke chain ancestry
I0520 20:21:18.151726 eth/downloader/downloader.go:1130] Rolled back 192 headers (LH: 192->0, FB: 192->0, LB: 192->0)
I0520 20:22:38.465731 core/blockchain.go:959] imported 2 block(s) (0 queued 0 ignored) including 0 txs in 472.406552ms. #2 [88e96d45 / b495a1d7]
I0520 20:22:45.178367 core/blockchain.go:959] imported 33 block(s) (0 queued 0 ignored) including 0 txs in 1.847313491s. #35 [3d612266 / 395005a5]
I0520 20:22:46.694117 core/blockchain.go:959] imported 35 block(s) (0 queued 0 ignored) including 0 txs in 1.439287081s. #70 [5f81bfa6 / 54cde713]
I0520 20:22:47.205693 core/blockchain.go:959] imported 12 block(s) (0 queued 0 ignored) including 0 txs in 510.61695ms. #82 [c7553e66 / 8614615e]

[ghost]

I had to delete contents of "chaindata" folder again today. I was on a 3G network and chopping/changing on to different WiFi networks all day. I was getting errors about "fast sync disabled":

I0522 09:53:12.246194 cmd/utils/flags.go:687] You're one of the lucky few that will try out the JIT VM (random). If you get a consensus failure please be so kind to report this incident with the block hash that failed. You can switch to the regular VM by setting --jitvm=false
I0522 09:53:12.248099 ethdb/database.go:82] Alloted 128MB cache and 1024 file handles to /Users/hynese/Library/Ethereum/chaindata
I0522 09:53:12.426900 ethdb/database.go:169] closed db:/Users/hynese/Library/Ethereum/chaindata
I0522 09:53:12.433158 ethdb/database.go:82] Alloted 128MB cache and 1024 file handles to /Users/hynese/Library/Ethereum/chaindata
I0522 09:53:12.497856 ethdb/database.go:82] Alloted 16MB cache and 16 file handles to /Users/hynese/Library/Ethereum/dapp
I0522 09:53:12.504219 eth/backend.go:170] Protocol Versions: [63 62 61], Network Id: 1
I0522 09:53:12.504386 eth/backend.go:199] Blockchain DB Version: 3
I0522 09:53:12.509427 core/blockchain.go:206] Last header: #1558562 [de581683…] TD=21084093975160792591
I0522 09:53:12.509461 core/blockchain.go:207] Last block: #1558562 [de581683…] TD=21084093975160792591
I0522 09:53:12.509479 core/blockchain.go:208] Fast block: #1558562 [de581683…] TD=21084093975160792591
I0522 09:53:12.511947 eth/handler.go:92] blockchain not empty, fast sync disabled

Then when I fired her up this morning, I had similar issues:

I0522 12:57:10.572794 ethdb/database.go:82] Alloted 128MB cache and 1024 file handles to /Users/hynese/Library/Ethereum/chaindata
I0522 12:57:10.927887 ethdb/database.go:169] closed db:/Users/hynese/Library/Ethereum/chaindata
I0522 12:57:10.932615 ethdb/database.go:82] Alloted 128MB cache and 1024 file handles to /Users/hynese/Library/Ethereum/chaindata
I0522 12:57:11.015033 ethdb/database.go:82] Alloted 16MB cache and 16 file handles to /Users/hynese/Library/Ethereum/dapp
I0522 12:57:11.020979 eth/backend.go:170] Protocol Versions: [63 62 61], Network Id: 1
I0522 12:57:11.023402 eth/backend.go:199] Blockchain DB Version: 3
I0522 12:57:11.025530 core/blockchain.go:206] Last header: #1563130 [52feb24f…] TD=21265099994639104878
I0522 12:57:11.025559 core/blockchain.go:207] Last block: #1563130 [52feb24f…] TD=21265099994639104878
I0522 12:57:11.025577 core/blockchain.go:208] Fast block: #1563130 [52feb24f…] TD=21265099994639104878
I0522 12:57:11.027957 eth/handler.go:92] blockchain not empty, fast sync disabled
I0522 12:57:11.030610 p2p/server.go:311] Starting Server
I0522 12:57:12.860249 p2p/discover/udp.go:217] Listening, enode://47b25862528236ac233c5a0f30626d9c85324a1897a9e9a636d2b893fe2efb0713e838ea139d6257a072096c95f41b510fd3e2d08e681a8005ca76f34c56bef5@[::]:30303
I0522 12:57:12.863348 p2p/server.go:554] Listening on [::]:30303
I0522 12:57:12.863844 node/node.go:298] IPC endpoint opened: /Users/hynese/Library/Ethereum/geth.ipc
I0522 12:57:22.865123 eth/downloader/downloader.go:299] Block synchronisation started

Now I'm starting from scratch again.

This is with geth (1.4.4-stable-fc0638f9) on OSX installed with homebrew.

[avastmick]

Is the sync protocol equivalent across all Ethereum clients? i.e. do they all implement the same specification with regard to block synchronisation?

I'm going to go and have a look at the syncing of other clients eth and parity to confirm whether or not this is seen outside of geth and is an implementation bug or a protocol issue. Anyone tried this? I have tried parity in its early alpha stage but I can't remember whether it worked or not, never used eth

Note: I'm dropping in from a related issue #2587 that may not be caused by --fast that this collation issue implies - I can reproduce the fragility of sync regardless of start point.

[pannous]

Why doesn't ethereum offer the whole .ethereum/chaindata folder as a torrent until this issue is fixed?
( --fast 'only' 2GB? )

[karalabe]

Fixed everything that's going to be fixed for now. Lets see how the network behaves after the 1.4.6 release and reevaluate any issues afterwards.