Catching up with the blockchain is slow

[PRabahy]

I have been syncing on and off for over 48 hours and have not caught up with the entire blockchain yet. Both CPU usage (<5%) and Disk usage (1MB/s) appear to be low.

I am running an instance of BitcoinQT on the same box and used --addpeer to try to kick it in the butt. Right now, I appear to be averaging less than a block every 5 seconds. Issue #12 mentions that various speed optimizations are planned.

20:13:59 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 11.59s (273
 transactions, height 235912, 2013-05-12 21:08:55 -0400 EDT)
20:14:10 2013-10-23 [INF] BTCD: BMGR: Processed 1 block in the last 10.85s (870
transactions, height 235913, 2013-05-12 21:40:11 -0400 EDT)
20:14:28 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 18.21s (846
 transactions, height 235915, 2013-05-12 21:53:58 -0400 EDT)
20:14:47 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 18.41s (101
6 transactions, height 235917, 2013-05-12 22:12:24 -0400 EDT)
20:14:58 2013-10-23 [INF] BTCD: BMGR: Processed 3 blocks in the last 11.43s (454
 transactions, height 235920, 2013-05-12 22:30:06 -0400 EDT)
20:15:16 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 18.22s (104
4 transactions, height 235922, 2013-05-12 22:47:41 -0400 EDT)
20:15:27 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 10.92s (601
 transactions, height 235924, 2013-05-12 23:01:45 -0400 EDT)
20:15:45 2013-10-23 [INF] BTCD: BMGR: Processed 5 blocks in the last 17.46s (103
8 transactions, height 235929, 2013-05-12 23:16:12 -0400 EDT)
20:15:56 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 11s (793 tr
ansactions, height 235931, 2013-05-12 23:38:01 -0400 EDT)
20:16:14 2013-10-23 [INF] BTCD: BMGR: Processed 1 block in the last 18.08s (587
transactions, height 235932, 2013-05-12 23:49:04 -0400 EDT)
20:16:27 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 12.82s (486
 transactions, height 235934, 2013-05-12 23:56:22 -0400 EDT)

[davecgh]

I've created several new issues which are targeted at optimizations areas revealed by profiling which can be used to track progress on this front. They are all tagged optimization.

There have already been a few optimization related commits in the past couple of days since this issue was created that are working toward speeding things up.

[davecgh]

As an update to this, current master has done #21, #22, #25, and various other optimizations which don't have issues opened against them.

Therefore, the next release should have some significant improvements in this area. As can be seen from the issue tracker, there are more planned optimizations.

[davecgh]

For another update to this, current master amongst btcd and its deps have completed #27 and #28.

[davecgh]

The signature verifications are now 10x faster as of commit btcsuite/btcec@ac7e4de. This results in a significant impact on the download speed after the final checkpoint (roughly 10x faster).

[davecgh]

Closing this as all of the optimizations noted above in conjunction with headers-first mode have been implemented and are included in the btcd 0.5.0-alpha release. Please note that the database is still slower than it should be and this is being tracked by #24.

Note that even with the slower database, several of us have done head-to-head tests of downloading the full block chain as compared to bitcoind and btcd is slightly faster in most scenarios. The only time it has been measurably slower is on machines with slow disk I/O. As noted above this is the result of the slower database.