create-gossipstore.c can read scid -> satoshis csv file. The csv is i… #5

A backtrace which makes it much easier to figure out what's happening. Signed-off-by: Rusty Russell <[email protected]>

…stream of gossip. The gossip is expected to be in format: 16-bit-big-endian-length [gossip message] Signed-off-by: Rusty Russell <[email protected]>

For giant nodes, it seems we spend a lot of time memmoving this array. Normally we'd go for a linked list, but that's actually hard: each channel has two nodes, so needs two embedded list pointers, and when iterating there's no good way to figure out which embedded pointer we'd be using. So we (ab)use htable; we don't really need an index, but it's good for cache-friendly iteration (our main operation). We can actually change to a hybrid later to avoid the extra allocation for small nodes. Signed-off-by: Rusty Russell <[email protected]>

This is useful for canned data, such as the million channels project. Signed-off-by: Rusty Russell <[email protected]>

We push a huge msg for listchannels with the million-channels project. We need to fix that, but this works around it so we can benchmark. Signed-off-by: Rusty Russell <[email protected]>

This lets us benchmark without a valid blockchain. Signed-off-by: Rusty Russell <[email protected]>

@cdecker

Suggested-by: @cdecker Signed-off-by: Rusty Russell <[email protected]>

…write. Signed-off-by: Rusty Russell <[email protected]>

… restarts Signed-off-by: Rusty Russell <[email protected]>

This causes natural batching, rather than on every little addition of JSON formatting. Before, to listchannels 100,000 channels took 82.48 seconds, after 6.82 seconds. Signed-off-by: Rusty Russell <[email protected]>

Adding a giant IO message simply causes it to be pruned immediately, so truncate it if it's more than 1/64 the max size. Signed-off-by: Rusty Russell <[email protected]>

…ced. Either private or simply not enough confirms. They would have been added on reconnect, but that's not ideal. Signed-off-by: Rusty Russell <[email protected]>

Otherwise we'd forget them on restart, again. Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

…hannels project Outputs CSV. We add some stats for load times in developer mode, so we can easily read them out. peer_read_all_sec doesn't work, since we seem to reject about half the updates for having bad signatures. It's also very slow... routing fails, for unknown reasons, so that failure is ignored in routing_sec. Results from 5 runs, min-max(mean +/- stddev): store_load_msec,vsz_kb,store_rewrite_sec,listnodes_sec,listchannels_sec,routing_sec,peer_write_all_sec 39275-44779(40466.8+/-2.2e+03),2899248,41.010000-44.970000(41.972+/-1.5),2.280000-2.350000(2.304+/-0.025),49.770000-63.390000(59.178+/-5),33.310000-34.260000(33.62+/-0.35),42.100000-44.080000(43.082+/-0.67) Signed-off-by: Rusty Russell <[email protected]>

@niftynei

…llion channels project Suggested-by: @niftynei

…llion channels project MCP filename change.

Signed-off-by: Rusty Russell <[email protected]>

…in the format scid ,satoshis where there is a black space after scid. Made a header file that contains a struct. Modified makefile. Added cmdline arg --scidfile /path/to/csv and made the constant capacity command optional. create-gossipstore prints stats at the end.

Signed-off-by: Rusty Russell <[email protected]>

…ets. eg. for running under valgrind. Signed-off-by: Rusty Russell <[email protected]>

…ssip. Revised style, mem allocation, and error checks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

create-gossipstore.c can read scid -> satoshis csv file. The csv is i… #5

create-gossipstore.c can read scid -> satoshis csv file. The csv is i… #5

Commits on Apr 5, 2019