High cpu usage and unresponding lightning-cli command

[ronaldvdmeer]

I did some drastic testing with the latest version. Opened around 20~30 payment channels.
While 2 of them had the status: CHANNELD_NORMAL the rest had status: CHANNELD_AWAITING_LOCKIN

At this point I decided to stop the daemon. Remove the directory (not the configuration directory) and updated to the latest version.

After that I've started the daemon again.

Results: High cpu usage for lightning_gossipd (100%)

24239 lightni+  20   0  213616 205716   2400 R  95.0  5.1   6:18.77 /home/lightning/lightning/lightningd/lightning_gossipd

Several lightning_channeld processes spawned as expected. Load minimal on these specific processes. However the logging (log-level=info because debug is untraceable due to high spam) says the following:

lightningd(24215): peer 02e00c2d35ad2eb9d4cde0dd97ded9af0e68faf82877ec00ffadca8ad1238b5092: activate_peer(OPENINGD) with no funding txid?
lightningd(24215): peer 026210049063949182ec8292c9c86814e2202fcd96477ba800ae753ae7f1f02e11: activate_peer(OPENINGD) with no funding txid?
lightningd(24215): peer 020f34961d098bd9236bd2d292a1f5b9113fd9011ad4b7feac6b26d37aa1bc4202: activate_peer(OPENINGD) with no funding txid?
lightningd(24215): peer 02db0930b9212df3604e6cfe9fcbd2028bca203648b4decaa880127b11f9d88c4b: activate_peer(OPENINGD) with no funding txid?
lightningd(24215): peer 026a3189afe3ec38b71e2485da779664cc43f11d9d1a463c62ee9227c3fb4caa51: activate_peer(OPENINGD) with no funding txid?
lightningd(24215): peer 033e90d7d899d83b1e6485b8fe63244c32aaf0f7f4ead86df08c0d8298cf6fa32d: activate_peer(OPENINGD) with no funding txid?
lightningd(24215): peer 036d8454b9c89af87b13b504af429a9ef05922dd084268370de2609b25e34aee21: activate_peer(OPENINGD) with no funding txid?
lightningd(24215): peer 02a56bdaad859cf3530d88e800236aad84ea0271d4ff366729aeb0e5f44c3a6355: activate_peer(OPENINGD) with no funding txid?
lightningd(24215): peer 031bfe54ce29635932bb3dc0df4a841e518f76167dfcb23f85617104d8f069bf3c: state: CHANNELD_AWAITING_LOCKIN -> CHANNELD_NORMAL
lightningd(24215): peer 022246556d72a1f470c214adf95343fd4ad03994ada76924ac389eeffcd883134a: state: CHANNELD_AWAITING_LOCKIN -> CHANNELD_NORMAL

Before upgrade these channels had a txid

{ "peers" : 
	[ 
		{ "id" : "03933884aaf1d6b108397e5efe5c86bcf2d8ca8d2f700eda99db9214fc2712b134", "connected" : true, "netaddr" : 
			[ "34.250.234.192:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_NORMAL", "owner" : "lightning_channeld", "short_channel_id" : "1260665:163:0", "funding_txid" : "4d00fe7ba38eb01a704c71ba4882c7a78db6fc282078ada27f00ce4f11feae39", "msatoshi_to_us" : 16764563740, "msatoshi_total" : 16777215000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "023358548fe587f5c35dac43af5653b1451df65006eb0962d94af57b0a04ff4cf5", "connected" : true, "netaddr" : 
			[ "104.232.113.97:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_NORMAL", "owner" : "lightning_channeld", "short_channel_id" : "1260723:116:0", "funding_txid" : "917a0905c0d6b4aec747c3d4b68cf1a5db94e0ac7bd97121d35157f43a2902cd", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "033f26b0efd9b786c463c27309fc55569506b54797f7e63c2a60fec367073bc027", "connected" : true, "netaddr" : 
			[ "188.166.97.251:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "898f372f0499212b3a6510a077a3b6ce39f0e2ff04205e06d34feffeb574b8c0", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "0299ee6e1b4568628309236247bdcbc371aa54e48c85a5a82ca8d6359f514608de", "connected" : true, "netaddr" : 
			[ "92.53.89.122:9736" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "7e674143c93c17f36b93a95daafefa6a7d1832eec1bb27ed6400d21d6449cca5", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "02ad23477886d4915890c77bdad39d430a204537ab0b7fef4969ebfac9342161aa", "connected" : true, "netaddr" : 
			[ "207.246.93.41:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "a6945d48479ccbe1837f97fe2fa6c530fa906f782600d6b545da501d6e958707", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "02925a2841b3cff73ca413bce68fa3529a940fa603731bdba9dd8306f5e9a74460", "connected" : true, "netaddr" : 
			[ "62.210.252.112:9737" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "95b4f80e2ff025c756a1907eef6514e2c986a91eebe231179d7d88c663d03ac9", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "031095ad9bf1b73b219f5efd1059257b1816eb97e30ff816b55ba5e51dd4e400b4", "connected" : true, "netaddr" : 
			[ "62.68.195.173:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "3d798ec80e2795f2e2f56ce5d450de29532ea329790811602f93e56c2275f072", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "03c6380abce3488b64b4d9ba10ce25ca1236133c6d57b86538df583ba10b24cb00", "connected" : true, "netaddr" : 
			[ "168.235.103.103:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "3d0f9e89d53fea51567d22f92f084639edf72d1574c971235b6321557157e65b", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "022246556d72a1f470c214adf95343fd4ad03994ada76924ac389eeffcd883134a", "connected" : true, "netaddr" : 
			[ "95.21.128.50:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "4973d0ce9a1b8668476a9c33433da26b3eab709bcbb2500b99047071d09a7f3a", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "03324ef21e77714cdc1ad7fadda808a7aca2b7f8d31cd737923d42164dac00a02d", "connected" : true, "netaddr" : 
			[ "139.162.14.142:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "2cdef2e21add52212a7489712495649ef29fa224234bdc7487603a69e90749fd", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "021a45969aea11e8a306defcecf52029628c98f7e900941e2fd4f628ccf2aece28", "connected" : true, "netaddr" : 
			[ "81.27.36.90:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "af26ca0cf175a70f8e86ee9e41e461482c480b2570f07a0143821eb1505399ce", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "028d8e28a55d5e09f9ce1669d33f79c41943f8aa1ccaa3fa8b6c90bab63ce26aac", "connected" : true, "netaddr" : 
			[ "40.114.146.211:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "cb6317b05506c7b2214d039689cb040ec91533d65c6090c24c3bdb9977295b2d", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "022c315d7588cf1714159915baae386dc87462b669a7036df33850e9c0a3265893", "connected" : true, "netaddr" : 
			[ "86.61.67.183:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "a416b48bdaaa39b6eac2662f9295f75b3127b224ee4f2654eaf8e49bd7fd1437", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "031bfe54ce29635932bb3dc0df4a841e518f76167dfcb23f85617104d8f069bf3c", "connected" : true, "netaddr" : 
			[ "18.218.207.229:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "fe84590ea0c1609c9ac5a1fec7a81605bbb36e8c563046b09e49494e10abc033", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "039641f2fde4ec6d013605c1530babad7dcd50eed3a07d8ada2dd78d6ee8e1cf9f", "connected" : true, "netaddr" : 
			[ "88.99.82.23:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "3e777d21c931e66d01af9540f5ff9f0b95a4e0603ac5ab3274aa8c1e2d557a78", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "03ad318b3ddaf8330c44d81f353c34942178a220da07fa687ddc5d82c637c4d57d", "connected" : true, "netaddr" : 
			[ "172.105.206.164:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "1f7fe5a36d1115390cf2c81456f3733fa46d38d4c237b00c0adda43e4109a76e", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "02df10abb5b6d9361888ee264ee023b023f2deb7003cb1341f20a54ea072121396", "connected" : true, "netaddr" : 
			[ "61.6.90.165:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "73d50fa1695f570aad26ebe117bed97f968c0a69499c0817f4d2ec4878a14ba8", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "02139a5bc1f1c929c04530a99160f2072e2a731f14c666abf87479510912279c11", "connected" : true, "netaddr" : 
			[ "138.197.121.209:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "a0c6bd6b76a3abe2197962695c060eda49b38981242909f6cfd78ffd11cfdd97", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "02b0627883f87f5d954c2ee217e51bf380745fe764b73d36150f0b80c32199ff27", "connected" : true, "netaddr" : 
			[ "47.144.212.37:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "945c93a5759f3add1a69aee1f7cd14676efb6df1356368aeb9420d57c667d539", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] }, 
		{ "id" : "0260a9711694672b4fd55add53def7cdbd18d767f455b052e0ff63e651548c6c56", "connected" : true, "netaddr" : 
			[ "5.51.143.110:9735" ], "channels" : 
			[ 
				{ "state" : "CHANNELD_AWAITING_LOCKIN", "owner" : "lightning_channeld", "funding_txid" : "b864d520382a3f71a4464caf8f2cfcf55770596b4182a5e20f97c357ea81528c", "msatoshi_to_us" : 100000000, "msatoshi_total" : 100000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 6, "max_accepted_htlcs" : 483 } ] } ] }

Also the following command seems to stall (just saw it returned information after around 15 minutes).

lightning@dawson:~/lightning$ /home/lightning/lightning/cli/lightning-cli --lightning-dir /home/lightning/.lightning-testnet listpeers

Is it normal that it takes so long to restore the channels?

[rustyrussell]

Those CHANNEL_OPENING peers never got far enough in negotiation to be permanently remembered: committing then to the db is basically a bug (though minor and not trivial to fix).

Will test with 20 awaiting_lockin channels and see what the CPU usage is all about...

[ronaldvdmeer]

When most channels are restored to the CHANNELD_NORMAL state load is still relatively high (35% cpu on the lightningd core proces). 75% overall load on the system.

Also, and this might also cause high system load, when debug-log is enabled lots of lines are written to the logfile(1~4mb/s). Hard to debug other stuff because of all these lines. Maybe its a good idea to move these spamlines to another loglevel (for example "EVERYTHING"). So that testers are able to use the debug loglevel for some debugging.

lightning_channeld(4671): TRACE: Read decrypt 0102eec78453148f831bbddd43dbd063d03c079fa04aea1e599d41b64a925f4b03c32f9e6afcd9ec5c3669fb27f2f5ac555d4690a9b62b51d98f4bac8d60fc0f734543497fd7f826957108f4a30fd9cec3aeba79972084e90ead01ea330900000000132d1b00002500015a4d86ff0001009000000000000003e8000003e800000001
lightning_channeld(4671): TRACE: peer_in WIRE_CHANNEL_UPDATE
lightning_channeld(4798): TRACE: peer_out WIRE_CHANNEL_ANNOUNCEMENT
lightning_channeld(4798): TRACE: Read decrypt 0101d7b1ca74fe331e2d9ff69e3fb8f8fc5ad31f91bfa92464220417b5f0426531307f585cbb9a1f7a1f3f1fe4ad7befdb2974bf8ac3a9bbf5a802a8f0d6250454e200005a4aecfb03459c42e997536a3347cb31c19c50df2d8e0d22e4b064de4c435a75bb275b9d473399ff303334353963343265393937353336613333343700000000000000000000000000070134123d322607
lightning_channeld(4694): TRACE: peer_out WIRE_CHANNEL_UPDATE
lightning_channeld(4798): TRACE: peer_in WIRE_NODE_ANNOUNCEMENT
lightning_channeld(4694): TRACE: Read decrypt 0101ff8fa51f110ef3b5f2a0ad28fb534b0e288bc72970ea87f064ccf7f2ebcb25a67463d33c015b3a1ee6ebf4f34491a2ee6999f83d2232a24c07a051e3ad4b029300005a4d97ba03e65880607578a108440df09580878f12e0800e88c09e313d91da02847da0cebe3399ff30336536353838303630373537386131303834340000000000000000000000000007018259a2892607
lightning_channeld(4694): TRACE: peer_in WIRE_NODE_ANNOUNCEMENT
lightning_channeld(4673): TRACE: peer_out WIRE_CHANNEL_UPDATE
lightning_channeld(4673): TRACE: Read decrypt 010168cc0853c41adfc8213d5a81321f37c40d8c424fa2c9880800c899e345cab9fc39369203528e058134c039f89214a2809667ca2a9d76b08decad74c6e9d0f3fc00005a563e4e03b83283765f36daa497207f214416177020d962e4d38a805f17bd5583bd8a18d23399ff30336238333238333736356633366461613439370000000000000000000000000007012d3f57832607
lightning_channeld(4673): TRACE: peer_in WIRE_NODE_ANNOUNCEMENT
lightning_channeld(4881): TRACE: peer_out WIRE_CHANNEL_ANNOUNCEMENT
lightning_channeld(4881): TRACE: Read decrypt 01003003ebf259097d906dea743a4670e484fabb8490e3d3f6b77496cf6cfaeb80024973d715ef9f30d773a9fbf81eb4eecfe79b93d013c62dae39376be7153800b58cc5438b959b9430228d688e3b3bf8bdb18b1dcc2dc3654331a6e6235777e28e62bdfe607ac3c7255497728616e358cb124c30bc20bed0077f125fee509d4f0b99e01c6117b108c59eda5da04bcfd5f841cc7c6d02e1243d620fcbb2e27bf03c0dc4bcac9358726989ac862e00474b8703bc51febaf5597cf4e0b9e9ddf8416c34e9b42e836d05ae05731bdc46943ae2bf57bc568b1fe5c60e74d0131626b70a4746febe18714192f7aa351f7df04a53d137034e95917e85c3a4a4dba634c9e6000043497fd7f826957108f4a30fd9cec3aeba79972084e90ead01ea33090000000013359a000065000003a2c860079a31f5a0183041c2d8405f4ae4bba651f2ddc5c17ec09b85f054eef503f113414ebdc6c1fb0f33c99cd5a1d09dd79e7fdf2468cf1fe1af6674361695d20322b98162c10015dae5530b7e27051032c6bc4c6425ca3159ebd52a1d59a24db803a93817b125d0b5049454f8d61b665763da480b1ec1ec9db5e73ba2d4babf5120
lightning_channeld(4881): TRACE: peer_in WIRE_CHANNEL_ANNOUNCEMENT
lightning_channeld(4809): TRACE: peer_out WIRE_CHANNEL_ANNOUNCEMENT
lightning_channeld(4984): TRACE: peer_out WIRE_CHANNEL_ANNOUNCEMENT
lightning_channeld(4809): TRACE: Read decrypt 01021fa732b7799f1682c8d5e3ff3844bf2a90b5bbbdaf2d5690afcb4bd54a20ac6831e7b92633a9ff11d565ee2f16c1a32173cd103a613dabc7fb3de5624734281943497fd7f826957108f4a30fd9cec3aeba79972084e90ead01ea33090000000013353c00006400015a6bb5b3000100900000000000000000000003e800000001
lightning_channeld(4984): TRACE: Read decrypt 0101c94f535a97e665dca209013293d3eefdd6133c836617a2576b908993cf0569f16d20b00997a58d7db52d95a5dddcaa15b4eee115e55b4a4cf563b449ac7381cd00005a5ea0e103d6b1e410540edfabdf9c1c3f6e3cb77e009090c23e557117b1305a42239c50de49daaa4661626f6f736500000000000000000000000000000000000000000000000000000701a9efb4a32607
lightning_channeld(4809): TRACE: peer_in WIRE_CHANNEL_UPDATE
lightning_channeld(4984): TRACE: peer_in WIRE_NODE_ANNOUNCEMENT
lightningd(4628): Adding block 1260726: 00000000001461b5635dce8138e5cf3e7b69d171ff166ad4e6912d35c7e6d357

[ronaldvdmeer]

I think I got clightning to the point it is no longer usable. Already waiting 30 minutes for a return on the listpeers cli-command.

I got 52 channeld processes running and a gossipd proces taking 75-100% cpu. Channels have various statuses last time I checked but because I can't perform a listpeers at this moment I don't know the current status of the channels.

Anyone else see the same behavior when trying a node with more then a couple of payment channels?

[cdecker]

It's most likely due to the initial gossip sync, and that some peers simply do not prune their channel_updates which means we'll revalidate the same updates over and over again. Should become much better once we track the UTXO ourselves.

[ronaldvdmeer]

That explains. Is UTXO tracking for clightning something that is implemented in the very near future or will that take some time to implement. I'm asking this so I can consider alternatives for my goal to run a high connectivity LN-node.

[cdecker]

I'll add a gettxout cache for the on-chain validation ASAP, since that also kills the store. The UTXO tracking will most likely take a bit longer, but the short-term solution should fix the worst problems.

[ronaldvdmeer]

Thank you. I'll wait for your fix before any further testing

[ZmnSCPxj]

#861 was merged which partially optimizes gossipd, which should help here. Another is #892 which promises even more improvements but is still work-in-progress.

I think you can reevaluate c-lightning at this point even with only #861 though.

[cdecker]

Closing due to inactivity and the issue is fixed imho.