net: Hardcoded seeds update for 0.20 #18506
Conversation
|
|
|
Are non-standard ports ok? |
laanwj
force-pushed
the
2020_04_hardcoded_seeds
branch
from
1e06cc7 to
5dc165f
Compare
Yes, that's ok. Might help when ISPs block the standard port. |
|
It is suspicious that the change in thd set of onion hosts is almost entirely red. |
|
Last time there was also a problem with onions and the DNS seed (#16999). |
|
@sipa Now that you say, I was surprised that adding Maybe if other people with DNS seeds post their list, I can combine them again and see if that gives more active onion addresses. |
|
@laanwj Ugh. It seems my crawler wasn't running with Tor enabled. Fixed. This probably means we'll want to use someone else's data (or even better, combine the results). |
|
@TheBlueMatt @luke-jr @cdecker @Sjors @petertodd @Emzy @jonasschnelli if possible please upload your DNS seed dump and post it here,especially if your crawler is running with Tor |
I believe non-standard ports should be encouraged as a fail over. Port blocking on the backbone should be anticipated. |
|
Our outgoing connection selection is very strongly avoiding non-standard ports (something that's been in the codebase for ages BTW; the original intent was to avoid turning the Bitcoin network into a DDoS service for non-Bitcoin ports). |
|
http://luke.dashjr.org/programs/bitcoin/files/charts/seeds.txt (162 MB, no Tor) |
I have deleted the dnsseed.dump an hour ago. So there are only 23 onion addresses in the dump. I will upload the file again later if I got more. |
|
seeds-emzy.txt.gz |
|
seeds-emzy.txt.gz |
|
dnsseed.dump.tar.gz with 1394 onions |
laanwj
force-pushed
the
2020_04_hardcoded_seeds
branch
from
5dc165f to
8859c52
Compare
Update hardcoded seeds from seeds_emzy.txt seeds_lukejr.txt seeds_sipa.txt seeds_sjors.txt, according to release process. Output from makeseeds.py: ``` IPv4 IPv6 Onion Pass 1364173 244127 2454 Initial 1364173 244127 2454 Skip entries with invalid address 1129552 213117 2345 After removing duplicates 1129548 213117 2345 Skip entries from suspicious hosts 338216 191944 2249 Enforce minimal number of blocks 336851 188993 2189 Require service bit 1 6998 1520 150 Require minimum uptime 5682 1290 89 Require a known and recent user agent 5622 1279 89 Filter out hosts with multiple bitcoin ports 512 146 89 Look up ASNs and limit results per ASN and per net ```
laanwj
force-pushed
the
2020_04_hardcoded_seeds
branch
from
8859c52 to
0eeb046
Compare
|
Ok, re-did the process with the concatenated files and updated the statistics. Hopefully it's better now (there's a few more onions). |
|
I'm only seeing two green onions in a see of red onions. Maybe loosen the criteria for onion nodes a bit? They don't have amazing up time as seen from my seed, but Tor itself (or my connection to it) might be flaky too. |
|
The uptime requirements for onions are already much more lenient than for IPv4 peers (10% instead of 50%). I could bring them down even more, but, do we need that many hardcoded onion seeds? Seems 89 is pretty ok to me. (Also, can we do changes to the seeds scripts between release please? the issue for that is #17020. Everyone seems to forget about it every time until the hurry before a release branch-off) |
|
5% uptime instead of 10% would 3x the number of nodes from my list, but I agree 89 sounds good enough, and 10% is quite bad already.
Guilty as charged :-) |
6866259 net: Hardcoded seeds update for 0.21 (Wladimir J. van der Laan) 36e875b contrib: Add new versions to makeseeds.py and update gitignore (RandyMcMillan) Pull request description: Stats: ``` IPv4 IPv6 Onion Pass 426728 59523 7900 Initial 426728 59523 7900 Skip entries with invalid address 426728 59523 7900 After removing duplicates 426727 59523 7900 Skip entries from suspicious hosts 123226 51785 7787 Enforce minimal number of blocks 121710 51322 7586 Require service bit 1 4706 1427 3749 Require minimum uptime 4124 1098 3681 Require a known and recent user agent 4033 1075 3681 Filter out hosts with multiple bitcoin ports 512 140 512 Look up ASNs and limit results per ASN and per net ``` I've credited RandyMcMillan for the first commit because of #20190. There are at least enough onions this time! Number of IPv6 nodes that pass all the requirements seems similar to last time in #18506. For the next major release we'll want TORv3 hardcoded peers as well. This makes no sense now as there are hardly any. But it'd make sense to think about how to collect them because they cannot come from the DNS seeds. ### Reviewing ``` 2020-10-28 12:04:45 jnewbery wumpus: Do you have any suggestions for how to review #20237 ? 2020-10-28 12:28:37 wumpus jnewbery: previous PRs like it might be a guide there (#18506, #16999), e.g. people could try to repeat the last step in https://github.com/bitcoin/bitcoin/tree/master/contrib/seeds#seeds and see if it ends up with the same .h file, you could also repeat the entire process but as the list of peers from the seeder will be different every time that will give a (slightly, hopefully) 2020-10-28 12:28:37 wumpus different output 2020-10-28 12:49:40 wumpus testing what part of the peers are connectable is also useful 2020-10-28 12:51:05 wumpus or to go deeper, whether most part of the nodes are 'good nodes' and not say spy nodes, but i don't know what means of testing ``` ACKs for top commit: jonatack: ACK 6866259 Tree-SHA512: 6b913ec92932de03304301a0cbf7b4a912ed09d890b019deeb449b8fa787c4994222368c6bf08b3c6e2bfa474442612e1c9de9327ec46ba59c37a5f38af50c75
Update hardcoded seeds from http://bitcoin.sipa.be/seeds.txt.gz,
according to release process.
Output from makeseeds.py: