Nydus lazy pull + Dragonfly performance issue

[fannaihao]

Hello~
We deploy an environment to test Dragonfly's transmission performance of OCI images.
We use Nydus lazy pull + Dragonfly;

version info:
Nydus v1.1.2
Dragonfly2 v2.0.2-rc.9

Dragonfly deployment config:
1. We have 2 regions(region-A and region-B) here;
2. network condition of two regions : ping latency is 3ms about,throughout is 1 Gbit;
3. 454 hosts of region-A; we deploy dfget daemon, containerd, nydus on each one;
4. 1 manager instance on 1 host of region-B;
5. 3 scheduler instances on 3 host of region-B;
6. 1 cdn instance in 1 host of region-B, and we start a python-fileserver listening on 8001 to distribute files downloaded by cdn;
7. 1 images registry instance on 1 host of region-B;

Application Info:
1. Wordpress , you can get this image using docker pull wordpress:latest
2. We convert raw image to Nydus format, follow this: Nydusify

How we measure performance:
1. Rpull the Nydus image, refer this Nydus ctr-remote image rpull --plain-http $ip:5000/wordpress-nydus:latest
2. Start Nydus image; ctr-remote run --rm -t --snapshotter=nydus $ip:5000/wordpress-nydus:latest test /bin/bash
3. Check whether WordPress is ready and record elapsed time;
4. We juge whether WordPress is ready by pobe the 80 port (telnet localhost 80) , when the application is ready, we record the total elapsed time to measure the image distribution performance(Nydus lazy pull + Dragonfly p2p).
5. We also test the native performace of this case: everything is same except that we use OCI native pull + containerd ctr run.

Reuslt:
1. Nydus + Dragonfly wordpress ready-time distribution: 80s ~ 400s, average 200s;
2. Native pull and start wordpress ready-time distribution: 20s ~ 35s, average 25s;
We'd like to provide detailed data if needed.

We found a performance regression when using Nydus lazy pull + Dragonfly.
We didn't not open Nydus prefetch option.

When look into logs, we found some long-tailed p2p task request here:
{"level":"info","ts":"2022-02-23 15:00:53.946","caller":"peer/peertask_conductor.go:1167","msg":"peer task done, cost: 4348ms","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"}
Above one cost 4348 ms.

Detailed dfget log of it (find these log by searching taskId or peeID):
{"level":"info","ts":"2022-02-23 15:00:49.597","caller":"client/client.go:85","msg":"generate hash key taskId: 01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5 and start to register peer task for peer_id(10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924) url(http://10.40.162.87:5000/v2/wordpress-nydus/blobs/sha256:135348e46ccb98df38152269e88f6b5a13b2c1f25b155f3d8b7998d9c043e3f9)","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} {"level":"info","ts":"2022-02-23 15:00:53.945","caller":"client/client.go:108","msg":"register peer task result success url: http://10.40.162.87:5000/v2/wordpress-nydus/blobs/sha256:135348e46ccb98df38152269e88f6b5a13b2c1f25b155f3d8b7998d9c043e3f9, scheduler: dns:///10.88.65.88:8002","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} {"level":"info","ts":"2022-02-23 15:00:53.945","caller":"peer/peertask_conductor.go:252","msg":"register task success, SizeScope: SMALL","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"info","ts":"2022-02-23 15:00:53.945","caller":"client/client.go:156","msg":"start to report piece result for taskID: 01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerId":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","errMsg":null} {"level":"info","ts":"2022-02-23 15:00:53.945","caller":"peer/peertask_conductor.go:283","msg":"step 2: start report piece result","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"info","ts":"2022-02-23 15:00:53.945","caller":"peer/peertask_conductor.go:639","msg":"single piece, dest peer id: 10.227.172.34-7-8f681751-907d-4f24-ad76-a2505f4cdb40, piece num: 0, size: 40529","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"debug","ts":"2022-02-23 15:00:53.945","caller":"storage/storage_manager.go:305","msg":"init local task storage, peer id: 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924, task id: 01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5"} {"level":"debug","ts":"2022-02-23 15:00:53.945","caller":"peer/piece_downloader.go:170","msg":"built request url: http://10.227.172.34:65002/download/01b/01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5?peerId=10.227.172.34-7-8f681751-907d-4f24-ad76-a2505f4cdb40"} {"level":"debug","ts":"2022-02-23 15:00:53.946","caller":"peer/piece_downloader.go:151","msg":"calculate digest for piece 0, digest: c588c321757bf3563a15866801f02df5","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"debug","ts":"2022-02-23 15:00:53.946","caller":"digestutils/digest_reader.go:77","msg":"digest match: c588c321757bf3563a15866801f02df5","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"debug","ts":"2022-02-23 15:00:53.946","caller":"storage/local_storage.go:134","msg":"wrote 40529 bytes to file /var/lib/dragonfly/01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5/10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924/data, piece 0, start 0, length: 40529","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","component":"localTaskStore"} {"level":"info","ts":"2022-02-23 15:00:53.946","caller":"peer/peertask_conductor.go:1167","msg":"peer task done, cost: 4348ms","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"debug","ts":"2022-02-23 15:00:53.946","caller":"storage/local_storage.go:175","msg":"update total pieces: 1","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","component":"localTaskStore"} {"level":"debug","ts":"2022-02-23 15:00:53.946","caller":"peer/peertask_conductor.go:1314","msg":"validate digest ok","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"debug","ts":"2022-02-23 15:00:53.946","caller":"peer/peertask_manager.go:301","msg":"delete done task 01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5 in running tasks"} {"level":"debug","ts":"2022-02-23 15:00:53.946","caller":"peer/peertask_conductor.go:1209","msg":"end piece result sent: <nil>, peer task finished","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"info","ts":"2022-02-23 15:00:53.946","caller":"transport/transport.go:259","msg":"download stream attribute: map[Content-Length:[40529] X-Dragonfly-Peer:[10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924] X-Dragonfly-Task:[01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5]]","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","component":"transport"} {"level":"info","ts":"2022-02-23 15:00:53.950","caller":"peer/peertask_conductor.go:1231","msg":"step 3: report successful peer result ok","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"} {"level":"info","ts":"2022-02-23 15:00:53.950","caller":"peer/peertask_conductor.go:672","msg":"single piece download success","peer":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924","task":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","component":"PeerTask"}

We find that scheduler use 4s+ to find a parent for the peer;
(By the way, there is a 8 hours gap between dfget and scheduler log timestamp, time zone problem.
Scheduler log:
core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.618","caller":"service/service.go:79","msg":"register peer task request: &scheduler.PeerTaskRequest{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(0xc0003ef400)}, sizeCache:0, unknownFields:[]uint8(nil), Url:\"http://10.40.162.87:5000/v2/wordpress-nydus/blobs/sha256:135348e46ccb98df38152269e88f6b5a13b2c1f25b155f3d8b7998d9c043e3f9\", UrlMeta:(*base.UrlMeta)(0xc00c770a10), PeerId:\"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924\", PeerHost:(*scheduler.PeerHost)(0xc006397220), HostLoad:(*base.HostLoad)(nil), IsMigrating:false}","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.651","caller":"service/service.go:83","msg":"task has been successful","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.681","caller":"service/service.go:109","msg":"task size scope is small","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.681","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.183.25-7-e0d9ff9e-46d4-46ee-8a45-74cb50ff4f2a is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.707","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.205.11-7-f6694a3f-41ad-49eb-b582-00ffcdb2e180 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.707","caller":"scheduler/scheduler.go:228","msg":"parent 10.228.233.167-7-48f5d70e-01d6-4477-ba62-9539d2d88941 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.708","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.183.38-7-4d71acf0-e467-418f-af73-4bc0c7538179 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.813","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.813","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.172.32-7-0de3d255-20fb-454e-9b95-1b3c99af6fef"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.939","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.939","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.205.147-7-a024dc35-85e0-4dff-a9dc-12f2a624d336"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:49.942","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.193.143-7-1086cda9-6dfb-4cfc-a2ee-39c35cd1f4b4 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.158","caller":"scheduler/scheduler.go:223","msg":"parent is not selected because it is same","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.372","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.182.150-7-17d264e8-63b5-4217-b3e6-4113b055e117 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.372","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.205.147-7-a024dc35-85e0-4dff-a9dc-12f2a624d336 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.493","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.227.40-7-c8e4b2c7-28c3-4d83-9f32-03e5348ab041 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.697","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.697","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.180.39-7-31a87028-f244-42eb-ba82-052158b12241"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.750","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.750","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.89.14-7-8fb4b308-c620-4577-b471-95e7813595e8"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.894","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.894","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.170.169-7-1056a398-de81-4b7c-871e-d6a338df29cd"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.896","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:50.896","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.229.150-7-4c270910-f44b-42dd-9113-587665dcea1b"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:51.316","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.189.159-7-a04c59d2-6231-4b8e-85c6-e42b123f2c45 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:51.318","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:51.318","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.89.13-7-31cad61f-1c43-4b38-a5a0-6bddba207bf2"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:51.546","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.181.157-7-412d6222-9c96-4aa9-a700-2a79ae2d4486 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:52.488","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.240.145-7-72b7a67c-c9f3-4b19-99be-051392638146 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:52.633","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.115.28-7-ccd24d79-417f-40ed-a4a8-6e21b5ad89b5 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:52.634","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:52.634","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.151.161-7-41d88415-3d60-40fe-97f4-980b350a3ebd"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.147","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.147","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.28.144-7-00b170b0-fd04-4747-a025-46aed1e960a3"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.369","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.369","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.205.21-7-fae3444d-6f69-4d72-b351-17382e26b576"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.608","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.118.161-7-149f3483-a87f-40bf-9ad9-2645e7bc66a9 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.900","caller":"scheduler/scheduler.go:228","msg":"parent 10.227.121.164-7-4d3021ac-ec24-4f81-98cc-251b2dbd8a11 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.906","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.906","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.187.156-7-b13b2458-b690-489e-a650-996e77bef440"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.909","caller":"evaluator/evaluator_base.go:177","msg":"peer is bad node because peer status is Pending","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.909","caller":"scheduler/scheduler.go:228","msg":"parent 10.229.84.149-7-788a1401-cfa6-491e-a36f-cde031b7d924 is not selected because it is bad node","taskID":"01bb460bf021d0848e7265c7bb653873161afccd5d5aba6e84e9e4503d0718a5","peerID":"10.227.156.26-7-26c9c6ad-8e40-4f7f-a34a-50a36d075e6c"} core-2022-02-23T23-00-58.463.log:{"level":"info","ts":"2022-02-23 23:00:53.959","caller":"scheduler/scheduler.go:252","msg":"candidate parents include []string

[fannaihao]

Long-tailed Dragonfly p2p task displayed above is one cause of the performance regression.
We also find that there need 400+ p2p tasks to start WordPress when we combine Nydus lazy pull and Dragonfly.