Lotus sync fall behind under load

[jsign]

I've a lotus node which as ~300 deals not in DealStorageActive.

At this point, I see that the node has a hard time syncing and falls behind. CPU usage is < 10% and memory looks far from being under pressure. Looks to me there might be some mutex contention or similar, but haven't digged in the pprofs.

I'm leaving here three pprof outputs that might be interesting to check out: goroutine, heap, and a 25s trace.

pprofStacksHeapAndTrace.zip

Lotus version: v1.1.0

[Kubuxu]

Can you also take CPU profile?

[jsign]

@Kubuxu , 45s CPU profile:
cpu45s.out.zip

[jsign]

@Kubuxu , this might be helpful too:
logs2hs.txt.zip

[jimpick]

Sounds like it could be this #4427 ... I was able to fix it by commenting out a single line of code

[jsign]

In this case, I see that it makes progress in lotus sync wait seeing Validated XXX messages, but in slow and small bursts. I feel this is pretty much related to the number of in-progress deals. I've >30 other Lotus nodes with less load and they sync up faster.

So maybe that change will help, but since this is the most important Lotus node that we have running I'm a bit off trying maybe some unsafe changes. That might be a good clue to devs though.