Very large block validation times during / after chocolate update

[travisperson]

We are running v1.13.0 on a few nodes with the experimental split-store enabled. A few of these nodes experienced desync issues immediately after the chocolate update (one node had zero issues) with block validation times up to 200+ seconds.

Profiles: https://bafybeiesexrmvtw7kftq4dul5hgbet7izan43imyhttphamwgg54bn3sd4.ipfs.dweb.link/

[travisperson]

The nodes seems to have recovered on their own at first, but then started to desync later. I restarted all the nodes with issues and they quickly started to sync again. I will monitor these nodes, and report if the issue occurs again.

I would like to review the above profiles with someone to understand what was actually happening during these times though.

[vyzo]

So it seems that there were problems with the migration.
My theory is that the premigration got garbage collected as unreachable and it left us running the migration online or something along these lines.

One possible solution for this problem is to register a protector for migrated state so that it doesn't get garbage collected.

[BigLep]

I'm assuming that as part of addressing this, we'll include a regression test as well. Let me know if that isn't the case.

[vyzo]

The simplest solution is to inhibit compaction around upgrade/migration; this will sidestep the issue in a non-intrusive way.
We will also want sortless compaction to make sure we can compact large state trees safely.

[vyzo]

Fix in #7734.

[TippyFlitsUK]

Hi @vyzo

This looks like a reminder. Should we close it or keep it open?

[vyzo]

afaict it should be fixed.

[TippyFlitsUK]

Many thanks!! Fix in #7734.