av-store: granular pruning of data

[sandreim]

Currently we block the subsystem for whatever times it takes to prune the data both on timer and on finality. We should make this process more granular as to allow the processing of messages in between otherwise we can end up in a situation where this takes > 10s and node crashes due to SubsystemStalled error.

polkadot/node/core/av-store/src/lib.rs

Line 617 in 19fdd19

prune_all(&subsystem.db, &subsystem.config, &*subsystem.clock)?;

In the past I have seen very high times up to 10s on a few nodes, but not as of recently on tests with small PoVs.

[bkchr]

Do you know what takes the time? The actual IO when applying the TX? Or the processing of what to delete? If it is the later, we should just move this to some background process.

[sandreim]

Do you know what takes the time? The actual IO when applying the TX? Or the processing of what to delete? If it is the later, we should just move this to some background process.

I am not sure which part exactly is heavy. Before moving this as a background task we'd have to be certain that we don't corrupt up the db with multiple writers/readers on same column.

[bkchr]

If the db is Sync & Send, I would assume that you can read and write from multiple threads :P

[sandreim]

Yes it is, but usually we do this from a single subsystem thread. Maybe in this case, it shouldn't be an issue as the keys should no longer be accessed by anything since this will happen after 25h hours.

[alexggh]

Moving prune_all to a separate blocking task: #7263

[alexggh]

Fixed with: #7263