backport: logs splitting

[cc32d9]

EOSIO/eos#9184
EOSIO/eos#9277
EOSIO/eos#9821
EOSIO/eos#9829
EOSIO/eos#10037
EOSIO/eos#10061
EOSIO/eos#10548
EOSIO/eos#10561

[matthewdarwin]

The key issues for log rotation is to capture 3 use cases:

• keep everything (existing 2.0) code
• keep some amount of blocks, and then throw out everything older
• don't keep anything

Need to support blocks.log and ship.

[matthewdarwin]

--- below is sent to mandel upgrade telegram and a few other places ---

Hello all,

As part of mandel 3.1 plans, it has been previously discussed that the "blocks.log splitting feature" is a necessary requirement for the release to be acceptable to the community. Recently, a question arose, as to what is the actual requirements necessary for 3.1 related to this feature.

It is my (and a few other people's view) that the desirable list of features for mandel 3.1 is as follows:

1. keep all the blogs.log, in a single file, as per nodeos 2.0 (no change). Starts from beginning or based on a snapshot.

2. keep some recent blocks, in one file (or optionally more files at the discretion of the developers), and remove old blocks when the configured number of blocks passes

• this is less features than is in 2.1 where it can split files, move them to archive directory, and other things

3. don't store any blocks at all (this is a new feature, not in 2.1)

If you have opinions one way or another, please add thoughts to the github issues:
#100
#92

@daniel_keyes I suggest mentioning this in the next meeting.

[brianjohnson5972]

One thing to remember in this decision is that even if we have to do a re-write of the existing stride implementation for block-log and ship, the unit tests and integration tests are likely a direct port and I would estimate that (not sure if there are ship integration tests) between a 1 and 2 week effort. That, in my mind, would lean toward the existing design (that assumes that there are not identified disconnects between desired use-cases and how the strides feature works).

[spoonincode]

If the existing design can just be cherry-picked over, that would be a significant reason to take it as is. But my understanding was that it was interwoven with CFA pruning stuff. (I haven't actually looked, so feel free to correct me)

If it needs to be refactored at all, I would encourage refactoring for simplicity as long as the requirements allow for it. The requirements seem to simply be: allow setting a maximum size for block log & ship log; where 'size' can be implemented as either block count or total byte size. That should allow a fair bit of simplification in the design.

[matthewdarwin]

I would prefer number of blocks rather than size in bytes. It keeps things more predictable from a user perspective. "I have 7 days of blocks". (of course it is less predicable from a disk space usage perspective)

[spoonincode]

bytes is probably easiest to implement because it's quite easy to just have a 1GB file (or whatever configured size) that is simply treated as a circular buffer that endlessly wraps around

[brianjohnson5972]

@spoonincode Kevin had assessed it as a re-write. My point was just that there are extensive tests already written, that should be able to be used as-is, which would significantly reduce the re-write time versus creating a simpler design and implementing that.

The existing stride design is based off of a "stride"(range) of blocks, so it is blocks rather than size based.

[matthewdarwin]

PRs for this are merged into 3.1.0:

• block log pruning: optionally reduce disk usage of block log file during operation by periodically trimming blocks #342 block log pruning: optionally reduce disk usage of block log file during operation by periodically trimming blocks
• ship log pruning: optionally reduce disk usage of ship log files during operation by periodically trimming logs #327 ship log pruning: optionally reduce disk usage of ship log files during operation by periodically trimming logs