[RFC] Rework log levels

[mratsim]

As we approach launch, user experience is paramount and we need to review our current logging. In particular several components have significantly matured and might not need the logging we once required during development.

The current debug and trace are very verbose, and filling disks really fast.

In contrary, the "NOTICE" log level is very dry, this is an example of a non-validator node following Spadina for 4 hours after restart in NOTICE

It looks like nothing is happening. Some suggestions for user oriented log at the Notice level

• A "Finality reached" log, this is somewhat different to the log "Finalized with rule XYZ" that we have as that one is also displayed on block replay. The heartbeat happen rarely enough and is significant enough.
• Warning on low participation level or every X epochs of non-finality
• Fork in the network
• High volume of slashing

And also everything related to the users own validators (attestation sent, block proposed, slashings)

[mratsim]

With have a couple of usability problem with trace logs, there are a couple of messages that are per peer but together flood the logs.

Libp2p (1 peer is repeated a lot but you receive one control message per peer per ~1 sec).

Fork choice

While we have or can write log tooling, we also don't want to fill disks too fast.

[stefantalpalaru]

You can do something similar to https://github.com/status-im/nim-libp2p/blob/03f5bbba6ddd5012753debd4b677df9f5822755d/libp2p/protocols/pubsub/gossipsub10.nim#L94

[mratsim]

Closed by #1788. Though we'll likely want to revisit them once again just before launch depending on what we find useful/useless in debug mode to quickly react to eventual launch issues.