What are the metrics to monitor an OpStack chain?

[0xtrou]

Issue Description

I've made my OpStack chain recently, so I would love to know what the metrics to monitor an OpStack chain, to make it high-availability and secured. I know the question is kinda general, but I'm kinda new to OpStack. I'm welcoming and eager to learn all the ideas/handy tips/documents ... from you folks.

Also this discussion would greatly contribute to the OpStack community as well.

Thank you.

Additional Information

No response

[opfocus]

Here are some metrics mentioned at https://community.optimism.io/docs/developers/bedrock/metrics/#. I'd appreciate your feedback on these, and then we can continue discussing them.

[sbvegan]

I went ahead and changed this discussion to General so we can keep this thread open ended. I'll see what information and resources I can dig up for you.

[sbvegan]

@0xtrou I asked our developer infrastructure team about this and here are some suggestions for monitoring your chains health. _Please note:_these are values are configurable and should depend on how sensitive you want the alerts to be. Starting on a high level:

• Node is synced, block height is increasing
• Block timestamp is close to wall clock time
• Unsafe head is ahead of the safe head (i.e. receiving blocks from the sequencer)
• Block hashes match between the sync test & a live replica

Here are other scenarios that would be cause for concern:

• L1 heads have not increased on the sequencer for more than 10m
• L2 unsafe heads have not increased on the sequencer for more than 5m
• L2 unsafe heads are increasing at less than 2 blocks/s on the sequencer for more than 5m
• L2 safe heads have not increased on the sequencer for more than 15m
• L2 unsafe heads have not increased on a replica for more than 5m
• Sequencer experienced a derivation error
• Sequencer experienced a sequencing error
• Slow block building
  • 95th percentile sequencer block building time over a 10m interval is > 1000ms
  • 90th percentile sequencer block building time over a 10m interval is > 700ms
  • 75th percentile sequencer block building time over a 10m interval is > 500ms

[opfocus]

I have collected some data related to monitoring nodes:

1. Node metrics mentioned in the OP documentation, implemented using Grafana Cloud:
   Grafana Cloud, in combination with Prometheus. To configure Prometheus on Grafana Cloud, there is a guide on the account page.
   Metrics
   requires additional commands ：

op-node \
....
--metrics.enabled \
--metrics.addr=0.0.0.0 \
--metrics.port=7300

2. Details mentioned in the Geth documentation: Monitoring Geth with InfluxDB and Grafana

3. A reference to an already integrated example: simple_node_dashboard

I've tried implementing the first one. Next, I will attempt other content, including some key metric monitoring as mentioned by Soyboy.

[Padraic-O-Mhuiris]

#516