Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve BN health check #1732

Closed
corverroos opened this issue Jan 31, 2023 · 0 comments
Closed

Improve BN health check #1732

corverroos opened this issue Jan 31, 2023 · 0 comments
Assignees
Labels
protocol Protocol Team tickets

Comments

@corverroos
Copy link
Contributor

Problem to be solved

BN health check didn't detect that a lighthouse BN wasn't healthy (synced) when it couldn't sync due to not having any connectivity to the p2p network. The readyz endpoint reported Healthy. Only once the network issues were resolve, did the node start syncing and did the readyz turn to "BN is syncing".

See https://discord.com/channels/930779837218562078/1069932708689629234/1069932710849683466

Proposed solution

Reproduce and look at the value of NodeSyncing response:

// SyncState is the data regarding the node's synchronization state to the chain.
type SyncState struct {
	// HeadSlot is the head slot of the chain as understood by the node.
	HeadSlot phase0.Slot
	// SyncDistance is the distance between the node's highest synced slot and the head slot.
	SyncDistance phase0.Slot
	// IsOptimistic is true if the node is optimistic.
	IsOptimistic bool
	// IsSyncing is true if the node is syncing.
	IsSyncing bool
}

SyncDistance might be high or HeadSlot might be zero.

Once the data has been identified, incorporate this check and new readzy value into app package.

@github-actions github-actions bot added the protocol Protocol Team tickets label Jan 31, 2023
@xenowits xenowits self-assigned this Feb 3, 2023
obol-bulldozer bot pushed a commit that referenced this issue Apr 1, 2023
Adds a new `readyz` error for scenarios when beacon node cannot sync because of zero peers. Without this fix, the grafana healthy panel shows "OK".

`IsSyncing: false` means that the "Beacon node is not syncing" because either
- "It is fully synced" OR
- "Can't sync as it has no peers" 

category: bug 
ticket: #1732
@xenowits xenowits closed this as completed Apr 3, 2023
LukeHackett12 pushed a commit that referenced this issue Apr 11, 2023
Adds a new `readyz` error for scenarios when beacon node cannot sync because of zero peers. Without this fix, the grafana healthy panel shows "OK".

`IsSyncing: false` means that the "Beacon node is not syncing" because either
- "It is fully synced" OR
- "Can't sync as it has no peers" 

category: bug 
ticket: #1732
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
protocol Protocol Team tickets
Projects
None yet
Development

No branches or pull requests

2 participants