ui: node reconnection events confused with "dead" node events

[dhartunian]

Describe the problem

From customer:

I deployed docker images of cockroach and all nodes are connected. But I see duplicate nodes in the admin UI.
For some reason of network issue/deployment problem, 3 of the nodes got disconnected and admin UI displayed these as dead.
When network is restored, these 3 nodes are restarted. Then these nodes got added and admin UI displayed that these are healthy. But the problem is those same nodes which are restarted are shown twice in admin UI, once as dead and other as healthy with exactly identical IP:PORT (only node number varies). So let’s say, instead of 10 nodes admin UI displays total nodes as 13 and shows 3 as dead. But those 3 dead ones became healthy after restarting.

To Reproduce

Expected behavior
A clear and concise description of what you expected to happen.

Additional data / screenshots
If the problem is SQL-related, include a copy of the SQL query and the schema
of the supporting tables.

If a node in your cluster encountered a fatal error, supply the contents of the
log directories (at minimum of the affected node(s), but preferably all nodes).

Note that log files can contain confidential information. Please continue
creating this issue, but contact [email protected] to submit the log
files in private.

If applicable, add screenshots to help explain your problem.

Environment:

• CockroachDB version [e.g. 2.0.x]
• Server OS: [e.g. Linux/Distrib]
• Client app [e.g. cockroach sql, JDBC, ...]

Additional context
What was the impact?

Add any other context about the problem here.

Jira issue: CRDB-2996

[dhartunian]

Possibly related to: #59322 and #58026 and #50707 and #50309

[thtruo]

Closing this issue as we have not been able to reproduce the issue. KV Observability will continue to address the related issues tagged in the previous comment