Make it easier to debug split brains and Raft-related Consul problems

[slackpad]

While interactively debugging an issue with @sean- a number of improvements were noted:

1. It's hard to tell who a server is replicating TO, add IPs to the logs.
2. It's hard to tell who the current leader is without logging into each and looking at consul info, make consul members show who the current leader is.
3. Consider adding a consul peers type command that shows the Raft details in a clear, parseable format. The peer set version is hard to understand, expose the Raft index.
4. Add better metrics for Raft-related events such as leader elections. Right now we can only look at events which are output by all the nodes, so it's really hard to see the actual number of leadership transitions.

[highlyunavailable]

#1378 already solves item 2 there, it just needs merging.

[trumant]

@slackpad - I've been thinking that it would be great expose an HTTP agent endpoint at /v1/status/info and perhaps another at /v1/status/members that would provide the info currently emitted in difficult to parse format from consul info and consul members.

I'd be happy to contribute a PR for this type of change if there is some agreement that it makes sense/would prove useful.

[trumant]

More fleshed out proposal: https://github.com/trumant/consul/blob/c1241c317f49edd6cfe0d9d69a8c201c1fb2cf79/website/source/docs/agent/http/status.html.markdown

[Dmitry1987]

Hi, I see it's an old topic, was it implemented?
how it's possible to contribute btw, and what are the chances for submitted code to be merged?

[slackpad]

Hi all - this hasn't yet been implemented. I'd hold off on working on this for right now - @sean- is looking at a cluster-level status endpoint that would support this type of info. We should have more details in a bit.

[CpuID]

@slackpad or @sean- do either of you have any updates on this issue?

[CpuID]

It looks like 6cf21fb exposes the current raft peers list via consul info (item 3), which is a start.

[CpuID]

Confirmed #1849 being merged resolves item 2 - appears in consul info also.

[slackpad]

This should be done now based on these features:

• https://www.consul.io/api/operator/raft.html
• https://www.consul.io/docs/commands/operator/raft.html
• https://www.consul.io/docs/guides/autopilot.html (in general)
• https://www.consul.io/api/operator/autopilot.html#read-health (in particular)