Raft state machine tweaks

[neilalexander]

This PR does the following:

1. Replaces state with an atomic, which can be accessed lock-free instead of needing to take the group mutex
2. Ensures that runAsFollower, runAsLeader, runAsCandidate can only continue running as long as the state is correct (they will stop looping and hand back to run if not)
3. Removes the isLeader atomic, as it is now unnecessary with the state atomic
4. Adds a unit test to prove that we can now recover from the situation of all nodes being forced into a leaderless follower state, which now passes with the above changes

Signed-off-by: Neil Twigg [email protected]

[neilalexander]

Currently investigating:

=== RUN   TestJetStreamClusterDetectOrphanNRGs
panic: close of closed channel
goroutine 72044 [running]:
github.com/nats-io/nats-server/v2/server.(*raft).shutdown(0xc014fc6300, 0x0)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:1509 +0xb2
github.com/nats-io/nats-server/v2/server.(*raft).Stop(0xc012ca67a0?)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:1494 +0x29
github.com/nats-io/nats-server/v2/server.(*jetStream).monitorConsumer(0xc0043a9000, 0xc00024b900, 0xc00171da70)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:4603 +0x1533
github.com/nats-io/nats-server/v2/server.(*jetStream).processClusterCreateConsumer.func1()
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/jetstream_cluster.go:4295 +0x47
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:3700 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 71984
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:3698 +0x1bb
FAIL	github.com/nats-io/nats-server/v2/server	370.200s
FAIL

[derekcollison]

LGTM

[derekcollison]

LGTM