etcdctl elect results in deadlock: won election never reported to winner, if key is a prefix of existing key

[glycerine]

It appears that a lone candidate cannot win some elections.

go1.7
osx 10.11.6 El Capitan
export ETCDCTL_API=3 in the env
etcd at tip, as of this writing, tip was: c388b2f22f1285e0a686d0594c85a053016df010

# terminal 1: start etcd
$ etcd
...

# terminal 2: create a key 'xyz'
$ etcdctl put xyz hello

# terminal 2: and run an election on key xy (note that xy is a prefix of xyz)
$ etcdctl elect xy xy-should-win
observed result: <hangs, no output, as if the election was lost>

expected result: output that the value xy-should-win is now the value of xy, so we know that we've won the election.

terminal 3: query all values, notice that xy-should-win has one the elction, but was never notified of this fact
jaten@jatens-MacBook-Pro ~ $ etcdctl get a z
xy/694d56cc2663a25c
xy-should-win
xyz
hello
jaten@jatens-MacBook-Pro ~ $ 


additional note: when reproducing this under 3 etcd servers in a cluster, the following is a typical 2-line complainst also issued in the etcd logs. Several examples follow:

------------------
14:59:16 etcd2 | 2016-08-27 14:59:16.233151 W | etcdserver: apply entries took too long [17.5324\
48ms for 1 entries]
14:59:16 etcd2 | 2016-08-27 14:59:16.233171 W | etcdserver: avoid queries with large range/delet\
e range!
-===============
15:04:24 etcd3 | 2016-08-27 15:04:24.750401 W | etcdserver: apply entries took too long [11.377838ms for 1 entries]
15:04:24 etcd3 | 2016-08-27 15:04:24.750421 W | etcdserver: avoid queries with large range/delete range!
-------------
15:04:58 etcd2 | 2016-08-27 15:04:58.998189 W | etcdserver: apply entries took too long [17.416764ms for 1 entries]
15:04:58 etcd2 | 2016-08-27 15:04:58.998202 W | etcdserver: avoid queries with large range/delete range!
==================
15:05:53 etcd3 | 2016-08-27 15:05:53.874651 W | etcdserver: apply entries took too long [11.660702ms for 1 entries]
15:05:53 etcd3 | 2016-08-27 15:05:53.874676 W | etcdserver: avoid queries with large range/delete range!

[xiang90]

@glycerine The prefix conflict bit should be improved.

For the query warning issue, are you running etcd on HDD? It seems that the fsync takes more than 10ms. We probably will make the warning time longer to 100ms or so. At least, etcd should not spam warning on normal HDD.

[xiang90]

/cc @heyitsanthony

[heyitsanthony]

The client code is watching on xy* instead of xy/* so it's picking up 'xyz', which is a bug. Mutexes are also affected. Also missing docs about what part of the keyspace it'll eat.

[glycerine]

For the query warning issue, are you running etcd on HDD?

I'm running on SSD; stock Apple SSD 512G.

[xiang90]

@glycerine Just noticed that you were running 3 members on the same osx machine. Probably there are IO or CPU contentions. It is unrelated to this bug.

[glycerine]

Just noticed that you were running 3 members on the same osx machine. Probably there are IO or CPU contentions. It is unrelated to this bug.

Agreed. Let's ignore that part.