Test: TestCtlV3Lock

[xiang90]

--- FAIL: TestCtlV3Lock (16.15s)
    ctl_v3_test.go:138: test timed out for 7s

[gyuho]

--- FAIL: TestCtlV3Lock (1.11s)
    ctl_v3_lock_test.go:91: expected different lock name, got l1="a/186d573fc1f99c04\r\n", l2="Error:  lost watcher waiting for delete\r\n"

Got another failure

[justinruggles]

I'm getting hanging/timeouts when trying to obtain a lock with concurrency.Mutex when using master. Using release-3.0 branch works fine. The hang appears to be in waitForDelete(), so it may be related to this.

[xiang90]

@justinruggles It would be great if you can help to debug this if you can reproduce. Also it would be useful to share how to reproduce it.

[justinruggles]

When I tried again using etcd server from master there was no problem, so I think this was just an issue with using release-3.0 etcd server with a master branch client. Sorry for the noise.

[xiang90]

@mitake @AkihiroSuda I had hard time to reproduce this. Can you try to help?

[mitake]

@xiang90 ok, I'll try it out

[mitake]

I tried to reproduce the fails with more than 2000 iterations by namazu process inspector (that increases randomness of thread scheduling) but couldn't reproduce. How long did you need to produce the above fails? @xiang90 @gyuho

[xiang90]

@mitake I cannot reproduce it myself. :( I suspect that it might be caused by CPU starvation. But I am not sure.

[AkihiroSuda]

Didn't hit as well..

[heyitsanthony]

Still happening. https://semaphoreci.com/coreos/etcd/branches/master/builds/1034

[heyitsanthony]

I was able to reproduce ctl_v3_test.go:138: test timed out for 7s by putting a if rand.Intn(2) == 0 { time.Sleep(15 * time.Second) } right after waitKeys in Lock() (the 7s is misleading; the timeout is really 15s). This simulates a broken watch cancel path where the watch channel never closes out after a cancel.

I don't think locks are broken here because otherwise the integration lock tests would be exploding as well. The difference from those tests and e2e is the e2e test will cancel a blocked lock (and therefore the watch) when it sends a SIGINT to the process. I suspect this is fixed by the watch refactor.

[xiang90]

@heyitsanthony Sounds reasonable. Can we:

1. fix the wrong timeout printing
2. panic when there is a timeout instead of fatal to print out the stack trace. if it happens again, it is easier for us to debug.

[heyitsanthony]

1. is OK
2. panic() probably won't do much for e2e debugging. It'll catch the stalled process Close but the interesting traces will be in separate processes. Some infrastructure around the process stuff that would SIGQUIT + collect the resulting stack traces would be much more useful.

[xiang90]

@heyitsanthony right. let's close this for now.

[heyitsanthony]

Still seeing timeouts,
https://semaphoreci.com/coreos/etcd/branches/master/builds/1274

[xiang90]

i have seen this for quite a while, and cannot reproduce it. so closing this for now.