Allow clients to request watch heartbeats to refresh the clients window

[smarterclayton]

In 0.4.6, etcd tracks a specific window of changes to the store and allows watches on indices within that window (from etcd index 1034, a client can watch starting from index 35 onwards). From a client that wants to atomically watch for all changes that occur to a subset of keys, the pattern is

• get a key or list of keys or recursive list of keys K, take the etcd index of the response X
• loop
  • begin a watch from that etcd index X
  • track the index of any changes that occur during a watch, and update X
  • If disconnected from the server, continue loop

In some cases, no changes to the keys K occurs before 1000 changes occur to etcd (index reached X+1000). A client watching the server does not receive a notification that their watch has exceeded the window, which means if they disconnect and reconnect, they must fetch the key set K again and begin watching. For very expensive sets this may create load disproportionate to the rate of change of the set, and clients must frequently reconnect.

As discussed in #2039, only the server knows when the window has passed without change. A client must receive a notification from the server to detect that their watch window has been closed. It would be valuable to clients to allow the client to request the server to close the watch when the window expires (or in the future with the new store, after some configurable time interval / index count) and to return the etcd index at the time the watch closes. This allows a client to reset their window to that value and continue watching, vs recalculating a set of results.

I would be happy to submit a pull if this is reasonable.

[bgrant0607]

cc @hchoudh

[xiang90]

@bgrant0607 @smarterclayton @hchoudh

see #2468 (comment)

[wojtek-t]

cc @wojtek-t

[xiang90]

fixed by #4674

[wojtek-t]

Awesome!

@lavalamp @timothysc - FYI