KeepAlive in etcd client cannot ensure the permanent validity of lease

[px3303]

KeepAlive in etcd client cannot ensure the permanent validity of lease. If automatic renewal of trillian lease is interrupted due to unexpected circumstances, trillian will still continue operating. However, the program will never be detected by ETCD server, and as a result it can no longer provide services. In addition, external process health check program cannot discover abnormalities and warnings cannot be sent to system administrators in time.

// AnnounceSelf announces this binary's presence to etcd.
func AnnounceSelf(ctx context.Context, client *clientv3.Client, etcdService, endpoint string) func() {
	if client == nil {
		return func() {}
	}

	// Get a lease so our entry self-destructs.
	leaseRsp, err := client.Grant(ctx, 30)
	if err != nil {
		glog.Exitf("Failed to get lease from etcd: %v", err)
	}
	client.KeepAlive(ctx, leaseRsp.ID)
...

// recvKeepAlive updates a lease based on its LeaseKeepAliveResponse
func (l *lessor) recvKeepAlive(resp *pb.LeaseKeepAliveResponse) {
	karesp := &LeaseKeepAliveResponse{
		ResponseHeader: resp.GetHeader(),
		ID:             LeaseID(resp.ID),
		TTL:            resp.TTL,
	}

	l.mu.Lock()
	defer l.mu.Unlock()

	ka, ok := l.keepAlives[karesp.ID]
	if !ok {
		return
	}

	if karesp.TTL <= 0 {
		// lease expired; close all keep alive channels
		delete(l.keepAlives, karesp.ID)
		ka.close()
		return
	}
...

Meanwhile, because trillian did not consume LeaseKeepAliveResponse channel immediately, ETCD client would print a large number of warnings #2249 .