Over-consumption of Consul connections

[loicalbertin]

Bug Report

Description

When Yorc runs a large number of async operation (typically for jobs monitoring) we noticed an over-consumption of connections to Consul. This is due to the way we monitor tasks errors and cancellation.
We use a long poll connection on tasks flags but we do not properly use context cancellation to detect that we could stop the monitoring.

yorc/tasks/tasks.go

Line 636 in e8759d1

kvp, qMeta, err := consulutil.GetKV().Get(path.Join(consulutil.TasksPrefix, taskID, flag), &api.QueryOptions{WaitIndex: lastIndex})

This leads to consul connections being still waiting for an HTTP timeout to be released while they are not useful anymore.
When action scheduling interval is low or in case of many different actions then we can consume a very large number of Consul connections on this.