Add option to further filter Consul Watch -type=checks -state=X by Tag

[mcousens]

Feature Description

We are requesting to add the ability to further filter Consul Watch type=checks state=<state> data by -tag to monitor a more narrow scope of data. Ideally, we'd love to be able to filter by -service as well to narrow it even further.

currently allowed:
consul watch -type=checks -state=critical

ideal state, example:
consul watch -type=checks -service=redis -state=critical -tag=autoscaling_group

Use Case(s)

AWS Autoscaling Groups provide a set-instance-health endpoint so that you can mark an instance as unhealthy from an external system.

When using Consul for service discovery, you have a variety of health checks for nodes that can let you know when a service is in a critical state and the application may not be able to run correctly. By monitoring the Consul health checks via a Consul Watch, you can call the AWS set-instance-health endpoint when a node has a critical health check so that the AWS Autoscaling Group can replace the instance.

We use the below Consul Watch to monitor critical health checks for our AWS Autoscaling Groups so that we can mark the node as Unhealthy in the AWS ASG and replace the instance with a healthy one.

consul watch -type=checks -state=critical

Because we can't filter checks in the critical state by Tag, we must monitor critical health checks across our entire system and constantly ingest and parse the input to find relevant health checks. It would be very helpful if we could tell the Consul Watch to only monitor critical health checks for nodes that are a part of the autoscaling group via a tag. Ideally, we could also further filter by Service to get a very specific view of what you need:
consul watch -type=checks -service=redis -state=critical -tag=autoscaling_group

[huikang]

@mcousens , thanks for your suggestions. This sounds a good enhancement and we will look into it.

[huikang]

@mcousens , we would like to gather more information from you regarding the feature request. The current version (1.15) of consul watch CLI supports the following options

-type - Watch type. Required, one of "key, keyprefix, services, nodes, service, checks, or event.
-state - Check state to filter on. Optional for checks type.
-service - Service to watch. Required for service type, optional for checks type.
-tag - Service tag to filter on. Optional for service type.

So, -tag doesn't supports the checks type. Based on your suggestions, are you proposing to make -tag support the checks watch type? Thanks.

ideal state, example:
consul watch -type=checks -service=redis -state=critical -tag=autoscaling_group

I noticed -service is an optional for checks type, so you can use it in the above ideal state.

[mcousens]

Hi @huikang, yes exactly - we are proposing to make the checks watch support the -tag parameter.

Ex:

consul watch -type=checks -state=critical -tag=autoscaling_group

To clarify the additional note we made about also being able to filter by -service, you cannot use -service with -state currently, it outputs the below error message:

$ consul watch -type=checks -service=redis -state=critical
Cannot specify service and state

While it would be a nice feature to also be able to filter by -service when using -state our priority is more on adding -tag. With the addition of -tag we can then monitor all critical health checks on a certain group (in our use case, nodes that our in our autooscaling group)

[huikang]

@mcousens , thanks for your response. That totall makes sense. We will look into it and get back to you soon.

[huikang]

@mcousens , we figured out a better solution to address both cases with -filter. Please checkout the example in this PR.

[david-yu]

Will go ahead and close this issue @mcousens as the PR which addresses this issue is now merged. Thanks for the feedback!

[david-yu]

We'll go ahead and try to backport to 1.16.x to get this out in the next 1.16.1 patch release.

[mcousens]

Apologies for the late reply, I was out of the office - thanks so much for the quick turnaround on this feature!