Allow registration of service in maint mode

[hopperd]

It would be nice to be able to register a service and have it be in maintenance mode upon registration. This way you could bring a service online in maintenance mode to be able to run tests against the node and verify it before making it directly available.

There is a "clumsy" way to work around this by registering the service with a TTL that will be failing and then put the node into maint and remove the TTL based check but this seems a little clunky and cumbersome when instead we could possibly provide a flag to the service registration like

"Maintenance": "disable | enable"

With the default being disable.

[ryanuber]

Hey @Split3,

You could accomplish what you need by registering a health check with your service and passing the critical health state. The service and its health check should be registered together, so there shouldn't be any need to do anything else until you are ready to bring the service into the pool, at which point you would simply deregister the failing health check.

Maintenance mode could be used to do something like this, and in fact you can "trick" consul into the exact functionality you want by naming your initial failing health check _service_maintenance. You could then use consul maint -disable to remove it, though this is a hack. I'll leave this ticket open for now to see if there is more interest in the additional flag.

[hopperd]

Hey @ryanuber,

Unfortunately I thought the same thing and tried these approaches already. The problem is a chicken and egg issue. You can't register a check to a service if that service doesn't exist yet, you will get an error.

I also tried to register the "maintenance mode" check by using the naming convention when the service itself is registered, but those values are ignored and a new checkId is generated for it when combined in the service registration. The only way I was able to accomplish was the method I mentioned earlier about the TTL.

[ryanuber]

@Split3 I see, I guess the agent endpoints currently don't cater to this use case. I was thinking about it from a config file perspective, which should allow services and checks to be registered at once. Let's leave this open to track the request. Thanks for the follow-up.

[mdaffin]

You might be hitting #740 when you register the check inside the service. It can be worked around if you can register the services and checks in the configs but I don't think it can via the http end points.

[hopperd]

Yes doing this via the config files would work, sorry I wasn't clear about it being the HTTP API, but we will have a continuous delivery pipeline that deploys multiple services on various hosts (i.e. docker) so ideally we just want to use the API to register/deregister the services are part of the deployment pipeline instead of writing out the configs.

@James147 Yes indeed this would help for sure as it was initially what I tried to do. I suppose if #740 was implemented then this issue could be closed to just use that approach instead of introducing a new mechanic to the configuration.

[bagelswitch]

I'm also facing a problem (new services registered with critical-state checks by nomad are briefly in a passing state) that I'm not sure is a bug/issue exactly, but would be addressed nicely by this if leveraged by nomad.

I suspect anyone using consul for service discovery with nomad in a similar fashion would have the same problem:
https://groups.google.com/forum/#!topic/consul-tool/mpUDUHG0hdQ

[hanshasselberg]

It seems to me that the goal is to register a service with Consul and not send traffic to it until it is verified. This is possible with Connect L7 Traffic Management: https://www.consul.io/docs/connect/l7-traffic-management.html now. Closing this issue now. Feel free to let me know if that doesn't solve it for you though.

[bagelswitch]

I can't speak for the original poster, but in my case, no, "use consul connect" is not a solution - we're using a linkerd-based L7 mesh with nomad and consul, not consul connect.

I'm moved to quote from Nomad's marketing material:

Nomad only aims to provide cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery and Vault for secret management.

[hanshasselberg]

I absolutely understand what you are saying @bagelswitch! I wouldn't be opposed to review a PR that implements this functionality. But from our side there is little reason to do this work ourselves since Consul is already capable of doing it.

[bagelswitch]

Understood - this is by no means a show-stopper for us, in any case. Just wanted to express support for the composable nature of the different Hashi components, as this is why we chose them in the first place :)