Honor "Connection: close" headers on all endpoints

[nugend]

Description of the Issue (and unexpected/desired result)

Connections made to Consul with the "Connection: close" header set are not closed by consul and the ephemeral ports remain in TIME_WAIT state for the timeout duration.

We noticed this after doing some testing with single-shot service discovery requests.

Reproduction steps

Make a request to any consul agent API endpoint with the "Connection: close" header present

consul version for both Client and Server

Client: 1.0.2
Server: 1.0.2

consul info for both Client and Server

Client:

agent:
        check_monitors = 0
        check_ttls = 96
        checks = 102
        services = 102
build:
        prerelease =
        revision = b55059f
        version = 1.0.2
consul:
        bootstrap = false
        known_datacenters = 1
        leader = false
        leader_addr = REDACTED:8300
        server = true
raft:
        applied_index = 326279
        commit_index = 326279
        fsm_pending = 0
        last_contact = 72.133421ms
        last_log_index = 326279
        last_log_term = 2
        last_snapshot_index = 319606
        last_snapshot_term = 2
        latest_configuration = [{Suffrage:Voter ID:347fe6e0-8dd3-dd31-e319-47ee34edd09c Address:REDACTED:8300} {Suffrage:Voter ID:9cc851b9-612e-bbfc-ae8e-16c1fb96c392 Address:REDACTED:8300} {Suffrage:Voter ID:5e47310a-5fb9-8785-1ed8-8f7dafc00cd7 Address:REDACTED:8300}]
        latest_configuration_index = 1
        num_peers = 2
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Follower
        term = 2
runtime:
        arch = amd64
        cpu_count = 24
        goroutines = 345
        max_procs = 24
        os = linux
        version = go1.9.2
serf_lan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 2
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 4
        members = 3
        query_queue = 0
        query_time = 1
serf_wan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 5
        members = 3
        query_queue = 0
        query_time = 1

Server:

Same as above

Operating system and Environment details

Linux REDACTED 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Log Fragments or Link to gist

2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/catalog/services (4.625461ms) from=127.0.0.1:32908
2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/health/service/db-24292fbf-3b57-4d25-bbbf-7d4359a6b6fc?passing (1.143435ms) from=127.0.0.1:32910
2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/health/service/db-30aa0fe9-2340-5530-c5d6-fd1ed1b8e96a?passing (818.99µs) from=127.0.0.1:32912
2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/health/service/db-4436aab6-a2cf-d83a-8deb-6b137ded35a6?passing (817.246µs) from=127.0.0.1:32914
2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/health/service/db-4a100e6e-cde4-8800-0469-7b413000b50d?passing (700.332µs) from=127.0.0.1:32916
2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/health/service/db-8e317232-0402-29f5-51a8-d44413242802?passing (766.036µs) from=127.0.0.1:32918
2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/health/service/db-8fc61925-3d48-b712-4451-8d7311af2212?passing (813.841µs) from=127.0.0.1:32920
2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/health/service/db-c305f123-f3cf-61db-6a0e-d5646ec33d4f?passing (1.119161ms) from=127.0.0.1:32922
2018/03/09 16:02:51 [DEBUG] http: Request GET http://localhost:8500/v1/health/service/db-e1554c2f-68fe-2cbc-28e9-19ecef79ccee?passing (1.092437ms) from=127.0.0.1:32924

[nugend]

Is there a good reason that the header is ignored?

[mkeeler]

@nugend Which side of the connection is remaining in TIMED_WAIT. Is it the server or client side?

[pearkes]

Given we haven't heard anything based on our suggestions/questions above I'm going to close this issue, but I encourage you to comment and we can re-open it if you want to pick this up again.

Alternatively, if things have changed dramatically, feel free to create a new issue or PR.

[nugend]

We changed our HTTP library to get around this and some other issues we had with it.

So... I don't care anymore, but it's probably still broken.