Error while using SRV records in upstream targets

[smalot]

Summary

In a Kubernetes environment, we try to use SRV records in upstream target. Each time we have a http code 500 error.
But with A records, everything works.
We double check the SRV DNS entry and it's correctly set.

Steps To Reproduce

In a Kubernetes environment (in Google Cloud Engine):

1. Create an Upstream (service_user)
2. Add a target using a SRV DNS record (_nginx._tcp.service-user-service-user-nginx.default.svc.cluster.local) - (verify the service is available using dig command line)
3. Create an API (Service User) based on on Uri (/user) and refer the Upstream previouly created
4. Call the kong-proxy using the "/user" in the url

The response is a http code 500 with this error message:

An unexpected error occurred

Additional Details & Logs

• Kong version ($ kong version)

0.11.0

• Kong debug-level startup logs ($ kong start --vv)

2017/10/16 08:52:54 [verbose] Kong: 0.11.0
2017/10/16 08:52:54 [debug] ngx_lua: 10008
2017/10/16 08:52:54 [debug] nginx: 1011002
2017/10/16 08:52:54 [debug] Lua: LuaJIT 2.1.0-beta2
2017/10/16 08:52:54 [verbose] no config file found at /etc/kong/kong.conf
2017/10/16 08:52:54 [verbose] no config file found at /etc/kong.conf
2017/10/16 08:52:54 [verbose] no config file, skipping loading
2017/10/16 08:52:54 [debug] KONG_PG_USER ENV found with "postgres"
2017/10/16 08:52:54 [debug] KONG_PG_HOST ENV found with "wise-indri-postgresql.kong.svc.cluster.local"
2017/10/16 08:52:54 [debug] KONG_DATABASE ENV found with "postgres"
2017/10/16 08:52:54 [debug] KONG_PG_DATABASE ENV found with "postgres"
2017/10/16 08:52:54 [debug] KONG_NGINX_DAEMON ENV found with "off"
2017/10/16 08:52:54 [debug] KONG_LOG_LEVEL ENV found with "debug"
2017/10/16 08:52:54 [debug] KONG_PG_PASSWORD ENV found with "******"
2017/10/16 08:52:54 [debug] admin_access_log = "logs/admin_access.log"
2017/10/16 08:52:54 [debug] admin_error_log = "logs/error.log"
2017/10/16 08:52:54 [debug] admin_http2 = false
2017/10/16 08:52:54 [debug] admin_listen = "0.0.0.0:8001"
2017/10/16 08:52:54 [debug] admin_listen_ssl = "0.0.0.0:8444"
2017/10/16 08:52:54 [debug] admin_ssl = true
2017/10/16 08:52:54 [debug] anonymous_reports = true
2017/10/16 08:52:54 [debug] cassandra_consistency = "ONE"
2017/10/16 08:52:54 [debug] cassandra_contact_points = {"127.0.0.1"}
2017/10/16 08:52:54 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2017/10/16 08:52:54 [debug] cassandra_keyspace = "kong"
2017/10/16 08:52:54 [debug] cassandra_lb_policy = "RoundRobin"
2017/10/16 08:52:54 [debug] cassandra_port = 9042
2017/10/16 08:52:54 [debug] cassandra_repl_factor = 1
2017/10/16 08:52:54 [debug] cassandra_repl_strategy = "SimpleStrategy"
2017/10/16 08:52:54 [debug] cassandra_schema_consensus_timeout = 10000
2017/10/16 08:52:54 [debug] cassandra_ssl = false
2017/10/16 08:52:54 [debug] cassandra_ssl_verify = false
2017/10/16 08:52:54 [debug] cassandra_timeout = 5000
2017/10/16 08:52:54 [debug] cassandra_username = "kong"
2017/10/16 08:52:54 [debug] client_body_buffer_size = "8k"
2017/10/16 08:52:54 [debug] client_max_body_size = "0"
2017/10/16 08:52:54 [debug] client_ssl = false
2017/10/16 08:52:54 [debug] custom_plugins = {}
2017/10/16 08:52:54 [debug] database = "postgres"
2017/10/16 08:52:54 [debug] db_cache_ttl = 3600
2017/10/16 08:52:54 [debug] db_update_frequency = 5
2017/10/16 08:52:54 [debug] db_update_propagation = 0
2017/10/16 08:52:54 [debug] dns_error_ttl = 1
2017/10/16 08:52:54 [debug] dns_hostsfile = "/etc/hosts"
2017/10/16 08:52:54 [debug] dns_no_sync = false
2017/10/16 08:52:54 [debug] dns_not_found_ttl = 30
2017/10/16 08:52:54 [debug] dns_order = {"LAST","SRV","A","CNAME"}
2017/10/16 08:52:54 [debug] dns_resolver = {}
2017/10/16 08:52:54 [debug] dns_stale_ttl = 4
2017/10/16 08:52:54 [debug] error_default_type = "text/plain"
2017/10/16 08:52:54 [debug] http2 = false
2017/10/16 08:52:54 [debug] latency_tokens = true
2017/10/16 08:52:54 [debug] log_level = "debug"
2017/10/16 08:52:54 [debug] lua_code_cache = "on"
2017/10/16 08:52:54 [debug] lua_package_cpath = ""
2017/10/16 08:52:54 [debug] lua_package_path = "?/init.lua;./kong/?.lua"
2017/10/16 08:52:54 [debug] lua_socket_pool_size = 30
2017/10/16 08:52:54 [debug] lua_ssl_verify_depth = 1
2017/10/16 08:52:54 [debug] mem_cache_size = "128m"
2017/10/16 08:52:54 [debug] nginx_daemon = "off"
2017/10/16 08:52:54 [debug] nginx_optimizations = true
2017/10/16 08:52:54 [debug] nginx_user = "nobody nobody"
2017/10/16 08:52:54 [debug] nginx_worker_processes = "auto"
2017/10/16 08:52:54 [debug] pg_database = "postgres"
2017/10/16 08:52:54 [debug] pg_host = "wise-indri-postgresql.kong.svc.cluster.local"
2017/10/16 08:52:54 [debug] pg_password = "******"
2017/10/16 08:52:54 [debug] pg_port = 5432
2017/10/16 08:52:54 [debug] pg_ssl = false
2017/10/16 08:52:54 [debug] pg_ssl_verify = false
2017/10/16 08:52:54 [debug] pg_user = "postgres"
2017/10/16 08:52:54 [debug] prefix = "/usr/local/kong/"

• Kong error logs (<KONG_PREFIX>/logs/error.log)

2017/10/16 12:45:58 [debug] 50#0: *28611 [lua] base_plugin.lua:20: rewrite(): executing plugin "loggly": rewrite
2017/10/16 12:45:58 [debug] 50#0: *28611 [lua] base_plugin.lua:24: access(): executing plugin "loggly": access
2017/10/16 12:45:58 [debug] 50#0: *28611 [lua] balancer.lua:285: queryDns(): [ringbalancer] querying dns for _nginx._tcp.service-user-service-user-nginx.default.svc.cluster.local
2017/10/16 12:45:58 [debug] 50#0: *28611 [lua] balancer.lua:394: queryDns(): [ringbalancer] unchanged dns record entry for _nginx._tcp.service-user-service-user-nginx.default.svc.cluster.local: service-user-service-user-nginx.default.svc.cluster.local:8080
2017/10/16 12:45:58 [debug] 50#0: *28611 [lua] balancer.lua:429: queryDns(): [ringbalancer] querying dns and updating for _nginx._tcp.service-user-service-user-nginx.default.svc.cluster.local completed
2017/10/16 12:45:58 [error] 50#0: *28611 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/core/handler.lua:407: attempt to concatenate local 'upstream_host' (a nil value)
stack traceback:
coroutine 0:
	/usr/local/share/lua/5.1/kong/core/handler.lua: in function 'after'
	/usr/local/share/lua/5.1/kong.lua:353: in function 'access'
	access_by_lua(nginx-kong.conf:88):2: in function <access_by_lua(nginx-kong.conf:88):1>, client: 127.0.0.1, server: kong, request: "GET /user HTTP/1.1", host: "localhost:8000"

• Kong configuration (registered APIs/Plugins & configuration file)
• Operating System

Docker image in Kubernetes environment (GCE)

[smalot]

This issue seems related to the preserve_host property set to the API object:

      do
        -- set the upstream host header if not `preserve_host`
        local upstream_host = var.upstream_host

        if not upstream_host or upstream_host == "" then
          local addr = ctx.balancer_address
          upstream_host = addr.hostname

          local upstream_scheme = var.upstream_scheme
          if upstream_scheme == "http"  and addr.port ~= 80 or
             upstream_scheme == "https" and addr.port ~= 443
          then
            upstream_host = upstream_host .. ":" .. addr.port
          end

          var.upstream_host = upstream_host
        end
      end

[Tieske]

This was already fixed Kong/lua-resty-dns-client#19

to be released

[smalot]

Is there a release date ?

[Tieske]

within the next 2 weeks probably

[coxon]

@Tieske in latest version(0.11.1), i use SRV records in upstream target with port 8080, like 'a-server.a-user.svc.cluster.local:8080',and i get
2017/11/20 19:11:47 [error] 38#0: *5820 connect() failed (113: No route to host) while connecting to upstream, client: 10.1.1.100, server: kong, request: "POST
uri HTTP/1.1", upstream: "http://172.30.247.246:0/servername/uri", host: "kong.domain.com"
the PORT is set to 0 from 8080.

[Tieske]

@coxon please do not respond to closed issues.

How is Kong configured? what is the upstream_url (or are you using an upstream entity as a loadbalancer?), and what does the name you have in there resolve to?

Note worthy: Kong trusts the nameserver, so any port configured in your upstream_url will be overridden by whatever the dns server (in case of an SRV record) or the loadbalancer come up with.