query-frontend logs failed to parse host port for http log decision

[yasharne]

Thanos, Prometheus and Golang version used:

Version	0.21.1
Revision	3558f4a5adc8d4068f82528570a439d440f98b9d
Branch	HEAD
BuildUser	circleci@eb60c35e294a
BuildDate	20210604-12:24:45
GoVersion	go1.16.4

What happened:
I have deployed the query-frontend to the Kubernetes and there are a lot of logs like this in the container:

level=error ts=2021-06-09T04:29:40.506819592Z caller=http.go:48 protocol=http http.component=server msg="failed to parse host port for http log decision" err="address thanos.apps.x.y.z: missing port in address"

Here are the command-line flags for the query-frontend:

            - --query-frontend.compress-responses
            - --query-frontend.downstream-url=http://thanos-query-http.thanos.svc:10902
            - --query-frontend.log-queries-longer-than=5s
            - --query-range.split-interval=24h
            - --query-range.response-cache-max-freshness=1m
            - --query-range.max-retries-per-request=5
            - --query-range.response-cache-config-file=/etc/thanos/frontend.yml
            - --labels.max-retries-per-request=10
            - --labels.split-interval=12h
            - --cache-compression-type=snappy

What you expected to happen:
I haven't defined any address for query-frontend in the command-line arguments, if the logger is fetching the address from the user's request why it cannot determine the port address?

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

level=error ts=2021-06-09T04:29:40.506819592Z caller=http.go:48 protocol=http http.component=server msg="failed to parse host port for http log decision" err="address thanos.apps.x.y.z: missing port in address"

Anything else we need to know:

[yeya24]

I feel this issue is related to the logging middleware instead of query frontend. CC @yashrsharma44.

The error happens at https://github.com/thanos-io/thanos/blob/main/pkg/logging/http.go#L43-L51. If there is no port in the address, can we do something to skip this error?

[MrYueQ]

@yeya24 @yashrsharma44 Hi. I also encountered this error log, although it does not affect the use of the service. I want to introduce the situation on my side.
This is an example of my current architecture

This is my version information and error log

thanos --version
thanos, version 0.21.0 (branch: HEAD, revision: d19aea21b178a6c08838c6a05b834dd44ef14273)
  build user:       root@30b80273f41d
  build date:       20210603-13:49:01
  go version:       go1.16.4
  platform:         linux/amd64
# error print
{"caller":"http.go:48","err":"address thanos.demo.io: missing port in address","http.component":"server","level":"error","msg":"failed to parse host port for http log decision","protocol":"http","ts":"2021-06-11T17:55:11.136477708Z"}

thanos/pkg/logging/http.go

Lines 43 to 51 in 7c6c505

	if hostPort == "" {
	hostPort = r.URL.Host
	}
	_, port, err := net.SplitHostPort(hostPort)
	if err != nil {
	level.Error(m.logger).Log("msg", "failed to parse host port for http log decision", "err", err)
	next.ServeHTTP(w, r)
	return
	}

[bsnyder788]

I am also seeing the same issue on a few Thanos clusters I am using. 0.19.0 did not have this issue, but when I upgraded to 0.21.1 I started seeing lots of these logs.

[spaparaju]

• This problem does not happen on standalone deployment as the r.Host maps to --http-address which contains a port #.

• If app. like Query-frontend is deployed under a hostname which in turn mapped to --http-address (host + port) of query-frontend component (eg: route on a Kube. cluster), then this problem shows up as SplitHostPort() of net package expect a ':' and throw up an error.

• Apart from lots of parsing errors in the log file, this problem results in 'not applying the decider logic' for logging.

[spaparaju]

This PR fixes this issue.

[wiardvanrij]

These might be related:

#4139
#4118
#4334

Linking them for reference

[yeya24]

Should be solved in #4348. Thanks @spaparaju