Too many "http: TLS handshake error" log

[jthann]

Describe the bug
When I deploy easegress as a single pod in aliyun k8s cluster and use LoadBalancer service，the pod print hugely too many TLS handshake error log in pod stdout console every one second or less than one second
Give one real log example as follow:

2022/01/26 02:05:00 http: TLS handshake error from 100.97.231.2:49030: read tcp 172.26.5.88:443->100.97.231.2:49030: read: connection reset by peer

The easegress HTTPServer object yaml config as follow:(leave out tls cert and key for security)

kind: HTTPServer
name: https
port: 443
keepAlive: true
https: true
globalFilter: globalFilter
certBase64: 
keyBase64: 
rules:
  - paths:
      - pathPrefix: /api
        backend: api-svc

The k8s service yaml config as follow:

apiVersion: v1
kind: Service
metadata:
  name: easegress
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: easegress
  sessionAffinity: None
  type: LoadBalancer

[localvar]

hi @jthann , is the easegress configuration work correctly outside of k8s?

[jthann]

hi @jthann , is the easegress configuration work correctly outside of k8s?

It's a little bit harder to test this outside of k8s，I guess it is caused by Aliyun LoadBalancer health check or something like this，because other gateway like apisix(nginx) don't have this type problem

[jthann]

I just found a similar issue from another project [https://github.com/cockroachdb/cockroach/issues/32102]
and golang net/http issue [https://github.com/golang/go/issues/26918]
It's not Easegress problem @localvar

[samutamm]

I also think that this is probably issue on Golang net/http side. We could still filter the logs on Easegress side, by providing a custom error logger for http.Server like proposed here. It could be injected in HTTPServer#runtime .
I haven't succeed in reproducing TLS handshake errors. Do you want to try filtering logs using such a custom filter @jthann ?

[jthann]

To reproduce this error， you should use aliyun or aws loadbalancer， it can be discarded like this way

httpsSrv := &http.Server{
		Addr:        fmt.Sprintf(":%d", 443),
		Handler:     &mux{},
		IdleTimeout: 60 * time.Second,
		ErrorLog:    log.New(io.Discard, "", log.LstdFlags),
}