Unexpected error from Go stdlib net/http package during roundtrip: unexpected EOF

[gdurandvadas]

Describe the bug
There are some cases where Skipper reports a 500 error after reporting this error:

Unexpected error from Go stdlib net/http package during roundtrip: unexpected EOF

As I was hoping from this bug report, the backend connection error should be reported as 499
#863

refs:

• Skipper produces 500 "Internal Server Error" HTTP status code on EOF #269
• Observability: detect when EOF is logged because of invalid Content-Length in the backend response #1134

To Reproduce
Skipper config (Used KOPS example):

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: skipper-ingress
  namespace: kube-system
  labels:
    component: ingress
spec:
  selector:
    matchLabels:
      component: ingress
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/scraped: 'true'
        prometheus.io/port: '9911'
      name: skipper-ingress
      labels:
        component: ingress
        application: skipper
    spec:
      hostNetwork: true
      serviceAccountName: ingress-controller
      containers:
      - name: skipper-ingress
        image: registry.opensource.zalan.do/pathfinder/skipper:v0.10.273
        ports:
        - name: ingress-port
          containerPort: 9999
          hostPort: 9999
        - name: metrics-port
          containerPort: 9911
        args:
          - "skipper"
          - "-kubernetes"
          - "-kubernetes-in-cluster"
          - "-address=:9999"
          - "-application-log-level=WARN"
          - "-wait-first-route-load"
          - "-proxy-preserve-host"
          - "-serve-host-metrics"
          - "-enable-ratelimits"
          - "-experimental-upgrade"
          - "-metrics-exp-decay-sample"
          - "-reverse-source-predicate"
          - "-lb-healthcheck-interval=3s"
          - "-metrics-flavour=codahale,prometheus"
          - "-enable-connection-metrics"
          - "-max-audit-body=0"
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
        readinessProbe:
          httpGet:
            path: /kube-system/healthz
            port: 9999
          initialDelaySeconds: 5
          timeoutSeconds: 5

Ingress definition (One of the ingresses but the issues happens in all of the ingresses configured in the cluster)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    zalando.org/skipper-ingress-redirect: "true"
    zalando.org/skipper-routes: |
      Path("/robots.txt") ->
      setResponseHeader("Content-Type", "text/plain") ->
      inlineContent("User-agent: *
      Allow: /graphql
      Allow: /graphql/*
      Disallow: /") ->
      status(200) -> <shunt>
  creationTimestamp: 2019-05-28T10:55:11Z
  generation: 1
  name: api
  namespace: frontend
  resourceVersion: "65708679"
  selfLink: /apis/extensions/v1beta1/namespaces/frontend/ingresses/api
  uid: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - backend:
          serviceName: website-api
          servicePort: api

Curl example of the request made by the frontend to the API

curl 'https://api.example.com/graphql/public/batch' -H 'accept: */*' -H 'Referer: https://www.example.com/' -H 'Origin: https://www.example.com' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36' -H 'Accept-Language: en' -H 'Authorization: ' -H 'content-type: application/json' --data-binary $'[{"example_json_data"}]' --compressed

Expected behavior
In any case that Skipper have a problem connecting with its backend, it should be reported as 499 instead of 500

Observed behavior

kube-system/skipper-ingress-z2plm[skipper-ingress]: [APP]time="2019-08-22T08:46:10Z" level=error msg="Unexpected error from Go stdlib net/http package during roundtrip: unexpected EOF"
kube-system/skipper-ingress-z2plm[skipper-ingress]: [APP]time="2019-08-22T08:46:10Z" level=error msg="error while proxying, route kube_frontend__api__api_example_com____website_api with backend , status code 500: dialing failed false: unexpected EOF"

Rate of the error
The rate of the error is super low, but it can be related with how the backend services behave, so if they are sturdy, we won't see much this error

From 170912 requests it happened only 80 times.

[szuecs]

Thanks for the detailed report!

499 status code should as far as I understand (nginx like), only be used if the client closed the connection (client->skipper), which seems not to be the case or do I miss understand the issue?

[szuecs]

I also had an internal case with 500 EOFs for a low traffic service and I tested to have a skipper proxy sidecar for that service, and all these errors are gone since >4 weeks.
I guess it's the backend server handler which somehow creates these problems.