server,ui: infinite HTTP redirect when accessing UI for secure cluster behind nginx-ingress in k8s

[sergeyshaykhullin]

Describe the problem
I am getting too many redirects when trying to access secure cockroachdb cluster behind ingress-nginx using helm chart.

Please describe the issue you observed, and any steps we can take to reproduce it:

It is happening because tls.enabled == secure cluster with tls termination.
Same thing i got with argo cd and solved using --insecure flag (Insecure - just about tls termination)
So i have to use insecure cluster behind ingress (losing auth screen, users with passwords etc) or secure cluster without ingress inside k8s (And i have to manage certificates, use not 80 and 443 ports, it's painful)

To Reproduce

1. Setup basic k8s cluster
2. Setup nginx-ingress, cert-manager
3. Setup secure cockroachdb cluster
4. Create k8s Ingress for cockroachdb service
5. Try to access admin panel

Expected behavior
I can disable tls termination in web ui, but don't lose security cluster benefits

Additional data / screenshots

• I am using helm charts:
  • https://hub.helm.sh/charts/stable/cockroachdb
  • https://hub.helm.sh/charts/stable/nginx-ingress
  • https://hub.helm.sh/charts/jetstack/cert-manager

Environment:

• CockroachDB version 20.1
• Server OS: Debian 10

Jira issue: CRDB-4219

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I have CC'd a few people who may be able to assist you:

• @florence-crl (member of the technical support engineering team)

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[knz]

xref cockroachdb/cockroach#44842

[knz]

cc @dhartunian this seems to fall between the cracks of server / obs infra. What to do?

[mihneastaub]

@knz any updates on this issue?

[knz]

We've not found the direct cause of the issue yet, but you might have interest in cockroachdb/cockroach#74329.

[mihneastaub]

We've not found the direct cause of the issue yet, but you might have interest in cockroachdb/cockroach#74329.

Thanks for the information, but it looks that neither of those flags will not resolve the problem with the redirect.

[udnay]

@sergeyshaykhullin can you connect to the web ui from inside the cluster if you do portforwarding and bypass the nginx ingress?

I assume you have something like
external request (https) -> nginx ingress (tls termination) -> web ui (http) and the redirect loop is happening at the CRDB side?

[udnay]

@mihneastaub or @sergeyshaykhullin can you also provide your k8s nginx annotations?

[mihneastaub]

@udnay I'm using istio gateway for the ingress controller.

[sergeyshaykhullin]

metadata:
    annotations:
      kubernetes.io/tls-acme: "true"
      meta.helm.sh/release-name: crdb
      meta.helm.sh/release-namespace: crdb
      nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri
      nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
      nginx.ingress.kubernetes.io/service-upstream: "true"

[pseudomuto]

While starting to take a look into this, I was able to reproduce this issue using the following steps. I figured these steps could be helpful if anyone else is interested in verifying the issue or playing around with different config settings.

Start by creating a few files we'll use for getting everything set up.

Support files for testing

# hack/issuer.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
     - http01:
         ingress:
           class: nginx

# hack/values.yaml
ingress:
  enabled: true
  annotations:
    cert-manager.io/issuer: letsencrypt-staging
    kubernetes.io/ingress.class: nginx
  hosts:
    - crdb.example.com
  tls:
    - hosts: [crdb.example.com]
      secretName: crdb-example-com-tls

# hack/exec (don't forget to chmod +x this file)
#!/usr/bin/env bash
set -euo pipefail

CLUSTER="${CLUSTER:-"$(whoami)-test"}"
REGION="${REGION:-us-east1}"

main() {
  case "${1:-}" in
    create-cluster) create_cluster;;
    delete-cluster) delete_cluster;;
    apply)
      # ensure we've got the necessary binaries
      make bin/helm bin/kubectl
      install_cert_manager
      install_ingress_nginx
      install_chart;;
    *)
      echo "Unknown command '${1:-}'." 1>&2
      echo "Usage: ${0} <command>"
      echo "  commands:"
      echo "    create-cluster    create a GKE cluster for testing"
      echo "    delete-cluster    delete the GKE cluster"
      echo "    apply             (idempotently) install cert-manager, ingress-nginx, and this chart"
      exit 1;;
  esac
}

create_cluster() {
  println "Creating GKE cluster ${CLUSTER}..."
  gcloud container clusters create "${CLUSTER}" --region "${REGION}" --num-nodes 1
}

delete_cluster() {
  println "Deleting GKE cluster ${CLUSTER}..."
  gcloud container clusters delete "${CLUSTER}" --region "${REGION}"
}

install_chart() {
  println "Installing local cockroachdb chart..."
  bin/helm upgrade -f hack/values.yaml --install crdb ./cockroachdb
}

install_ingress_nginx() {
  println "Installing ingress-nginx..."
  bin/helm upgrade --install ingress-nginx ingress-nginx \
    --repo https://kubernetes.github.io/ingress-nginx \
    --namespace ingress-nginx --create-namespace

  println "Waiting for ingress controller to be available..."
 bin/kubectl wait --namespace ingress-nginx \
    --for=condition=ready pod \
    --selector=app.kubernetes.io/component=controller \
    --timeout=90s
}

install_cert_manager() {
  println "Installing cert-manager..."
  bin/helm upgrade --install cert-manager cert-manager \
    --repo https://charts.jetstack.io \
    --namespace cert-manager \
    --create-namespace \
    --set installCRDs=true

  println "Creating letencrypt issuer..."
  bin/kubectl apply -f hack/issuer.yaml
}

println() {
  echo -e "\033[36m${1}\033[0m"
}

main "$@"

You'll need a working K8s cluster. If you don't have one you can run hack/exec create-cluster to create a GKE cluster. This assumes you've got gcloud installed and configured with an account that can create/manage GKE clusters. The cluster can be deleted with hack/exec delete-cluster when you're done.

Install cert-manager, ingress-nginx, and the cockroachdb chart by running hack/exec apply. This can be re-run as needed (idempotent) which can be useful if you change the values or issuer files and want to upgrade the charts.

With everything set up, get a hold of the external IP for the ingress object:

bin/kubectl get ing crdb-cockroachdb-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

cURL the ingress and verify you get an error about maximum redirects reached:

curl -kL --resolve crdb.example.com:443:<LB_IP_FROM_ABOVE> https://crdb.example.com
> curl: (47) Maximum (50) redirects followed

[dhartunian]

I think this behavior is due to this snippet below which redirects all traffic on the HTTP port to HTTPS except for the healthcheck. @pseudomuto I'm guessing /health resolves correctly in your example.

Maybe what we should do is inspect X-Forwarded-Proto and disable redirect when it's set to https. What do you think @udnay @pseudomuto ? (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto, I assume nginx sets this)

https://github.com/cockroachdb/cockroach/blob/946b9c545127557bc8a6e1e05905a85db5095e7f/pkg/server/server_http.go#L222-L238

[nv30]

Is there any workaround for this problem?

[Djankaa]

Any progress here?

[HcgRandon]

Waiting to see a solution for this? A lot of people use ingresses...

[HcgRandon]

Heads up for others; If your ingress is nginx based you can add the nginx.ingress.kubernetes.io/backend-protocol: HTTPS annotation to the ingress definition and this will fix the issue.

Found this solution here: https://stackoverflow.com/questions/54459015/how-to-configure-ingress-to-direct-traffic-to-an-https-backend-using-https/54459898#54459898

[Djankaa]

@HcgRandon
Works perfectly! Thank you!

[ghost]

NOPE STILL DOESNT WORK

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: traefik
    meta.helm.sh/release-name: cockroachdb
    meta.helm.sh/release-namespace: whocares-prod
    traefik.ingress.kubernetes.io/service.serversscheme: https
  labels:
    app.kubernetes.io/instance: cockroachdb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cockroachdb
    helm.sh/chart: cockroachdb-7.0.3
  name: cockroachdb-ingress
  namespace: whocares-prod
spec:
  rules:
  - host: some.hostname
    http:
      paths:
      - backend:
          service:
            name: cockroachdb-public
            port:
              number: 8080
        path: /
        pathType: Prefix
status:
  loadBalancer: {}

[JonatanCruz]

@ghost
you can try add the ingress configuration in operator deploy file

apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
  # this translates to the name of the statefulset that is created
  name: cockroachdb
spec:
  dataStore:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: "20Gi"
        volumeMode: Filesystem
  resources:
    requests:
      # This is intentionally low to make it work on local kind clusters.
      cpu: 2
      memory: 4Gi
    limits:
      cpu: 4
      memory: 8Gi
  tlsEnabled: true
# You can set either a version of the db or a specific image name
# cockroachDBVersion: v21.2.8
  image:
    name: cockroachdb/cockroach:v22.1.1
  # nodes refers to the number of crdb pods that are created
  # via the statefulset
  nodes: 3
  additionalLabels:
    crdb: is-cool
  # affinity is a new API field that is behind a feature gate that is
  # disabled by default.  To enable please see the operator.yaml file.

  # The affinity field will accept any podSpec affinity rule.
  # affinity:
  #   podAntiAffinity:
  #      preferredDuringSchedulingIgnoredDuringExecution:
  #      - weight: 100
  #        podAffinityTerm:
  #          labelSelector:
  #            matchExpressions:
  #            - key: app.kubernetes.io/instance
  #              operator: In
  #              values:
  #              - cockroachdb
  #          topologyKey: kubernetes.io/hostname

  # nodeSelectors used to match against
  # nodeSelector:
  #   worker-pool-name: crdb-workers
  
  # Ingress for access the services
  ingress:
    ui:
      ingressClassName: nginx
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" <-- add here
      host: some.hostname

[rajavarapusaiprasanth-hpe]

has anyone tried using path based routing instead of host based routing? I am facing issue on /api/v2/sql and /uiconfig when using a different path instead of /

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: cockroachdb-ingress
  namespace: cockroach
spec:
  rules:
  - host: shared.host.name
    http:
      paths:
      - backend:
          service:
            name: croach-cockroachdb-public
            port:
              number: 8080
        path: /cockroach(/|$)(.*)
        pathType: ImplementationSpecific

[udnay]

@prafull01 can you triage?

[prafull01]

@himanshu-cockroach Can you please have a look

[himanshu-cockroach]

@rajavarapusaiprasanth-hpe The path based routing is not working currently because of this issue: cockroachdb/cockroach#91429. since all the other paths including UI dashboard, other than the absolute paths which are being redirected to via the code, its working through path based routing, we can close this thread.