CLI returns "certificate signed by unknown authority"

[Yaytay]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I am unable to establish a secure connection to ArgoCD:

$ argocd login argocd-server
WARNING: server certificate had error: x509: certificate signed by unknown authority. Proceed insecurely (y/n)? n

To Reproduce
On a clean installation of Docker Desktop:

1. Install nginx-ingress

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

2. Install argocd:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

3. Create the ingress:
   Save this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: argocd-server
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port: 
               number: 443

as argocd-ingress.yml and then apply:

kubectl apply -n argocd -f argocd-ingress.yml

4. Ensure that argocd-server is in localhost:

grep argocd-server /etc/hosts || echo '127.0.0.1     argocd-server' | sudo tee -a /etc/hosts

5. Test:

argocd login argocd-server
WARNING: server certificate had error: x509: certificate signed by unknown authority. Proceed insecurely (y/n)? n

Checking with openssl indicates that the cert is coming from ArgoCD:

$ echo quit | openssl s_client -showcerts -servername argocd-server -connect argocd-server:443 | head
depth=0 O = Argo CD
verify error:num=18:self signed certificate
verify return:1
depth=0 O = Argo CD
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:O = Argo CD
   i:O = Argo CD
-----BEGIN CERTIFICATE-----
MIIDYjCCAkqgAwIBAgIQXxa6rX0ZcIgeHI4opRxuyzANBgkqhkiG9w0BAQsFADAS
MRAwDgYDVQQKEwdBcmdvIENEMB4XDTIyMDUxNzEyMDc1M1oXDTIzMDUxNzEyMDc1
M1owEjEQMA4GA1UEChMHQXJnbyBDRDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAOhItSIG+Xa60BZQ+NupQ3TV7W1HPJbi3uH4Xk8hs3KBlIlGOlLgiPBt
DONE

I have also tried using the argocd-server-tls secret to validate that the certificate being used is not the nginx-ingress one, and that produces errors that include the valid names of the cert, so the cert is definitely coming from the argocd pod - but the argocd CLI doesn't accept it.

I cannot find any documentation about what the cert is supposed to be.
At the moment the FAQ entry for this contains no more information than the CLI output.
It would be really helpful if that entry contained details on what the cert should be.

Setting up a port forwarder:

kubectl port-forward svc/argocd-server -n argocd 8080:443

also gives the same error.

Expected behavior
Some sort of request for credentials, anything other than a cert error.

Version

$ argocd version
argocd: v2.3.3+07ac038.dirty
  BuildDate: 2022-03-30T05:17:09Z
  GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
  GitTreeState: dirty
  GoVersion: go1.18
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified

Running on Windows 11 using WSL2.

[Yaytay]

I got this to work using a custom cert (argocd-server-tls) and adding the root CA to the OS cert store.
Am I right in thinking that the "Argo CD" cert created by default has no special status with the CLI and would need to be manually added to the OS cert store (or requests would need to be --insecure)?
If so, that should be reflected in the Getting Started docs.

[Yaytay]

Please can someone confirm that I'm right about the default cert requiring insecure access?
If that gets confirmed I'll be happy to provide minor PRs for the FAQ and Getting Started pages.

[GeorgiDimv]

I got this to work using a custom cert (argocd-server-tls) and adding the root CA to the OS cert store. Am I right in thinking that the "Argo CD" cert created by default has no special status with the CLI and would need to be manually added to the OS cert store (or requests would need to be --insecure)? If so, that should be reflected in the Getting Started docs.

Hello could you please provide information how you created and consumed the created argocd-server-tls cert.

[crenshaw-dev]

Am I right in thinking that the "Argo CD" cert created by default has no special status with the CLI and would need to be manually added to the OS cert store (or requests would need to be --insecure)?

If my understanding is accurate, that is correct - the CLI makes no assumptions about the server's TLS cert. From the docs:

If no tls.crt and tls.key keys are found in neither of the two mentioned secrets, Argo CD will generate a self-signed certificate and persist it in the argocd-secret secret.

If the cert is self-signed or otherwise not verifiable by the CLI host, the CLI will reject it. So if it's self-signed, you'll need to add it to the host's root CAs.

[GeorgiDimv]

Am I right in thinking that the "Argo CD" cert created by default has no special status with the CLI and would need to be manually added to the OS cert store (or requests would need to be --insecure)?

If my understanding is accurate, that is correct - the CLI makes no assumptions about the server's TLS cert. From the docs:

If no tls.crt and tls.key keys are found in neither of the two mentioned secrets, Argo CD will generate a self-signed certificate and persist it in the argocd-secret secret.

If the cert is self-signed or otherwise not verifiable by the CLI host, the CLI will reject it. So if it's self-signed, you'll need to add it to the host's root CAs.

I have a valid certificate issued by Letsencrypt. When I log in via the UI I have a secure connection but when I use the CLI it doesn’t recognize the root CA. What to do in this situation because argocd login --insecure is not an option?

[YaytayAtWork]

@GeorgiDimv
My cert is created using:

apiVersion: v1
data:
  tls.crt: L...==
  tls.key: L...==
kind: Secret
metadata:
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"
  name: argocd-server-tls
  namespace: argocd
type: kubernetes.io/tls

Note that I create it using SealedSecrets, so mine is actually a bit different, but that should work (with the Base64 cert and key).

There is no additional configuration, argocd picks it up and uses it.
I create the secret after creating argocd, but before creating the ingress (I don't think the order matters as long as you give it time to react to the new manifests).

Mine is a test instance, everything is running on localhost using *.localtest.me hostnames and the cert/key come from our company PKI which our client OS is already setup to accept.

My ingress for argocd does not use that TLS (because it uses passthrough):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: argocd.localtest.me
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port: 
               number: 443

And finally my nginx-ingress is installed via helm:

  helm upgrade 'nginx-ingress' ingress-nginx/ingress-nginx \
     --namespace nginx-ingress \
     --install \
     --create-namespace \
     --set "controller.image.tag=2.2.2-alpine-ot" \
     --set "controller.extraArgs.enable-ssl-passthrough=true" \
     --set "controller.enableCustomResources=true" \
     --set "controller.config.metrics.enabled=true" \
     --set-string "controller.podAnnotations.prometheus\.io/scrape"="true" \
     --set-string "controller.podAnnotations.prometheus\.io/port"="..." \
     --set "controller.config.enable-opentracing=true" \
     --set "controller.config.zipkin-collector-host=zipkin.monitoring"

(some of that is irrelevant to you, my ingress is used for all our test setup)

[GeorgiDimv]

@GeorgiDimv My cert is created using:

apiVersion: v1
data:
  tls.crt: L...==
  tls.key: L...==
kind: Secret
metadata:
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"
  name: argocd-server-tls
  namespace: argocd
type: kubernetes.io/tls

Note that I create it using SealedSecrets, so mine is actually a bit different, but that should work (with the Base64 cert and key).

There is no additional configuration, argocd picks it up and uses it. I create the secret after creating argocd, but before creating the ingress (I don't think the order matters as long as you give it time to react to the new manifests).

Mine is a test instance, everything is running on localhost using *.localtest.me hostnames and the cert/key come from our company PKI which our client OS is already setup to accept.

Thank you a lot, I got it working!

[crenshaw-dev]

@GeorgiDimv what was the difference between the failing config and the now-working config?

[GeorgiDimv]

@GeorgiDimv what was the difference between the failing config and the now-working config?

I was using letsencrypt to create certificate and on login via the cli argocd printed
WARNING: server certificate had error: x509: certificate signed by unknown authority.
I guess the problem was with the issuer rather than the certificate itself.

[crenshaw-dev]

Possibly. But letsencrypt should be valid. Maybe there's something wrong with the root CA bundle?