ArgoCD v.2.4.0 Terminal read message err: websocket: unexpected reserved bits 0x70

[chrism417]

My argocd-cm:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
data:
  exec.enabled: 'true'

My argocd-params-cm:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-cmd-params-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cmd-params-cm
data:
  server.insecure: 'true'
  server.enable.exec: 'true'

My argocd-server cluster role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    argocd.argoproj.io/instance: argocd-itself
  name: argocd-server
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - delete
  - get
  - patch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - list
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - pods/exec
  verbs:
  - create

I've enabled websockets on my HTTPProxy and I can see the terminal for a pod, but I can't type in the terminal. I see the error in the argocd-server:

time="2022-06-13T14:22:05Z" level=error msg="read message err: websocket: unexpected reserved bits 0x70"
E0613 14:22:05.001903 1 v2.go:105] websocket: unexpected reserved bits 0x70

[mkilchhofer]

Same for me with kubectl port-forward.

With my ingress controller (project contour) I didn't have success at all yet. Still testing random things to bring websockets alive. (Already use websockets for other apps with contour, no issues so far with these apps, but Argo won't work yet).

But I have 0x60 errors:

time="2022-06-14T12:19:23Z" level=info msg="terminal session starting" application=chaoskube cluster=apps-with-clusterroles container=chaoskube namespace=base-infra podName=chaoskube-5d49569df8-ds4ql userName=marco@<REDACTED>
E0614 12:19:24.022648       1 v2.go:105] EOF
time="2022-06-14T12:19:27Z" level=error msg="read message err: websocket: unexpected reserved bits 0x60"
E0614 12:19:27.802163       1 v2.go:105] websocket: unexpected reserved bits 0x60

[chrism417]

With my ingress controller (project contour) I didn't have success at all yet. Still testing random things to bring websockets alive. (Already use websockets for other apps with contour, no issues so far with these apps, but Argo won't work yet).

I'm also using contour and have enabled websockets on the HTTPProxy. Like you said, it works everywhere else. I keep getting the error with Argo though

[mkilchhofer]

Okay I can eliminate contour as I converted the argocd-server Service to a NodePort service. Same behavior :(

[chrism417]

Thanks @mkilchhofer, so definitely an Argo issue.

[arthurk]

Same issue when opening the web terminal:

{"level":"error","msg":"read message err: websocket: unexpected reserved bits 0x60","time":"2022-06-15T04:15:27Z"}

It seems to work fine for containers that have bash installed but not for ones that are using sh (based on alpine). Maybe something wrong with the logic here?

argo-cd/server/application/terminal.go

Line 225 in dee550a

// FIXME: if the first shell fails then the first keyboard event is lost

[crenshaw-dev]

It seems to work fine for containers that have bash installed but not for ones that are using sh (based on alpine). Maybe something wrong with the logic here?

Is this the case for others on the thread? If so, I'll close as duplicate to consolidate discussion here: #9641

If not, we have two different issues.

[mkilchhofer]

Tested both nginx flavours:

I cannot open the terminal for both. 😢

[djfinnoy]

My experience is that ui exec issues caused by RBAC are almost indistinguishable from those caused by the pod not having bash installed.

@mkilchhofer are you able to run bash via kubectl exec -it <pod name> -- bash ? If so, are you able to UI exec into the pod with an admin account?

[crenshaw-dev]

Keep in mind that Argo CD doesn't use your account to exec into a Pod. It uses its own ServiceAccount. So even if you can kubectl exec in, that doesn't necessarily mean the RBAC (either the k8s RBAC restricting the Argo CD API Server or the Argo CD RBAC restricting the user) is permissive enough.

[chrism417]

I'm in the same boat as @mkilchhofer, I have the issue with alpine, non-alpine, and windows terminals. This is not an RBAC issue as the service account has the required permissions.

[m-yosefpor]

all the following 3 issues are the same:

• ArgoCD v.2.4.0 Terminal read message err: websocket: unexpected reserved bits 0x70 #9643
• Problem with Terminal and linux shell different from bash #9641
• OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown #9605

I think we can follow in one thread.