OpenUnison Server error when forward-auth0 requests access token

[plcarterco]

Hello,

Background
we've been using Openunison successfully in regards to k8s dashbard, grafana, argocd, etc.
we also successfully used forward-auth0 (not to be confused with traefik-forward-auth) for authentication against Auth0 as idp
we're now trying to have forward-auth0 utilize Openunison (via Application CR) as idp within the cluster.

Issue
Authentication is successful. But the forward-auth0 access token request to openunison fails with OpenUnison server error.

We were wondering if you all had encountered similar before or had any thoughts or advice. (We couldn't determine the mechanism to increase Opeunison logging level.)

OpenUnison log entries:

[2024-01-09 18:07:48,570][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/completefed - uid=patrickx-64-xneosavant.ai,ou=shadow,o=Tremolo - NONE [10.244.0.26] - [f80611c3db9a5fff7039260ae062fa712855a0cde]
[2024-01-09 18:07:48,627][XNIO-1 task-5] INFO AccessLog - [Error] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/token - cn=none - NONE [10.244.0.26] - [f8f6134e320a1bdfd6713491ccf6a63827721dbb3]

forward-auth0 log entires:

2024-01-09 18:07:47.304 INFO 1 --- [nio-8080-exec-7] d.f.infrastructure.auth0.Auth0Service : AuthorizationCodeExchange: code=LZK5rqNIAAD/xSkrmfvYzGAabMDcV2sSHrQBQ/MacxgYzb/vk3aSykqqoH6f2vX07wl86dZCDSsn+/d+NFNQpS2sfy00zVX/8/TPCQ3leyczqgI0LmiafzylYEZmO7/2B2wOpurQ4h6bVZKOitrwM+L0zaTiFmjFrkXtZTXyZEbGIbPXeASGYoOq5Acc5nCXzhLC7pgVlmYfsouKCdTD+hSvpmnGaiArvNAIo+K/uKR2S/pNT3rIFArE9KMBwvlR1CDPD4XGUyeNZmTsji8ukmLIF+AuWc/0QtOO3M1hpas9PAfMKuUX2ns+E4eyKkbQ6rUT9SVV8BwfOE7V3olA5d8ARp4BKd15HakkghTR2cPuEfRL88K+ENGyxnzHTTgxsUWPtOH4dSvPkCjbJwQiVzYa2T4RJ1At00kKIxIQqXjoWuMnG2kUlO9pm5AkjiN2Ct1Df6T+5YKHFTvJ7fvNMm1c2rKgJ1/nOfjUccUegrXGixK7qrEHhLexjIbOPs6UOkTB1Ze0HiZHOUYmM7l1juWGyl09zoEkX1RnUW6kaGM1k/2XI+ryM9Nels26395bNKgzCSHa7RE4xBQW3MmQ2d3zcgCGzCX08pwQy05Wnka1pZp0AFsvEqJGTGfhuuamx2HOy5+QEYxsZIXrPTs+8nfS0ccN9TjkA3qIVog299FdQu1ZBptPd7P1+oAu44uz8ZiaHt5yYk/TnCXva7VvXu7MvNW9klnHqdPNnuc91GhOluSWtAhfidnAkOpVydI0KgjC99NBONnNDwpVfxV3f1vvbK2r7KzhvzOf/vwH
2024-01-09 18:07:48.030 ERROR 1 --- [nio-8080-exec-7] d.f.i.e.ServerFaultExceptionMapper : ServerFaultExceptionMapper.toResponse: error=JSONObject["error_description"] not found.

Openunison Application CR configuration

---
apiVersion: openunison.tremolo.io/v2
kind: Application
metadata:
  labels:
    app.kubernetes.io/component: openunison-applications
    app.kubernetes.io/instance: openunison-orchestra-login-portal
    app.kubernetes.io/name: openunison
    app.kubernetes.io/part-of: openunison
  name: forward-auth
  namespace: openunison
spec:
  azTimeoutMillis: 3000
  cookieConfig:
    cookiesEnabled: true
    domain: '#[OU_HOST]'
    httpOnly: true
    keyAlias: session-unison
    logoutURI: /logout
    scope: -1
    secure: true
    sessionCookieName: tremolosession
    timeout: 900
  isApp: false
  urls:
  - azRules:
    # - constraint: (groups=k8s-admins)
    - constraint: "(groups=Platform:Admin)"
      scope: filter
    - constraint: "(groups=Platform:Editor)"
      scope: filter
    - constraint: "(groups=Platform:Viewer)"
      scope: filter
    filterChain: []
    hosts:
    - '#[OU_HOST]'
    results:
      auFail: default-login-failure
      azFail: default-login-failure
    uri: /auth/idp/forward-auth
    # signedUserInfo: true
    idp:
      className: com.tremolosecurity.idp.providers.OpenIDConnectIdP
      mappings:
        map:
        - sourceType: user
          targetAttributeName: sub
          targetAttributeSource: sub
        - sourceType: composite
          targetAttributeName: name
          targetAttributeSource: ${givename} ${sn}
        - sourceType: user
          targetAttributeName: preferred_username
          targetAttributeSource: uid
        - sourceType: user
          targetAttributeName: email
          targetAttributeSource: email
        - sourceType: user
          targetAttributeName: groups
          targetAttributeSource: groups
        strict: true
      params:
        jwtSigningKey: unison-saml2-rp-sig
        k8sNameSpace: 'openunison'
        k8sTarget: k8s
        sessionStoreClassName: com.tremolosecurity.oidc.k8s.K8sSessionStore
      trusts:
      - name: 'forward-auth'
        params:
          accessTokenSkewMillis: "120000"
          accessTokenTimeToLive: '60000'
          authChainName: login-service
          clientID: <REDACTED>
          codeLastMileKeyName: lastmile-oidc
          codeTokenSkewMilis: '60000'
          publicEndpoint: "false"
          redirectURI: 
          - https://JUMPSTART.PATCHED-DOMAIN.COM/signin
          - https://postgres-ui.JUMPSTART.PATCHED-DOMAIN.COM/signin
          - https://kubecost-cost-analyzer.JUMPSTART.PATCHED-DOMAIN.COM/signin
          - https://prometheus.JUMPSTART.PATCHED-DOMAIN.COM/signin
          - https://alertmanager.JUMPSTART.PATCHED-DOMAIN.COM/signin
          - https://kube-state.JUMPSTART.PATCHED-DOMAIN.COM/signin
          - https://kubeclarity.JUMPSTART.PATCHED-DOMAIN.COM/signin
          - https://chaos-mesh.JUMPSTART.PATCHED-DOMAIN.COM/signin
        secretParams:
        - name: clientSecret
          secretName: orchestra-secrets-source
          secretKey: OU_FORWARD_AUTH_SECRET
---

[mlbiam]

'[2024-01-09 18:07:48,570][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - forward-auth - https://k8sou.athens-aks.patb68.redacted.ai/auth/idp/forward-auth/completefed - uid=patrickx-64-xneosavant.ai,ou=shadow,o=Tremolo - NONE [10.244.0.26] - [f80611c3db9a5fff7039260ae062fa712855a0cde]'
'[2024-01-09 18:07:48,627][XNIO-1 task-5] INFO AccessLog - [Error] - forward-auth - https://k8sou.athens-aks.patb68.redacted.ai/auth/idp/forward-auth/token - cn=none - NONE [10.244.0.26] - [f8f6134e320a1bdfd6713491ccf6a63827721dbb3]'

is there a stack trace after this? It would start with "Sendnig JSON Error"?

Also, what verion of the helm charts are you on?

[plcarterco]

hello,

we're using orchestra-login-portal-argocd via the ArgoCD Application Object - version: 2.3.38
(saw the recent versions to 2.3.41 but it breaks in other areas regarding groups and how we are using them I believe....We haven't had time to investigate and didn't want to introduce new variables into the mix. )

there's no stack trace, but here's fuller log context before and after:

[2024-01-09 18:07:47,153][XNIO-1 task-5] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:07:47,211][XNIO-1 task-5] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:07:47,211][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/completefed - uid=patrickx-64-xneosavant.ai,ou=shadow,o=Tremolo - NONE [10.244.0.26] - [f80611c3db9a5fff7039260ae062fa712855a0cde]
[2024-01-09 18:07:48,006][XNIO-1 task-5] INFO AccessLog - [Error] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/token - cn=none - NONE [10.244.0.26] - [f8f6134e320a1bdfd6713491ccf6a63827721dbb3]
[2024-01-09 18:07:48,494][XNIO-1 task-5] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:07:48,570][XNIO-1 task-5] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:07:48,570][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/completefed - uid=patrickx-64-xneosavant.ai,ou=shadow,o=Tremolo - NONE [10.244.0.26] - [f80611c3db9a5fff7039260ae062fa712855a0cde]
[2024-01-09 18:07:48,627][XNIO-1 task-5] INFO AccessLog - [Error] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/token - cn=none - NONE [10.244.0.26] - [f8f6134e320a1bdfd6713491ccf6a63827721dbb3]
[2024-01-09 18:07:51,196][Thread-8] INFO K8sWatcher - Resource 36753 already processed, skipping
[2024-01-09 18:07:52,584][Thread-13] INFO K8sWatcher - Resource 37101 already processed, skipping
[2024-01-09 18:07:54,161][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - k8sidp - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - uid=Anonymous,o=Tremolo - NONE [127.0.0.1] - [f6dfd36ec006b1545c4767118e3f52195449e143a]
[2024-01-09 18:07:54,162][XNIO-1 task-6] INFO AccessLog - [AzSuccess] - k8sidp - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - uid=Anonymous,o=Tremolo - NONE [127.0.0.1] - [f7b242b275c8bafed4e91e16f594eef2134f4eb0c]
[2024-01-09 18:07:54,386][Thread-9] INFO K8sWatcher - Resource 34772 already processed, skipping
[2024-01-09 18:07:57,362][Thread-12] INFO K8sWatcher - Resource 37550 already processed, skipping
[2024-01-09 18:07:58,096][XNIO-1 task-6] INFO AccessLog - [AzSuccess] - scale-session-check - https://k8sou.athens-aks.patb68.REDACTED.ai/scale/sessioncheck - uid=Anonymous,o=Tremolo - NONE [10.244.0.26] - [f015c2e843b69a5404c633b2011cbccd98aa68320]
[2024-01-09 18:07:58,106][XNIO-1 task-6] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:08:04,156][XNIO-1 task-6] INFO AccessLog - [AzSuccess] - k8sidp - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - uid=Anonymous,o=Tremolo - NONE [127.0.0.1] - [fc9e5a9078ac66f09a8c4d5c8d6e260f931de5bc3]

here's the OU config:

network:
  openunison_host: "k8sou.${OU_BASEDOMAIN}"
  dashboard_host: "k8sdb.${OU_BASEDOMAIN}"
  api_server_host: "k8sapi.${OU_BASEDOMAIN}"
  session_inactivity_timeout_seconds: 900
  k8s_url: "https://kubernetes.default.svc"
  force_redirect_to_tls: false
  createIngressCertificate: false
  ingress_certificate: ou-tls-certificate
  ingress_type: traefik
  ingress_annotations:
    kubernetes.io/ingress.class: traefik0
  traefik:
    secure: true
    entrypoints:
      plaintext: web
      tls: websecure

cert_template:
  ou: "Kubernetes"
  o: "NeoEnvironments"
  l: "${OU_BASEDOMAIN}"
  st: "Colorado"
  c: "USA"

myvd_config_path: "WEB-INF/myvd.conf"
k8s_cluster_name: "${OU_CLUSTER_NAME}"
enable_impersonation: true

impersonation:
  use_jetstack: true
  jetstack_oidc_proxy_image: docker.io/tremolosecurity/kube-oidc-proxy:latest
  explicit_certificate_trust: true

dashboard:
  enabled: true
  namespace: "kubernetes-dashboard"
  cert_name: "kubernetes-dashboard-certs"
  label: "k8s-app=kubernetes-dashboard"
  service_name: kubernetes-dashboard
  require_session: true

certs:
  use_k8s_cm: false

trusted_certs: []

monitoring:
  prometheus_service_account: system:serviceaccount:monitoring:prometheus-k8s

oidc:
  client_id_is_secret: true
  issuer: "${OU_ISSUER}"
  user_in_idtoken: true
  domain: ""
  scopes: openid email profile groups
  claims:
    sub: email
    email: email
    given_name: given_name
    family_name: family_name
    display_name: name
    groups: ${OU_CLAIM_NAMESPACE}/groups

network_policies:
  enabled: false
  ingress:
    enabled: true
    labels:
      app.kubernetes.io/name: traefik
  monitoring:
    enabled: true
    labels:
      app.kubernetes.io/name: monitoring
  apiserver:
    enabled: true
    labels:
      app.kubernetes.io/name: kube-system

services:
  enable_tokenrequest: false
  token_request_audience: api
  token_request_expiration_seconds: 600
  node_selectors: []
  
openunison:
  replicas: 1
  non_secret_data:
    K8S_DB_SSO: oidc
    PROMETHEUS_SERVICE_ACCOUNT: system:serviceaccount:monitoring:prometheus-k8s
    SHOW_PORTAL_ORGS: "false"
  secrets: []
  enable_provisioning: false
  use_standard_jit_workflow: true
  az_groups: []
  html:
    image: docker.io/tremolosecurity/openunison-k8s-html
    logosConfigMap: custom-logos

operator:
  validators: []
  mutators: []
  imagePullPolicy: Always
  resources:
    requests:
      memory: 200Mi
      cpu: 0.3
    limits:
      memory: 200Mi
      cpu: 1.0

[mlbiam]

nothing stands out. it's odd that there's no error in the logs. i've never used auth0 forward, so i'll try to get it up and running and see if i can reproduce

[plcarterco]

If you can email me directly, we can easily spin up a full temporary environment with everything in place and provide you access. From: Marc Boorshtein ***@***.***> Sent: Tuesday, January 9, 2024 3:40 PM To: OpenUnison/openunison-k8s ***@***.***> Cc: plcarterco ***@***.***>; Author ***@***.***> Subject: Re: [OpenUnison/openunison-k8s] OpenUnison Server error when forward-auth0 requests access token (Issue #97) nothing stands out. it's odd that there's no error in the logs. i've never used auth0 forward, so i'll try to get it up and running and see if i can reproduce — Reply to this email directly, view it on GitHub<#97 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEWCXAUJ3YURTSL5WKSUKALYNXBMJAVCNFSM6AAAAABBTSXDKCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBTHEYTENZQGE>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[mlbiam]

Thank you for the offer, but we reserve direct interaction for customers with support contracts, as we don't produce any closed source or "enterprise" editions that require licensing. If you're interested in a support contract, please reach out at https://www.tremolosecurity.com/contact/contact-us.

That said, I put together a debug build of 1.0.38 for you (the same version as is running per your helm chart) - docker.io/tremolosecurity/openunison-k8s-debug:1.0.38

This should have the build number 1.0.38-2023120501-DEBUG in the logs. I added an extra catch for errors, for some reason the error is being caught but not logged so hopefully this will find it. You can set the top level image option in your values.yaml and redeploy.

[plcarterco]

Completely understand, and thank you for the debug build. We'll try it out and let you know.