Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[UI] Dex/OIDC Integration #167

Closed
richard-cox opened this issue Oct 6, 2022 · 35 comments
Closed

[UI] Dex/OIDC Integration #167

richard-cox opened this issue Oct 6, 2022 · 35 comments

Comments

@richard-cox
Copy link
Member

richard-cox commented Oct 6, 2022

See epinio/epinio#1571 for details

  • The UI backend needs updating to
    • Configure mocked norman api response to return OIDC auth provider config
    • Proxy single sign on requests to the correct epinio API (this may already work using the normal proxy mechanism)
  • The UI frontend needs to
    • Ensure that the correct log in page components show
    • That the users can use the standard single sign on flow
    • Ensure that epinio api requests contain the correct auth headers

Questions/TBD

  • Will the original username/password basic auth process still need to be supported Yes
@enrichman
Copy link
Member

  • Will the original username/password basic auth process still need to be supported

If we want to kee the BasicAuth and the current authentication we probably just need to add a link to the new auth, something like "Insert Username/Password, or login with SSO". So both of the flows will work.

To avoid having two different login flows we could also decide to drop the username and password login, moving to the new flow. We would just need to document how to migrate the old user into the Dex configuration.

To avoid breaking changes I would probably go for the first option, and see if we want to keep both of them, or starting to deprecate the old one.

@thehejik
Copy link
Contributor

I've built and pushed custom thehejik/epinio-ui:oidc-dev image based on the PRs above, then deployed epinio by using the helm-chart PR and patched the original image in epinio-ui deployment with the custom one. So far so good, I could see the new login page providing dex option.

image

But after pressing Log in with Dex button I was forwarded back to the epinio login page. So I tried to perform dex login over epinio cli which was succesfull. Then I tried dex login over web again but with the same result as before.

@richard-cox please did I miss any step here?

When looking into epinio-ui pod logs I found out it requires endpoints/service http://epinio-server.auth.svc.cluster.local/ which is not present in the cluster:

ERRO[Wed Nov 23 14:26:57 UTC 2022] Failed to create Dex Client: failed to create dex OIDC provider: Get "http://epinio-server.auth.svc.cluster.local/.well-known/openid-configuration": dial tcp: lookup epinio-server.auth.svc.cluster.local: no such host

request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/:///" Status:200 Latency:1.029246ms Bytes-In:0 Bytes-Out:1628
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/1ee8649.js" Status:304 Latency:1.526848ms Bytes-In:0 Bytes-Out:0
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/d9393c1d97130f7e19f5.js" Status:304 Latency:733.057µs Bytes-In:0 Bytes-Out:0
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fd4820f8a3014d153ea0.js" Status:304 Latency:218.694µs Bytes-In:0 Bytes-Out:0
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/6ad9448902b91a3f894f.js" Status:304 Latency:230.731µs Bytes-In:0 Bytes-Out:0
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:559.738µs Bytes-In:0 Bytes-Out:33572
ERRO[Wed Nov 23 14:26:56 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-11-23T14:26:56.851721857Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/users" Status:401 Latency:253.282µs Bytes-In:0 Bytes-Out:60
ERRO[Wed Nov 23 14:26:56 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-11-23T14:26:56.905808657Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"POST" Path:"/pp/v1/epinio/rancher/v3/tokens" Status:401 Latency:109.814µs Bytes-In:2 Bytes-Out:60
ERRO[Wed Nov 23 14:26:56 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-11-23T14:26:56.909382314Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/principals" Status:401 Latency:97.563µs Bytes-In:0 Bytes-Out:60
Request: [2022-11-23T14:26:56Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/2d02c6d886c12adf7cfe.js" Status:200 Latency:1.417221ms Bytes-In:0 Bytes-Out:44306
Request: [2022-11-23T14:26:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3-public/authProviders" Status:200 Latency:105.219µs Bytes-In:0 Bytes-Out:841
Request: [2022-11-23T14:26:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:245.671µs Bytes-In:0 Bytes-Out:33572
Request: [2022-11-23T14:26:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/8bdb609a93df73cddc5c.js" Status:200 Latency:354.322µs Bytes-In:0 Bytes-Out:520
Request: [2022-11-23T14:26:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/shell/assets/images/pl/login-landscape.svg" Status:304 Latency:228.064µs Bytes-In:0 Bytes-Out:0
Request: [2022-11-23T14:26:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/lato-v17-latin-regular.b4d2c4c.woff2" Status:304 Latency:198.174µs Bytes-In:0 Bytes-Out:0
Request: [2022-11-23T14:26:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/icons.425626d.ttf" Status:304 Latency:146.622µs Bytes-In:0 Bytes-Out:0
ERRO[Wed Nov 23 14:26:57 UTC 2022] Failed to create Dex Client: failed to create dex OIDC provider: Get "http://epinio-server.auth.svc.cluster.local/.well-known/openid-configuration": dial tcp: lookup epinio-server.auth.svc.cluster.local: no such host
creating the provider
github.com/epinio/ui-backend/src/jetstream/dex.NewOIDCProviderWithEndpoint
	/home/sles/git/ui-backend/src/jetstream/dex/dex.go:72
main.(*portalProxy).GetDex
	/home/sles/git/ui-backend/src/jetstream/main.go:1325
github.com/epinio/ui-backend/src/jetstream/plugins/epinio/dex.RedirectUrl
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/dex/api.go:18
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).AddRootGroupRoutes.func6
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:220
main.(*portalProxy).setSecureCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:242
main.(*portalProxy).setSecureCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:242
github.com/labstack/echo/v4.(*Echo).add.func1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:520
main.(*portalProxy).urlCheckMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:222
main.(*portalProxy).setStaticCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:230
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).SessionEchoMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:110
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).EchoMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:97
main.retryAfterUpgradeMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:299
main.errorLoggingMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:271
github.com/labstack/echo/v4/middleware.SecureWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/secure.go:142
github.com/labstack/echo/v4/middleware.CORSWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/cors.go:142
github.com/labstack/echo/v4/middleware.RecoverWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/recover.go:119
github.com/labstack/echo/v4/middleware.LoggerWithConfig.func2.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/logger.go:117
main.sessionCleanupMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:201
github.com/labstack/echo/v4.(*Echo).ServeHTTP
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:630
net/http.serverHandler.ServeHTTP
	/usr/lib64/go/1.19/src/net/http/server.go:2947
net/http.(*conn).serve
	/usr/lib64/go/1.19/src/net/http/server.go:1991
runtime.goexit
	/usr/lib64/go/1.19/src/runtime/asm_amd64.s:1594 
{"time":"2022-11-23T14:26:57.221732456Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=500, message={\"status\":\"error\",\"error\":\"Failed to create Dex Client\"}"}
Request: [2022-11-23T14:26:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/dex/redirectUrl" Status:500 Latency:9.67546ms Bytes-In:0 Bytes-Out:56

List of services present in my cluster:

$ kubectl get svc -A
NAMESPACE      NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
default        kubernetes             ClusterIP      10.43.0.1       <none>          443/TCP                      73m
cert-manager   cert-manager           ClusterIP      10.43.138.172   <none>          9402/TCP                     73m
cert-manager   cert-manager-webhook   ClusterIP      10.43.254.250   <none>          443/TCP                      73m
kube-system    kube-dns               ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP       73m
kube-system    metrics-server         ClusterIP      10.43.197.162   <none>          443/TCP                      73m
kube-system    traefik                LoadBalancer   10.43.224.30    10.100.103.16   80:31669/TCP,443:31829/TCP   73m
epinio         minio-svc              ClusterIP      None            <none>          9000/TCP                     52m
epinio         registry               ClusterIP      10.43.140.55    <none>          5000/TCP                     52m
epinio         dex                    ClusterIP      10.43.110.40    <none>          5556/TCP,5558/TCP            52m
epinio         epinio-ui              ClusterIP      10.43.207.38    <none>          80/TCP                       52m
epinio         epinio-server          ClusterIP      10.43.30.121    <none>          80/TCP                       52m
epinio         minio-console          ClusterIP      10.43.41.99     <none>          9001/TCP                     52m
epinio         registry-node          NodePort       10.43.91.74     <none>          30500:30500/TCP              52m
epinio         kubed                  ClusterIP      10.43.10.95     <none>          443/TCP                      52m
epinio         minio                  ClusterIP      10.43.108.50    <none>          9000/TCP                     52m

@richard-cox
Copy link
Member Author

richard-cox commented Nov 23, 2022

I can't really validate the steps you've taken (they're a little off-piste).

When your ui container starts up what are the log lines around

"Epinio Auth url:

I suspect, if the dex url isn't passed in, the internal epinio api url is getting mangled

epinioAuthUrlValue, _ := portalProxy.Env().Lookup(epinioDexAuthUrl)
	if len(epinioAuthUrlValue) == 0 {
		epinioAuthUrlValue = strings.Replace(epinioApiUrlValue, "epinio.", "auth.", 1)
		log.Infof("Didn't find `%s`, falling back to `%s`", epinioDexAuthUrl, epinioAuthUrlValue)
	}

@thehejik
Copy link
Contributor

When your ui container starts up what are the log lines around

"Epinio Auth url:
Epinio Auth url: 'https://auth.10.100.103.16.omg.howdoi.website'

@thehejik
Copy link
Contributor

I moved a bit forward by setting following ENV in epinio-ui deployment.

		- name: EPINIO_DEX_AUTH_URL                             
		  value: https://auth.10.100.103.16.omg.howdoi.website
		- name: EPINIO_API_SKIP_SSL
		  value: "true"

so I setup a github connector in dex-config's config.yaml but then I'm getting another error after pressing Log in with GitHub:

image

and this is what I had in my dex config.yaml:

staticClients:
...
- id: epinio-ui
  name: 'Epinio UI'
  secret: 'jetstream-dex-epinio-ui'
  # Shouldn't be public, https://dexidp.io/docs/custom-scopes-claims-clients/#public-clients
  redirectURIs:
  - "https://epinio.10.100.103.16.omg.howdoi.website/verify-auth"

but if I add - "http://epinio-server.epinio.svc.cluster.local/verify-auth" to redirectURIs my browser cannot reach this local cluster URL.

@richard-cox
Copy link
Member Author

richard-cox commented Dec 5, 2022

The Epinio UI backend sets the Dex redirect URL (which needs to be reachable by the browser) via the EPINIO_UI_URL property. If that property is empty it uses EPINIO_API_URL, which by default is http://epinio-server.%s.svc.cluster.local (which is internal).

I've updated the helm chart PR with changes to ensure the property is set and doesn't default to the internal API url - epinio/helm-charts@1bd17ff.

@thehejik Does that now work?

@thehejik
Copy link
Contributor

thehejik commented Dec 14, 2022

I rebased the chart PR on top of main branch, ran helm package epinio-ui and updated the original epinio-ui-v1.5.1.tgz archive - the changes are reflected because I see the new ENV value set in epinio-ui deployment:

...
      - env:                                                                                                                                                                                                                           
        - name: ALLOWED_ORIGINS                                                                                                                                                                                                        
          value: https://epinio.10.100.103.16.nip.io                                                                                                                                                                                   
        - name: EPINIO_API_URL                                                                                                                                                                                                         
          value: http://epinio-server.epinio.svc.cluster.local                                                                                                                                                                         
        - name: EPINIO_WSS_URL                                                                                                                                                                                                         
          value: ws://epinio-server.epinio.svc.cluster.local                                                                                                                                                                           
        - name: EPINIO_UI_URL                                                                                                                                                                                                          
          value: https://epinio.10.100.103.16.nip.io                                                                                                                                                                                   
        - name: EPINIO_API_SKIP_SSL                                                                                                                                                                                                    
        - name: EPINIO_VERSION                                                                                                                                                                                                         
          value: v1.5.1               
...

Also my local chart uses chart/epinio-ui/values.yaml with custom image docker.io/thehejik/epinio-ui:oidc-dev5 based on dashboard and ui-backend PRs.

Epinio was installed by helm upgrade --install epinio --namespace epinio ./chart/epinio --set skipTraefik=true --set global.domain=10.100.103.16.nip.io --create-namespace

These are logs from epinio-ui pod just after installation:

INFO[Wed Dec 14 13:46:34 UTC 2022] ========================================     
INFO[Wed Dec 14 13:46:34 UTC 2022] === Stratos Jetstream Backend Server ===     
INFO[Wed Dec 14 13:46:34 UTC 2022] ========================================     
INFO[Wed Dec 14 13:46:34 UTC 2022]                                              
INFO[Wed Dec 14 13:46:34 UTC 2022] Initialization started.                      
INFO[Wed Dec 14 13:46:34 UTC 2022] Setting log level to: info                   
INFO[Wed Dec 14 13:46:34 UTC 2022] Configuration loaded.                        
INFO[Wed Dec 14 13:46:34 UTC 2022] Stratos Version: v99.0.0                     
INFO[Wed Dec 14 13:46:34 UTC 2022] HTTP client initialized.                     
INFO[Wed Dec 14 13:46:34 UTC 2022] Encryption key set.                          
INFO[Wed Dec 14 13:46:34 UTC 2022] SQLite Database file: console-database.db    
INFO[Wed Dec 14 13:46:34 UTC 2022] Database appears to now be available.        
INFO[Wed Dec 14 13:46:34 UTC 2022] Database connection pool created.            
INFO[Wed Dec 14 13:46:34 UTC 2022] Session Store Secret detected okay           
INFO[Wed Dec 14 13:46:34 UTC 2022] ========================                     
INFO[Wed Dec 14 13:46:34 UTC 2022] = Stratos DB Migration =                     
INFO[Wed Dec 14 13:46:34 UTC 2022] ========================                     
INFO[Wed Dec 14 13:46:34 UTC 2022] Database provider: sqlite3                   
INFO[Wed Dec 14 13:46:34 UTC 2022] Current 0                                    
INFO[Wed Dec 14 13:46:34 UTC 2022] Target: 20200902162200                       
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migrations ....                      
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20170818120003_InitialSchema 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20170818162837_SetupSchema 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20170829154900_TokenDisconnected 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20171108102900_AuthType   
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20180413135700_MetricsSchema 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20180627111300_UpdateMetadata 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20180703142800_SetupSchema 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20180813110300_RemoveStaleTokens 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20180824092600_LinkedTokens 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20180831104300_SSOEndpointFlag 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20180907123000_SSOSetupFlag 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20181129140500_UserFavorites 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20190305144600_EndpointSubtype 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20190515133200_AuthEndpoint 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20190522121200_LocalUsers 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20190621212700_ConfigSchema 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20190918092300_LocalUsersUpdates 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20190930092500_LocalUsersTriggerFix 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20191008121900_PrimaryKeys 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20200117152200_SesssionData 
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20200814140918_ApiKeys    
INFO[Wed Dec 14 13:46:34 UTC 2022] Running migration: 20200902162200_HelmSubtype 
INFO[Wed Dec 14 13:46:34 UTC 2022] Session expiration (minutes): 1440           
INFO[Wed Dec 14 13:46:34 UTC 2022] Session Cookie Domain:                       
INFO[Wed Dec 14 13:46:34 UTC 2022] Creating SQLite session store                
INFO[Wed Dec 14 13:46:34 UTC 2022] Session store initialized.                   
INFO[Wed Dec 14 13:46:34 UTC 2022] Session data store initialized.              
INFO[Wed Dec 14 13:46:34 UTC 2022] Session Cookie name: console-session         
INFO[Wed Dec 14 13:46:34 UTC 2022] Initialization complete.                     
INFO[Wed Dec 14 13:46:34 UTC 2022] Migrating setup data to config store         
INFO[Wed Dec 14 13:46:34 UTC 2022] Can not migrate setup data - setup table is empty 
INFO[Wed Dec 14 13:46:34 UTC 2022] Initialising plugins                         
ERRO[Wed Dec 14 13:46:34 UTC 2022] Can't generate plugins from YAML: open plugins.yaml: no such file or directory  
INFO[Wed Dec 14 13:46:34 UTC 2022] Didn't find `EPINIO_DEX_AUTH_URL`, falling back to `http://epinio-server.auth.svc.cluster.local` 
INFO[Wed Dec 14 13:46:34 UTC 2022] 
Epinio API url: 'http://epinio-server.epinio.svc.cluster.local'
Epinio WSS url: 'ws://epinio-server.epinio.svc.cluster.local'
Epinio Auth url: 'http://epinio-server.auth.svc.cluster.local'
Epinio UI url: 'https://epinio.10.100.103.16.nip.io'
Skipping SSL Validation: 'false' 
INFO[Wed Dec 14 13:46:34 UTC 2022] Loaded plugin: epinio                        
INFO[Wed Dec 14 13:46:34 UTC 2022] Loaded plugin: userfavorites                 
ERRO[Wed Dec 14 13:46:34 UTC 2022] failed to find an epinio endpoint            
INFO[Wed Dec 14 13:46:34 UTC 2022] Auto-registering epinio endpoint http://epinio-server.epinio.svc.cluster.local as "default" (ZThhPrryoUeUuimbk4Gb-G3-I9A) 
INFO[Wed Dec 14 13:46:34 UTC 2022] Plugins initialized                          
INFO[Wed Dec 14 13:46:34 UTC 2022] Stratos is initialized with the following setup: 
INFO[Wed Dec 14 13:46:34 UTC 2022] ... Auth Endpoint Type      : epinio         
INFO[Wed Dec 14 13:46:34 UTC 2022] ... Epinio Auth             : true           
INFO[Wed Dec 14 13:46:34 UTC 2022] ... Skip SSL Validation     : false          
INFO[Wed Dec 14 13:46:34 UTC 2022] ... Setup Complete          : true           
INFO[Wed Dec 14 13:46:34 UTC 2022] SSO Configuration:                           
INFO[Wed Dec 14 13:46:34 UTC 2022] ... SSO Enabled             : false          
INFO[Wed Dec 14 13:46:34 UTC 2022] ... SSO Options             :                
INFO[Wed Dec 14 13:46:34 UTC 2022] ... SSO Redirect Allow-list :                
INFO[Wed Dec 14 13:46:34 UTC 2022] Storing Diagnostics                          
INFO[Wed Dec 14 13:46:34 UTC 2022] Serving static UI resources                  
INFO[Wed Dec 14 13:46:34 UTC 2022] Starting HTTP Server at address: 0.0.0.0:8000 

@richard-cox
Copy link
Member Author

@thehejik Were you able to log in via Dex? I don't see any obvious error?

@thehejik
Copy link
Contributor

No, I wasn't able, once I clicked on Login over DEX I was returned back to Login page.

Following looks wrong, it's using non-existing internal endpoint http://epinio-server.auth.svc.cluster.local but correct internal ep would be IMO http://dex.epinio.svc.cluster.local:

Can't generate plugins from YAML: open plugins.yaml: no such file or directory  
INFO[Wed Dec 14 13:46:34 UTC 2022] Didn't find `EPINIO_DEX_AUTH_URL`, falling back to `http://epinio-server.auth.svc.cluster.local` 
INFO[Wed Dec 14 13:46:34 UTC 2022] 
Epinio API url: 'http://epinio-server.epinio.svc.cluster.local'
Epinio WSS url: 'ws://epinio-server.epinio.svc.cluster.local'
Epinio Auth url: 'http://epinio-server.auth.svc.cluster.local'
Epinio UI url: 'https://epinio.10.100.103.16.nip.io'
Skipping SSL Validation: 'false' 

@richard-cox
Copy link
Member Author

What were the UI backend logs for when the log in attempt was made, and were there any errors in the browser's network tab?

The http://epinio-server.auth.svc.cluster.local style address was working when I implemented this, maybe there was a change epinio side?

The workaround for this might just be to add something like below to the ui chart

        - name: EPINIO_DEX_AUTH_URL
          value: {{ default (printf "http://dex.%s.svc.cluster.local" .Release.Namespace) .Values.epinioAPIURL }}

@thehejik
Copy link
Contributor

It didn't help, complete ui logs incl. attempt to login via DEX follows, I was returned to login page again.

Click here to expand the logs
INFO[Thu Dec 15 15:45:34 UTC 2022] ========================================     
INFO[Thu Dec 15 15:45:34 UTC 2022] === Stratos Jetstream Backend Server ===     
INFO[Thu Dec 15 15:45:34 UTC 2022] ========================================     
INFO[Thu Dec 15 15:45:34 UTC 2022]                                              
INFO[Thu Dec 15 15:45:34 UTC 2022] Initialization started.                      
INFO[Thu Dec 15 15:45:34 UTC 2022] Setting log level to: info                   
INFO[Thu Dec 15 15:45:34 UTC 2022] Configuration loaded.                        
INFO[Thu Dec 15 15:45:34 UTC 2022] Stratos Version: v99.0.0                     
INFO[Thu Dec 15 15:45:34 UTC 2022] HTTP client initialized.                     
INFO[Thu Dec 15 15:45:34 UTC 2022] Encryption key set.                          
INFO[Thu Dec 15 15:45:34 UTC 2022] SQLite Database file: console-database.db    
INFO[Thu Dec 15 15:45:34 UTC 2022] Database appears to now be available.        
INFO[Thu Dec 15 15:45:34 UTC 2022] Database connection pool created.            
INFO[Thu Dec 15 15:45:34 UTC 2022] Session Store Secret detected okay           
INFO[Thu Dec 15 15:45:34 UTC 2022] ========================                     
INFO[Thu Dec 15 15:45:34 UTC 2022] = Stratos DB Migration =                     
INFO[Thu Dec 15 15:45:34 UTC 2022] ========================                     
INFO[Thu Dec 15 15:45:34 UTC 2022] Database provider: sqlite3                   
INFO[Thu Dec 15 15:45:34 UTC 2022] Current 0                                    
INFO[Thu Dec 15 15:45:34 UTC 2022] Target: 20200902162200                       
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migrations ....                      
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20170818120003_InitialSchema 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20170818162837_SetupSchema 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20170829154900_TokenDisconnected 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20171108102900_AuthType   
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20180413135700_MetricsSchema 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20180627111300_UpdateMetadata 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20180703142800_SetupSchema 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20180813110300_RemoveStaleTokens 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20180824092600_LinkedTokens 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20180831104300_SSOEndpointFlag 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20180907123000_SSOSetupFlag 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20181129140500_UserFavorites 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20190305144600_EndpointSubtype 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20190515133200_AuthEndpoint 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20190522121200_LocalUsers 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20190621212700_ConfigSchema 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20190918092300_LocalUsersUpdates 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20190930092500_LocalUsersTriggerFix 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20191008121900_PrimaryKeys 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20200117152200_SesssionData 
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20200814140918_ApiKeys    
INFO[Thu Dec 15 15:45:34 UTC 2022] Running migration: 20200902162200_HelmSubtype 
INFO[Thu Dec 15 15:45:34 UTC 2022] Session expiration (minutes): 1440           
INFO[Thu Dec 15 15:45:34 UTC 2022] Session Cookie Domain:                       
INFO[Thu Dec 15 15:45:34 UTC 2022] Creating SQLite session store                
INFO[Thu Dec 15 15:45:34 UTC 2022] Session store initialized.                   
INFO[Thu Dec 15 15:45:34 UTC 2022] Session data store initialized.              
INFO[Thu Dec 15 15:45:34 UTC 2022] Session Cookie name: console-session         
INFO[Thu Dec 15 15:45:34 UTC 2022] Initialization complete.                     
INFO[Thu Dec 15 15:45:34 UTC 2022] Migrating setup data to config store         
INFO[Thu Dec 15 15:45:34 UTC 2022] Can not migrate setup data - setup table is empty 
INFO[Thu Dec 15 15:45:34 UTC 2022] Initialising plugins                         
ERRO[Thu Dec 15 15:45:34 UTC 2022] Can't generate plugins from YAML: open plugins.yaml: no such file or directory  
INFO[Thu Dec 15 15:45:34 UTC 2022] 
Epinio API url: 'http://epinio-server.epinio.svc.cluster.local'
Epinio WSS url: 'ws://epinio-server.epinio.svc.cluster.local'
Epinio Auth url: 'http://dex.epinio.svc.cluster.local'
Epinio UI url: 'https://epinio.10.100.103.16.nip.io'
Skipping SSL Validation: 'false' 
INFO[Thu Dec 15 15:45:34 UTC 2022] Loaded plugin: epinio                        
INFO[Thu Dec 15 15:45:34 UTC 2022] Loaded plugin: userfavorites                 
ERRO[Thu Dec 15 15:45:34 UTC 2022] failed to find an epinio endpoint            
INFO[Thu Dec 15 15:45:34 UTC 2022] Auto-registering epinio endpoint http://epinio-server.epinio.svc.cluster.local as "default" (ZThhPrryoUeUuimbk4Gb-G3-I9A) 
INFO[Thu Dec 15 15:45:34 UTC 2022] Plugins initialized                          
INFO[Thu Dec 15 15:45:34 UTC 2022] Stratos is initialized with the following setup: 
INFO[Thu Dec 15 15:45:34 UTC 2022] ... Auth Endpoint Type      : epinio         
INFO[Thu Dec 15 15:45:34 UTC 2022] ... Epinio Auth             : true           
INFO[Thu Dec 15 15:45:34 UTC 2022] ... Skip SSL Validation     : false          
INFO[Thu Dec 15 15:45:34 UTC 2022] ... Setup Complete          : true           
INFO[Thu Dec 15 15:45:34 UTC 2022] SSO Configuration:                           
INFO[Thu Dec 15 15:45:34 UTC 2022] ... SSO Enabled             : false          
INFO[Thu Dec 15 15:45:34 UTC 2022] ... SSO Options             :                
INFO[Thu Dec 15 15:45:34 UTC 2022] ... SSO Redirect Allow-list :                
INFO[Thu Dec 15 15:45:34 UTC 2022] Storing Diagnostics                          
INFO[Thu Dec 15 15:45:34 UTC 2022] Serving static UI resources                  
INFO[Thu Dec 15 15:45:34 UTC 2022] Starting HTTP Server at address: 0.0.0.0:8000 
Request: [2022-12-15T15:46:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/" Status:200 Latency:1.222808ms Bytes-In:0 Bytes-Out:1628
Request: [2022-12-15T15:46:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/32d4fa2.js" Status:200 Latency:698.049µs Bytes-In:0 Bytes-Out:5614
Request: [2022-12-15T15:46:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/9612c993c818b54654e6.js" Status:200 Latency:17.909858ms Bytes-In:0 Bytes-Out:285526
Request: [2022-12-15T15:46:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/e1ce16294c3abbc329e4.js" Status:200 Latency:102.628561ms Bytes-In:0 Bytes-Out:1908773
Request: [2022-12-15T15:46:05Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/f4e3def3f733ae290120.js" Status:200 Latency:320.286615ms Bytes-In:0 Bytes-Out:9955727
Request: [2022-12-15T15:46:05Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/bd3b95dee2d6560ec319.js" Status:200 Latency:461.164µs Bytes-In:0 Bytes-Out:791
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:655.098µs Bytes-In:0 Bytes-Out:33500
ERRO[Thu Dec 15 15:46:06 UTC 2022] User session could not be found: securecookie: the value is not valid 
{"time":"2022-12-15T15:46:06.253447755Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/users" Status:401 Latency:228.113µs Bytes-In:0 Bytes-Out:60
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/favicon.png" Status:200 Latency:1.01591ms Bytes-In:0 Bytes-Out:759
ERRO[Thu Dec 15 15:46:06 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-15T15:46:06.285257824Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/principals" Status:401 Latency:181.964µs Bytes-In:0 Bytes-Out:60
ERRO[Thu Dec 15 15:46:06 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-15T15:46:06.285507994Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"POST" Path:"/pp/v1/epinio/rancher/v3/tokens" Status:401 Latency:61.716µs Bytes-In:2 Bytes-Out:60
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/2463fffff3150156856b.js" Status:200 Latency:1.522352ms Bytes-In:0 Bytes-Out:44298
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3-public/authProviders" Status:200 Latency:199.843µs Bytes-In:0 Bytes-Out:781
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:250.936µs Bytes-In:0 Bytes-Out:33500
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/ba5ddf4cbe7c4451b0d3.js" Status:200 Latency:263.512µs Bytes-In:0 Bytes-Out:540
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/shell/assets/images/pl/login-landscape.svg" Status:200 Latency:1.712294ms Bytes-In:0 Bytes-Out:34525
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/icons.425626d.ttf" Status:200 Latency:1.479098ms Bytes-In:0 Bytes-Out:19556
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/lato-v17-latin-regular.b4d2c4c.woff2" Status:200 Latency:4.00102ms Bytes-In:0 Bytes-Out:23484
ERRO[Thu Dec 15 15:46:06 UTC 2022] Failed to create Dex Client: failed to create dex OIDC provider: Get "http://dex.epinio.svc.cluster.local/.well-known/openid-configuration": dial tcp 10.43.165.235:80: connect: no route to host
creating the provider
github.com/epinio/ui-backend/src/jetstream/dex.NewOIDCProviderWithEndpoint
	/home/sles/git/ui-backend/src/jetstream/dex/dex.go:72
main.(*portalProxy).GetDex
	/home/sles/git/ui-backend/src/jetstream/main.go:1325
github.com/epinio/ui-backend/src/jetstream/plugins/epinio/dex.RedirectUrl
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/dex/api.go:18
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).AddRootGroupRoutes.func6
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:220
main.(*portalProxy).setSecureCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:242
main.(*portalProxy).setSecureCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:242
github.com/labstack/echo/v4.(*Echo).add.func1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:520
main.(*portalProxy).urlCheckMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:222
main.(*portalProxy).setStaticCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:230
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).SessionEchoMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:110
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).EchoMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:97
main.retryAfterUpgradeMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:299
main.errorLoggingMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:271
github.com/labstack/echo/v4/middleware.SecureWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/secure.go:142
github.com/labstack/echo/v4/middleware.CORSWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/cors.go:142
github.com/labstack/echo/v4/middleware.RecoverWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/recover.go:119
github.com/labstack/echo/v4/middleware.LoggerWithConfig.func2.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/logger.go:117
main.sessionCleanupMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:201
github.com/labstack/echo/v4.(*Echo).ServeHTTP
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:630
net/http.serverHandler.ServeHTTP
	/usr/lib64/go/1.19/src/net/http/server.go:2947
net/http.(*conn).serve
	/usr/lib64/go/1.19/src/net/http/server.go:1991
runtime.goexit
	/usr/lib64/go/1.19/src/runtime/asm_amd64.s:1594 
{"time":"2022-12-15T15:46:06.487608884Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=500, message={\"status\":\"error\",\"error\":\"Failed to create Dex Client\"}"}
Request: [2022-12-15T15:46:06Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/dex/redirectUrl" Status:500 Latency:4.676522ms Bytes-In:0 Bytes-Out:56
Request: [2022-12-15T15:46:09Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/:///" Status:200 Latency:318.993µs Bytes-In:0 Bytes-Out:1628
Request: [2022-12-15T15:46:09Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/9612c993c818b54654e6.js" Status:304 Latency:229.508µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-15T15:46:09Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/32d4fa2.js" Status:304 Latency:165.512µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-15T15:46:09Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/e1ce16294c3abbc329e4.js" Status:304 Latency:273.07µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-15T15:46:09Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/f4e3def3f733ae290120.js" Status:304 Latency:141.864µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:297.969µs Bytes-In:0 Bytes-Out:33500
ERRO[Thu Dec 15 15:46:10 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-15T15:46:10.439538661Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/users" Status:401 Latency:101.305µs Bytes-In:0 Bytes-Out:60
ERRO[Thu Dec 15 15:46:10 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-15T15:46:10.453304619Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"POST" Path:"/pp/v1/epinio/rancher/v3/tokens" Status:401 Latency:106.248µs Bytes-In:2 Bytes-Out:60
ERRO[Thu Dec 15 15:46:10 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-15T15:46:10.453657895Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/principals" Status:401 Latency:90.076µs Bytes-In:0 Bytes-Out:60
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/2463fffff3150156856b.js" Status:200 Latency:1.319675ms Bytes-In:0 Bytes-Out:44298
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3-public/authProviders" Status:200 Latency:69.238µs Bytes-In:0 Bytes-Out:781
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:289.243µs Bytes-In:0 Bytes-Out:33500
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/ba5ddf4cbe7c4451b0d3.js" Status:200 Latency:244.581µs Bytes-In:0 Bytes-Out:540
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/shell/assets/images/pl/login-landscape.svg" Status:304 Latency:217.3µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/icons.425626d.ttf" Status:304 Latency:239.954µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/lato-v17-latin-regular.b4d2c4c.woff2" Status:304 Latency:229.975µs Bytes-In:0 Bytes-Out:0
ERRO[Thu Dec 15 15:46:10 UTC 2022] Failed to create Dex Client: failed to create dex OIDC provider: Get "http://dex.epinio.svc.cluster.local/.well-known/openid-configuration": dial tcp 10.43.165.235:80: connect: no route to host
creating the provider
github.com/epinio/ui-backend/src/jetstream/dex.NewOIDCProviderWithEndpoint
	/home/sles/git/ui-backend/src/jetstream/dex/dex.go:72
main.(*portalProxy).GetDex
	/home/sles/git/ui-backend/src/jetstream/main.go:1325
github.com/epinio/ui-backend/src/jetstream/plugins/epinio/dex.RedirectUrl
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/dex/api.go:18
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).AddRootGroupRoutes.func6
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:220
main.(*portalProxy).setSecureCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:242
main.(*portalProxy).setSecureCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:242
github.com/labstack/echo/v4.(*Echo).add.func1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:520
main.(*portalProxy).urlCheckMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:222
main.(*portalProxy).setStaticCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:230
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).SessionEchoMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:110
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).EchoMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:97
main.retryAfterUpgradeMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:299
main.errorLoggingMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:271
github.com/labstack/echo/v4/middleware.SecureWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/secure.go:142
github.com/labstack/echo/v4/middleware.CORSWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/cors.go:142
github.com/labstack/echo/v4/middleware.RecoverWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/recover.go:119
github.com/labstack/echo/v4/middleware.LoggerWithConfig.func2.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/logger.go:117
main.sessionCleanupMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:201
github.com/labstack/echo/v4.(*Echo).ServeHTTP
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:630
net/http.serverHandler.ServeHTTP
	/usr/lib64/go/1.19/src/net/http/server.go:2947
net/http.(*conn).serve
	/usr/lib64/go/1.19/src/net/http/server.go:1991
runtime.goexit
	/usr/lib64/go/1.19/src/runtime/asm_amd64.s:1594 
{"time":"2022-12-15T15:46:10.618450815Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=500, message={\"status\":\"error\",\"error\":\"Failed to create Dex Client\"}"}
Request: [2022-12-15T15:46:10Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/dex/redirectUrl" Status:500 Latency:2.996459ms Bytes-In:0 Bytes-Out:56

@richard-cox
Copy link
Member Author

@thehejik I've updated the helm chart following the discussion in slack, should be ready to test again

@thehejik
Copy link
Contributor

thehejik commented Dec 16, 2022

@richard-cox doesn't work, I was redirected back to login page on attempt to login over DEX. I did use default values and --set global.domain=10.100.103.16.nip.io

Now this seems to be a problem:

ERRO[Fri Dec 16 10:50:43 UTC 2022] Failed to create Dex Client: failed to create dex OIDC provider: oidc: issuer did not match the issuer returned by provider, expected "http://dex.epinio.svc.cluster.local:5556" got "https://auth.10.100.103.16.nip.io"
Click here to expand full logs
INFO[Fri Dec 16 10:50:08 UTC 2022] ========================================     
INFO[Fri Dec 16 10:50:08 UTC 2022] === Stratos Jetstream Backend Server ===     
INFO[Fri Dec 16 10:50:08 UTC 2022] ========================================     
INFO[Fri Dec 16 10:50:08 UTC 2022]                                              
INFO[Fri Dec 16 10:50:08 UTC 2022] Initialization started.                      
INFO[Fri Dec 16 10:50:08 UTC 2022] Setting log level to: info                   
INFO[Fri Dec 16 10:50:08 UTC 2022] Configuration loaded.                        
INFO[Fri Dec 16 10:50:08 UTC 2022] Stratos Version: v99.0.0                     
INFO[Fri Dec 16 10:50:08 UTC 2022] HTTP client initialized.                     
INFO[Fri Dec 16 10:50:08 UTC 2022] Encryption key set.                          
INFO[Fri Dec 16 10:50:08 UTC 2022] SQLite Database file: console-database.db    
INFO[Fri Dec 16 10:50:08 UTC 2022] Database appears to now be available.        
INFO[Fri Dec 16 10:50:08 UTC 2022] Database connection pool created.            
INFO[Fri Dec 16 10:50:08 UTC 2022] Session Store Secret detected okay           
INFO[Fri Dec 16 10:50:08 UTC 2022] ========================                     
INFO[Fri Dec 16 10:50:08 UTC 2022] = Stratos DB Migration =                     
INFO[Fri Dec 16 10:50:08 UTC 2022] ========================                     
INFO[Fri Dec 16 10:50:08 UTC 2022] Database provider: sqlite3                   
INFO[Fri Dec 16 10:50:08 UTC 2022] Current 0                                    
INFO[Fri Dec 16 10:50:08 UTC 2022] Target: 20200902162200                       
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migrations ....                      
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20170818120003_InitialSchema 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20170818162837_SetupSchema 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20170829154900_TokenDisconnected 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20171108102900_AuthType   
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20180413135700_MetricsSchema 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20180627111300_UpdateMetadata 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20180703142800_SetupSchema 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20180813110300_RemoveStaleTokens 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20180824092600_LinkedTokens 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20180831104300_SSOEndpointFlag 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20180907123000_SSOSetupFlag 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20181129140500_UserFavorites 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20190305144600_EndpointSubtype 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20190515133200_AuthEndpoint 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20190522121200_LocalUsers 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20190621212700_ConfigSchema 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20190918092300_LocalUsersUpdates 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20190930092500_LocalUsersTriggerFix 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20191008121900_PrimaryKeys 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20200117152200_SesssionData 
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20200814140918_ApiKeys    
INFO[Fri Dec 16 10:50:08 UTC 2022] Running migration: 20200902162200_HelmSubtype 
INFO[Fri Dec 16 10:50:08 UTC 2022] Session expiration (minutes): 1440           
INFO[Fri Dec 16 10:50:08 UTC 2022] Session Cookie Domain:                       
INFO[Fri Dec 16 10:50:08 UTC 2022] Creating SQLite session store                
INFO[Fri Dec 16 10:50:08 UTC 2022] Session store initialized.                   
INFO[Fri Dec 16 10:50:08 UTC 2022] Session data store initialized.              
INFO[Fri Dec 16 10:50:08 UTC 2022] Session Cookie name: console-session         
INFO[Fri Dec 16 10:50:08 UTC 2022] Initialization complete.                     
INFO[Fri Dec 16 10:50:08 UTC 2022] Migrating setup data to config store         
INFO[Fri Dec 16 10:50:08 UTC 2022] Can not migrate setup data - setup table is empty 
INFO[Fri Dec 16 10:50:08 UTC 2022] Initialising plugins                         
ERRO[Fri Dec 16 10:50:08 UTC 2022] Can't generate plugins from YAML: open plugins.yaml: no such file or directory  
INFO[Fri Dec 16 10:50:08 UTC 2022] 
Epinio API url: 'http://epinio-server.epinio.svc.cluster.local'
Epinio WSS url: 'ws://epinio-server.epinio.svc.cluster.local'
Epinio Auth url: 'http://dex.epinio.svc.cluster.local:5556'
Epinio UI url: 'https://epinio.10.100.103.16.nip.io'
Skipping SSL Validation: 'false' 
INFO[Fri Dec 16 10:50:08 UTC 2022] Loaded plugin: epinio                        
INFO[Fri Dec 16 10:50:08 UTC 2022] Loaded plugin: userfavorites                 
ERRO[Fri Dec 16 10:50:08 UTC 2022] failed to find an epinio endpoint            
INFO[Fri Dec 16 10:50:08 UTC 2022] Auto-registering epinio endpoint http://epinio-server.epinio.svc.cluster.local as "default" (ZThhPrryoUeUuimbk4Gb-G3-I9A) 
INFO[Fri Dec 16 10:50:08 UTC 2022] Plugins initialized                          
INFO[Fri Dec 16 10:50:08 UTC 2022] Stratos is initialized with the following setup: 
INFO[Fri Dec 16 10:50:08 UTC 2022] ... Auth Endpoint Type      : epinio         
INFO[Fri Dec 16 10:50:08 UTC 2022] ... Epinio Auth             : true           
INFO[Fri Dec 16 10:50:08 UTC 2022] ... Skip SSL Validation     : false          
INFO[Fri Dec 16 10:50:08 UTC 2022] ... Setup Complete          : true           
INFO[Fri Dec 16 10:50:08 UTC 2022] SSO Configuration:                           
INFO[Fri Dec 16 10:50:08 UTC 2022] ... SSO Enabled             : false          
INFO[Fri Dec 16 10:50:08 UTC 2022] ... SSO Options             :                
INFO[Fri Dec 16 10:50:08 UTC 2022] ... SSO Redirect Allow-list :                
INFO[Fri Dec 16 10:50:08 UTC 2022] Storing Diagnostics                          
INFO[Fri Dec 16 10:50:08 UTC 2022] Serving static UI resources                  
INFO[Fri Dec 16 10:50:08 UTC 2022] Starting HTTP Server at address: 0.0.0.0:8000 
Request: [2022-12-16T10:50:42Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/:///" Status:200 Latency:926.457µs Bytes-In:0 Bytes-Out:1628
Request: [2022-12-16T10:50:42Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/32d4fa2.js" Status:304 Latency:383.978µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-16T10:50:42Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/e1ce16294c3abbc329e4.js" Status:304 Latency:1.771587ms Bytes-In:0 Bytes-Out:0
Request: [2022-12-16T10:50:42Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/9612c993c818b54654e6.js" Status:304 Latency:3.46033ms Bytes-In:0 Bytes-Out:0
Request: [2022-12-16T10:50:42Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/f4e3def3f733ae290120.js" Status:304 Latency:1.966975ms Bytes-In:0 Bytes-Out:0
Request: [2022-12-16T10:50:42Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:551.044µs Bytes-In:0 Bytes-Out:33500
ERRO[Fri Dec 16 10:50:43 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T10:50:43.088166166Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/users" Status:401 Latency:278.444µs Bytes-In:0 Bytes-Out:60
ERRO[Fri Dec 16 10:50:43 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T10:50:43.127262621Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"POST" Path:"/pp/v1/epinio/rancher/v3/tokens" Status:401 Latency:136.486µs Bytes-In:2 Bytes-Out:60
ERRO[Fri Dec 16 10:50:43 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T10:50:43.128312869Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/principals" Status:401 Latency:71.393µs Bytes-In:0 Bytes-Out:60
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/2463fffff3150156856b.js" Status:200 Latency:1.526334ms Bytes-In:0 Bytes-Out:44298
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3-public/authProviders" Status:200 Latency:112.929µs Bytes-In:0 Bytes-Out:781
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:236.603µs Bytes-In:0 Bytes-Out:33500
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/ba5ddf4cbe7c4451b0d3.js" Status:200 Latency:794.767µs Bytes-In:0 Bytes-Out:540
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/shell/assets/images/pl/login-landscape.svg" Status:304 Latency:263.116µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/icons.425626d.ttf" Status:304 Latency:315.383µs Bytes-In:0 Bytes-Out:0
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/lato-v17-latin-regular.b4d2c4c.woff2" Status:304 Latency:1.165091ms Bytes-In:0 Bytes-Out:0
ERRO[Fri Dec 16 10:50:43 UTC 2022] Failed to create Dex Client: failed to create dex OIDC provider: oidc: issuer did not match the issuer returned by provider, expected "http://dex.epinio.svc.cluster.local:5556" got "https://auth.10.100.103.16.nip.io"
creating the provider
github.com/epinio/ui-backend/src/jetstream/dex.NewOIDCProviderWithEndpoint
	/home/sles/git/ui-backend/src/jetstream/dex/dex.go:72
main.(*portalProxy).GetDex
	/home/sles/git/ui-backend/src/jetstream/main.go:1325
github.com/epinio/ui-backend/src/jetstream/plugins/epinio/dex.RedirectUrl
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/dex/api.go:18
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).AddRootGroupRoutes.func6
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:220
main.(*portalProxy).setSecureCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:242
main.(*portalProxy).setSecureCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:242
github.com/labstack/echo/v4.(*Echo).add.func1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:520
main.(*portalProxy).urlCheckMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:222
main.(*portalProxy).setStaticCacheContentMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:230
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).SessionEchoMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:110
github.com/epinio/ui-backend/src/jetstream/plugins/epinio.(*Epinio).EchoMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/plugins/epinio/main.go:97
main.retryAfterUpgradeMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:299
main.errorLoggingMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:271
github.com/labstack/echo/v4/middleware.SecureWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/secure.go:142
github.com/labstack/echo/v4/middleware.CORSWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/cors.go:142
github.com/labstack/echo/v4/middleware.RecoverWithConfig.func1.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/recover.go:119
github.com/labstack/echo/v4/middleware.LoggerWithConfig.func2.1
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/logger.go:117
main.sessionCleanupMiddleware.func1
	/home/sles/git/ui-backend/src/jetstream/middleware.go:201
github.com/labstack/echo/v4.(*Echo).ServeHTTP
	/home/sles/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:630
net/http.serverHandler.ServeHTTP
	/usr/lib64/go/1.19/src/net/http/server.go:2947
net/http.(*conn).serve
	/usr/lib64/go/1.19/src/net/http/server.go:1991
runtime.goexit
	/usr/lib64/go/1.19/src/runtime/asm_amd64.s:1594 
{"time":"2022-12-16T10:50:43.381914918Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=500, message={\"status\":\"error\",\"error\":\"Failed to create Dex Client\"}"}
Request: [2022-12-16T10:50:43Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/dex/redirectUrl" Status:500 Latency:7.85122ms Bytes-In:0 Bytes-Out:56

@richard-cox
Copy link
Member Author

Ok, last attempt before I switch back and dive in to code (this is burning a lot of your time).

There's something that's not aligning, but if the auth url should match the public something like this could work

        - name: EPINIO_DEX_AUTH_URL
          value: {{ default (printf "https://auth.%s" .Values.global.domain) .Values.epinioDexURL }}

@enrichman
Copy link
Member

enrichman commented Dec 16, 2022

If you see the dex.yaml file we had to split the issuer URL from the actual URL used to reach the instance, I guess it's the same issue that you are facing:

https://github.com/epinio/helm-charts/blob/ef8621b543d8db0ae0daa5c932627116ea9c0378/chart/epinio/templates/dex.yaml#L12-L13

https://github.com/epinio/epinio/blob/794fcb049c41687ff3834c6e80409f070adb9654/internal/dex/dex.go#L43-L50

@richard-cox
Copy link
Member Author

@thehejik What flavour of k8s are you using?

@thehejik
Copy link
Contributor

@thehejik What flavour of k8s are you using?

Single-node k3s 1.24.x with Traefik Ingress Controller

@thehejik
Copy link
Contributor

Ok, last attempt before I switch back and dive in to code (this is burning a lot of your time).

There's something that's not aligning, but if the auth url should match the public something like this could work

        - name: EPINIO_DEX_AUTH_URL
          value: {{ default (printf "https://auth.%s" .Values.global.domain) .Values.epinioDexURL }}

Even that didn't work. It failed on certificate error:

ERRO[Fri Dec 16 11:41:18 UTC 2022] Failed to create Dex Client: failed to create dex OIDC provider: Get "https://auth.10.100.103.16.nip.io/.well-known/openid-configuration": x509: certificate signed by unknown authority
creating the provider

So I set in addition this in chart/epinio-ui/templates/server.yaml:

         - name: EPINIO_API_SKIP_SSL
-          value: {{ .Values.epinioAPISkipSSL | quote }}
+          value: {{ default "true" .Values.epinioAPISkipSSL | quote }}

I moved a bit forward, I was redirected to DEX page, then I use preconfigured email account and Pressed Grant Access but I was forwarded back to the epinio login page which is showing red string Log in again to continue..

Full logs with `EPINIO_API_SKIP_SSL=true` here
INFO[Fri Dec 16 11:48:22 UTC 2022] ========================================     
INFO[Fri Dec 16 11:48:22 UTC 2022] === Stratos Jetstream Backend Server ===     
INFO[Fri Dec 16 11:48:22 UTC 2022] ========================================     
INFO[Fri Dec 16 11:48:22 UTC 2022]                                              
INFO[Fri Dec 16 11:48:22 UTC 2022] Initialization started.                      
INFO[Fri Dec 16 11:48:22 UTC 2022] Setting log level to: info                   
INFO[Fri Dec 16 11:48:22 UTC 2022] Configuration loaded.                        
INFO[Fri Dec 16 11:48:22 UTC 2022] Stratos Version: v99.0.0                     
INFO[Fri Dec 16 11:48:22 UTC 2022] HTTP client initialized.                     
INFO[Fri Dec 16 11:48:22 UTC 2022] Encryption key set.                          
INFO[Fri Dec 16 11:48:22 UTC 2022] SQLite Database file: console-database.db    
INFO[Fri Dec 16 11:48:22 UTC 2022] Database appears to now be available.        
INFO[Fri Dec 16 11:48:22 UTC 2022] Database connection pool created.            
INFO[Fri Dec 16 11:48:22 UTC 2022] Session Store Secret detected okay           
INFO[Fri Dec 16 11:48:22 UTC 2022] ========================                     
INFO[Fri Dec 16 11:48:22 UTC 2022] = Stratos DB Migration =                     
INFO[Fri Dec 16 11:48:22 UTC 2022] ========================                     
INFO[Fri Dec 16 11:48:22 UTC 2022] Database provider: sqlite3                   
INFO[Fri Dec 16 11:48:22 UTC 2022] Current 0                                    
INFO[Fri Dec 16 11:48:22 UTC 2022] Target: 20200902162200                       
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migrations ....                      
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20170818120003_InitialSchema 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20170818162837_SetupSchema 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20170829154900_TokenDisconnected 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20171108102900_AuthType   
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20180413135700_MetricsSchema 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20180627111300_UpdateMetadata 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20180703142800_SetupSchema 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20180813110300_RemoveStaleTokens 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20180824092600_LinkedTokens 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20180831104300_SSOEndpointFlag 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20180907123000_SSOSetupFlag 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20181129140500_UserFavorites 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20190305144600_EndpointSubtype 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20190515133200_AuthEndpoint 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20190522121200_LocalUsers 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20190621212700_ConfigSchema 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20190918092300_LocalUsersUpdates 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20190930092500_LocalUsersTriggerFix 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20191008121900_PrimaryKeys 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20200117152200_SesssionData 
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20200814140918_ApiKeys    
INFO[Fri Dec 16 11:48:22 UTC 2022] Running migration: 20200902162200_HelmSubtype 
INFO[Fri Dec 16 11:48:22 UTC 2022] Session expiration (minutes): 1440           
INFO[Fri Dec 16 11:48:22 UTC 2022] Session Cookie Domain:                       
INFO[Fri Dec 16 11:48:22 UTC 2022] Creating SQLite session store                
INFO[Fri Dec 16 11:48:22 UTC 2022] Session store initialized.                   
INFO[Fri Dec 16 11:48:22 UTC 2022] Session data store initialized.              
INFO[Fri Dec 16 11:48:22 UTC 2022] Session Cookie name: console-session         
INFO[Fri Dec 16 11:48:22 UTC 2022] Initialization complete.                     
INFO[Fri Dec 16 11:48:22 UTC 2022] Migrating setup data to config store         
INFO[Fri Dec 16 11:48:22 UTC 2022] Can not migrate setup data - setup table is empty 
INFO[Fri Dec 16 11:48:22 UTC 2022] Initialising plugins                         
ERRO[Fri Dec 16 11:48:22 UTC 2022] Can't generate plugins from YAML: open plugins.yaml: no such file or directory  
INFO[Fri Dec 16 11:48:22 UTC 2022] 
Epinio API url: 'http://epinio-server.epinio.svc.cluster.local'
Epinio WSS url: 'ws://epinio-server.epinio.svc.cluster.local'
Epinio Auth url: 'https://auth.10.100.103.16.nip.io'
Epinio UI url: 'https://epinio.10.100.103.16.nip.io'
Skipping SSL Validation: 'true' 
INFO[Fri Dec 16 11:48:22 UTC 2022] Loaded plugin: epinio                        
INFO[Fri Dec 16 11:48:22 UTC 2022] Loaded plugin: userfavorites                 
ERRO[Fri Dec 16 11:48:22 UTC 2022] failed to find an epinio endpoint            
INFO[Fri Dec 16 11:48:22 UTC 2022] Auto-registering epinio endpoint http://epinio-server.epinio.svc.cluster.local as "default" (ZThhPrryoUeUuimbk4Gb-G3-I9A) 
INFO[Fri Dec 16 11:48:22 UTC 2022] Plugins initialized                          
INFO[Fri Dec 16 11:48:22 UTC 2022] Stratos is initialized with the following setup: 
INFO[Fri Dec 16 11:48:22 UTC 2022] ... Auth Endpoint Type      : epinio         
INFO[Fri Dec 16 11:48:22 UTC 2022] ... Epinio Auth             : true           
INFO[Fri Dec 16 11:48:22 UTC 2022] ... Skip SSL Validation     : false          
INFO[Fri Dec 16 11:48:22 UTC 2022] ... Setup Complete          : true           
INFO[Fri Dec 16 11:48:22 UTC 2022] SSO Configuration:                           
INFO[Fri Dec 16 11:48:22 UTC 2022] ... SSO Enabled             : false          
INFO[Fri Dec 16 11:48:22 UTC 2022] ... SSO Options             :                
INFO[Fri Dec 16 11:48:22 UTC 2022] ... SSO Redirect Allow-list :                
INFO[Fri Dec 16 11:48:22 UTC 2022] Storing Diagnostics                          
INFO[Fri Dec 16 11:48:22 UTC 2022] Serving static UI resources                  
INFO[Fri Dec 16 11:48:22 UTC 2022] Starting HTTP Server at address: 0.0.0.0:8000 
Request: [2022-12-16T11:48:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/" Status:200 Latency:949.997µs Bytes-In:0 Bytes-Out:1628
Request: [2022-12-16T11:48:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/32d4fa2.js" Status:200 Latency:593.321µs Bytes-In:0 Bytes-Out:5614
Request: [2022-12-16T11:48:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/9612c993c818b54654e6.js" Status:200 Latency:29.03128ms Bytes-In:0 Bytes-Out:285526
Request: [2022-12-16T11:48:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/e1ce16294c3abbc329e4.js" Status:200 Latency:179.883489ms Bytes-In:0 Bytes-Out:1908773
Request: [2022-12-16T11:48:57Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/f4e3def3f733ae290120.js" Status:200 Latency:438.552773ms Bytes-In:0 Bytes-Out:9955727
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/bd3b95dee2d6560ec319.js" Status:200 Latency:320.023µs Bytes-In:0 Bytes-Out:791
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:446.473µs Bytes-In:0 Bytes-Out:33500
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/favicon.png" Status:200 Latency:366.371µs Bytes-In:0 Bytes-Out:759
ERRO[Fri Dec 16 11:49:03 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T11:49:03.766423818Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/users" Status:401 Latency:291.188µs Bytes-In:0 Bytes-Out:60
ERRO[Fri Dec 16 11:49:03 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T11:49:03.80767618Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"POST" Path:"/pp/v1/epinio/rancher/v3/tokens" Status:401 Latency:119.058µs Bytes-In:2 Bytes-Out:60
ERRO[Fri Dec 16 11:49:03 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T11:49:03.81005615Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/principals" Status:401 Latency:76.381µs Bytes-In:0 Bytes-Out:60
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/2463fffff3150156856b.js" Status:200 Latency:1.584738ms Bytes-In:0 Bytes-Out:44298
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3-public/authProviders" Status:200 Latency:98.373µs Bytes-In:0 Bytes-Out:781
Request: [2022-12-16T11:49:03Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:315.07µs Bytes-In:0 Bytes-Out:33500
Request: [2022-12-16T11:49:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/ba5ddf4cbe7c4451b0d3.js" Status:200 Latency:276.103µs Bytes-In:0 Bytes-Out:540
Request: [2022-12-16T11:49:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/lato-v17-latin-regular.b4d2c4c.woff2" Status:200 Latency:1.127866ms Bytes-In:0 Bytes-Out:23484
Request: [2022-12-16T11:49:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/shell/assets/images/pl/login-landscape.svg" Status:200 Latency:1.928386ms Bytes-In:0 Bytes-Out:34525
Request: [2022-12-16T11:49:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/icons.425626d.ttf" Status:200 Latency:975.973µs Bytes-In:0 Bytes-Out:19556
Request: [2022-12-16T11:49:04Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/dex/redirectUrl" Status:200 Latency:34.212831ms Bytes-In:0 Bytes-Out:510
Request: [2022-12-16T11:49:19Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/verify-auth" Status:200 Latency:257.345µs Bytes-In:0 Bytes-Out:1628
Request: [2022-12-16T11:49:19Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/32d4fa2.js" Status:200 Latency:405.42µs Bytes-In:0 Bytes-Out:5614
Request: [2022-12-16T11:49:19Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/9612c993c818b54654e6.js" Status:200 Latency:16.07899ms Bytes-In:0 Bytes-Out:285526
Request: [2022-12-16T11:49:19Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/e1ce16294c3abbc329e4.js" Status:200 Latency:189.735356ms Bytes-In:0 Bytes-Out:1908773
Request: [2022-12-16T11:49:20Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/f4e3def3f733ae290120.js" Status:200 Latency:417.993131ms Bytes-In:0 Bytes-Out:9955727
Request: [2022-12-16T11:49:25Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:421.551µs Bytes-In:0 Bytes-Out:33500
ERRO[Fri Dec 16 11:49:25 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T11:49:25.675204962Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T11:49:25Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/users" Status:401 Latency:180.934µs Bytes-In:0 Bytes-Out:60
ERRO[Fri Dec 16 11:49:25 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T11:49:25.713421461Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T11:49:25Z] Remote-IP:"10.42.0.1" Method:"POST" Path:"/pp/v1/epinio/rancher/v3/tokens" Status:401 Latency:188.152µs Bytes-In:2 Bytes-Out:60
ERRO[Fri Dec 16 11:49:25 UTC 2022] User session could not be found: Session value not found user_id 
{"time":"2022-12-16T11:49:25.715130022Z","level":"ERROR","prefix":"echo","file":"main.go","line":"1258","message":"code=401, message={\"status\":\"error\",\"error\":\"User session could not be found\"}"}
Request: [2022-12-16T11:49:25Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3/principals" Status:401 Latency:71.562µs Bytes-In:0 Bytes-Out:60
Request: [2022-12-16T11:49:25Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/2463fffff3150156856b.js" Status:200 Latency:1.587674ms Bytes-In:0 Bytes-Out:44298
Request: [2022-12-16T11:49:25Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v3-public/authProviders" Status:200 Latency:92.989µs Bytes-In:0 Bytes-Out:781
Request: [2022-12-16T11:49:25Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/v1/management.cattle.io.setting" Status:200 Latency:270.843µs Bytes-In:0 Bytes-Out:33500
Request: [2022-12-16T11:49:25Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/ba5ddf4cbe7c4451b0d3.js" Status:200 Latency:343.389µs Bytes-In:0 Bytes-Out:540
Request: [2022-12-16T11:49:26Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/lato-v17-latin-regular.b4d2c4c.woff2" Status:200 Latency:1.532356ms Bytes-In:0 Bytes-Out:23484
Request: [2022-12-16T11:49:26Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/shell/assets/images/pl/login-landscape.svg" Status:200 Latency:1.727139ms Bytes-In:0 Bytes-Out:34525
Request: [2022-12-16T11:49:26Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/_nuxt/fonts/icons.425626d.ttf" Status:200 Latency:1.119363ms Bytes-In:0 Bytes-Out:19556
Request: [2022-12-16T11:49:26Z] Remote-IP:"10.42.0.1" Method:"GET" Path:"/pp/v1/epinio/rancher/dex/redirectUrl" Status:200 Latency:7.40377ms Bytes-In:0 Bytes-Out:510

And this is from dex pod:

time="2022-12-16T12:59:10Z" level=info msg="login successful: connector \"local\", username=\"admin\", preferred_username=\"\", email=\"[email protected]\", groups=[]"

@enrichman enrichman moved this from To Review to In Progress in Epinio planning Dec 19, 2022
@richard-cox
Copy link
Member Author

richard-cox commented Jan 4, 2023

Ok, image pull issues were due to a private personal repo.

Managed to recreate the issue, and also locally when serving the frontend files directly via the backend. Looks like there's a redirect that's required from verify-auth to auth/verify. Normally this is handled directly by Rancher, or from what i was seeing in the dev env that serves the UI.... neither of which was happening when deployed in epinio world

@thehejik
Copy link
Contributor

thehejik commented Jan 4, 2023

@richard-cox I'm sorry for the delay. Yes, I use my private dockerhub account for the dev ui image push. Then using modified epinio helm chart pointing to the dev image.

@richard-cox
Copy link
Member Author

@thehejik I've updated the charts and ui-backend PRs to redirect users directly to where they need to get back to. Hopefully this should resolve the issue, so ready for another test. Thanks!

@thehejik
Copy link
Contributor

thehejik commented Jan 4, 2023

@richard-cox thanks for the update.

On first attempt I was getting x509: certificate signed by unknown authority in epinio-ui pod logs so I had to set epinioAPISkipSSL=true.

On second attempt with disabled SSL there is no obvious problem with epinio-ui neither dex. I could enter credentials into DEX page and press Grant button but this is what I see in browser then:
image

Dex reports successful login when using incorporated simple-password (login: [email protected], password: password):

time="2023-01-04T17:18:31Z" level=info msg="login successful: connector \"local\", username=\"admin\", preferred_username=\"\", email=\"[email protected]\", groups=[]"

Full logs from epinio-ui pod with disabled SSL here

@richard-cox
Copy link
Member Author

@thehejik are there any errors in the browsers dev tools console? Could you also provide the output after enabling the dev tool setting Preferences --> Console --> Preserve log upon navigation? That should show entries starting Navigated to every time the user gets taken away from a location (epinio --> dex --> epinio), hopefully there's one showing the fix has been applied (Navigated to https://l..../auth/verify?code=... rather than Navigated to https://l..../verify-auth?code=...)

There's a similar setting to retain network requests (Network --> Preserve log). Is there anything suspicious there?

@thehejik
Copy link
Contributor

thehejik commented Jan 5, 2023

@richard-cox I see one error in console Failed to parse nonce, logs attached here.

Seems the fix has been applied, as there is Navigated to https://epinio.10.100.103.16.nip.io/auth/verify/

Nothing suspicious in the Network pane.

EDIT: The exact same error occurs when I rebase all 3 PRs on top of main/epinio-dev branches.

@richard-cox
Copy link
Member Author

Follow on work #176

@richard-cox
Copy link
Member Author

When the ui image is built, deployed and serves up the UI the dex redirect url (supplied by dex) misses both state and code params. When the ui is served up locally the dex redirect url contains both state and code. The code is the same so i don't think it's a setting.

@richard-cox
Copy link
Member Author

The issue wasn't related to dex or it's configuration. Something in the ui backend redirects (301 - moved) requests, in certain conditions, and in doing so strips query params. I spent a long time trying to track the cause down and failed, but appending a / to the original url seems to stop the behaviour

@richard-cox
Copy link
Member Author

richard-cox commented Jan 25, 2023

@thehejik I've pushed updates to the backend and helm chart PRs, that should resolve the issue. I've tracked some follow on work that should be done before we release the feature though - #180

An example epinio value.yaml

global:
  domain: '<snip>'
dex:
  enabled: true
  ui:
    secret: 'dev-secret1'
epinio-ui:
  epinioUI:
    apiSkipSSL: true
  dex:
    enabled: true
    ui:
      secret: "dev-secret1"

Update - I've fixed #180 via epinio/helm-charts@c4cf206

I've had success with the below yaml (and freshly built ui image).

global:
  domain: '<your domain'
epinio-ui:
  enabled: true
  epinioUI:
    theme: light
    version: "v1.6.1"
    apiSkipSSL: true
    image:
      registry: ghcr.io
      repository: <your repo>/epinio-ui
      tag: <your tag>
    imagePullPolicy: Always

@thehejik
Copy link
Contributor

thehejik commented Jan 27, 2023

It's working! I used the incorporated email account for dex login.

image

I didn't need to provide any special values.yaml (not even apiSkipSSL: true). I just rebased richard-cox:dex-ui-client helm-chart branch on top of origin/main branch where the frequent dex cert handling has been resolved.

@richard-cox could you please squash and rebase all 3 PRs on top of main branches? Then I will be happy to test it again and approve the PRs.

Command for installing:

helm upgrade --install epinio --namespace epinio ./chart/epinio/ --set skipTraefik=true --set global.domain=1.1.1.1.nip.io --create-namespace

@richard-cox
Copy link
Member Author

@thehejik I'd like to keep the commit's themselves in both UI and UI Backend worlds. Will bring them both up to date and squash+rebase the helm chart repo

@richard-cox
Copy link
Member Author

  1. Add support for Epinio's Dex Auth provider rancher/dashboard#7299 - No changes since rebase
  2. Add support for Epinio's Dex Auth Provider ui-backend#9 - Unrelated changes (.github/release-drafter.yml, .github/workflows/cleanup-images.yml
  3. Add Epinio UI Dex client helm-charts#295 - Lots of changes. Have rebased and tested (with the ui-backend image built previously)

I've re-deployed epinio using the rebased chart (using a ui-backend image previously built) and it looks alright.

@thehejik If you're happy could you give the green tick of goodness to helm-charts repo and I can get all three PRs in together

@richard-cox
Copy link
Member Author

All code / helm changes have been merged.

However a UI build will be needed to pick up the changes in an epinio installed via helm chart

@mmartin24
Copy link
Contributor

Checked after regenerating ui locally:
image

To be rechecked after merge of epinio/helm-charts#354

@mmartin24
Copy link
Contributor

This is already present in Epinio main stream (server version v1.6.2-53-g775b8167) after latest bumps.
Added automated UI tests [Cypress] Basic OIDC automation in CI and and working ok both in STD UI and Rancher Dashboard:

Closing issue

@github-project-automation github-project-automation bot moved this from In Progress to Done in Epinio planning Feb 21, 2023
@andreas-kupries andreas-kupries moved this from Done to Archivable in Epinio planning Mar 27, 2023
@richard-cox richard-cox moved this to Done in UI Apr 17, 2023
@richard-cox richard-cox added this to UI Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Status: Done
Development

No branches or pull requests

4 participants