ESS jobs stuck in RUNNING due to authentication issues on ESS-INT

[ds-alexander-bulgakov]

Description

ESS jobs can be run on ESS-INT which contains predefined testdata set. Currently all related tavern-tests fail because the jobs never COMPLETE and stuck in RUNNING. Logs were investigated and authentication issues detected as most probably reason.

Current behaviour

ESS jobs never complete and stuck in status RUNNING. In the logs a lot of authentication error messages could be found:
�[30m2024-01-25T03:14:42,191�[m �[32mINFO �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.e.t.i.r.d.DecentralDigitalTwinRegistryService�[m: Fetching shell(s) for 1 key(s) �[30m2024-01-25T03:14:42,192�[m �[32mINFO �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.e.t.i.r.d.DecentralDigitalTwinRegistryService�[m: Fetching 1 shells for bpn BPNL00ARBITRARY4 �[30m2024-01-25T03:14:42,192�[m �[32mINFO �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.e.t.i.r.d.ConnectorEndpointsService�[m: Requesting connector endpoints for BPN BPNL00ARBITRARY4 �[30m2024-01-25T03:14:42,192�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: HTTP POST https://irs-ess-discovery-mock.int.demo.catena-x.net/api/administration/connectors/discovery/search �[30m2024-01-25T03:14:42,192�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Accept=[application/json, application/*+json] �[30m2024-01-25T03:14:42,192�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Writing [DiscoveryFinderRequest[types=[bpn]]] with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter �[30m2024-01-25T03:14:42,192�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: HTTP POST https://centralidp.int.demo.catena-x.net/auth/realms/CX-Central/protocol/openid-connect/token �[30m2024-01-25T03:14:42,192�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Accept=[application/json, application/*+json] �[30m2024-01-25T03:14:42,192�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Writing [{grant_type=[client_credentials]}] as "application/x-www-form-urlencoded;charset=UTF-8" �[30m2024-01-25T03:14:42,193�[m �[32mINFO �[m[�[1;34mhttp-nio-8080-exec-2�[m] �[1;33mo.e.t.i.s.IrsItemGraphQueryService�[m: Found job with id 329dce5a-0af5-4f09-8f6d-073faab3b9b5 in status RUNNING with 0 relationships, 0 tombstones and 0 submodels �[30m2024-01-25T03:14:42,242�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Response 401 UNAUTHORIZED �[30m2024-01-25T03:14:52,243�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: HTTP POST https://irs-ess-discovery-mock.int.demo.catena-x.net/api/administration/connectors/discovery/search �[30m2024-01-25T03:14:52,244�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Accept=[application/json, application/*+json] �[30m2024-01-25T03:14:52,244�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Writing [DiscoveryFinderRequest[types=[bpn]]] with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter �[30m2024-01-25T03:14:52,244�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: HTTP POST https://centralidp.int.demo.catena-x.net/auth/realms/CX-Central/protocol/openid-connect/token �[30m2024-01-25T03:14:52,244�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Accept=[application/json, application/*+json] �[30m2024-01-25T03:14:52,244�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Writing [{grant_type=[client_credentials]}] as "application/x-www-form-urlencoded;charset=UTF-8" �[30m2024-01-25T03:14:52,259�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Response 401 UNAUTHORIZED �[30m2024-01-25T03:15:12,260�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: HTTP POST https://irs-ess-discovery-mock.int.demo.catena-x.net/api/administration/connectors/discovery/search �[30m2024-01-25T03:15:12,260�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Accept=[application/json, application/*+json] �[30m2024-01-25T03:15:12,260�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Writing [DiscoveryFinderRequest[types=[bpn]]] with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter �[30m2024-01-25T03:15:12,260�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: HTTP POST https://centralidp.int.demo.catena-x.net/auth/realms/CX-Central/protocol/openid-connect/token �[30m2024-01-25T03:15:12,260�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Accept=[application/json, application/*+json] �[30m2024-01-25T03:15:12,260�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Writing [{grant_type=[client_credentials]}] as "application/x-www-form-urlencoded;charset=UTF-8" �[30m2024-01-25T03:15:12,297�[m �[36mDEBUG �[m[�[1;34mpool-4-thread-272�[m] �[1;33mo.s.c.l.CompositeLog�[m: Response 401 UNAUTHORIZED Exception in thread "pool-4-thread-272" org.springframework.security.oauth2.client.ClientAuthorizationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: 401 Unauthorized: [no body] at org.springframework.security.oauth2.client.ClientCredentialsOAuth2AuthorizedClientProvider.getTokenResponse(ClientCredentialsOAuth2AuthorizedClientProvider.java:96) ... Caused by: org.springframework.security.oauth2.core.OAuth2AuthorizationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: 401 Unauthorized: [no body] at org.springframework.security.oauth2.client.endpoint.DefaultClientCredentialsTokenResponseClient.getResponse(DefaultClientCredentialsTokenResponseClient.java:96) ... Caused by: org.springframework.web.client.HttpClientErrorException$Unauthorized: 401 Unauthorized: [no body] at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:106) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137) ...

Expected behavior

ESS jobs can be run on ESS-INT with expected result.

Steps to reproduce the Bug

Can be reproduced with requesting ESS-INT jobs.

[ds-alexander-bulgakov]

Hi @jzbmw,
FYI this bug ticket as announced in the todays daily for further prioritizations.
Thank you!

[mkanal]

Wontfix already resolved