Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SAML IDP Provider metadata site down causes exceptions #1531

Closed
ChrisMcGowan opened this issue Mar 5, 2021 · 3 comments · Fixed by #1563
Closed

SAML IDP Provider metadata site down causes exceptions #1531

ChrisMcGowan opened this issue Mar 5, 2021 · 3 comments · Fixed by #1563
Labels
accepted Accepted the issue

Comments

@ChrisMcGowan
Copy link

SECURITY NOTICE: If you have found a security problem in the UAA, please do not file a public github issue. Instead, please send an email to [email protected]

Thanks for taking the time to file an issue. You'll minimize back and forth and help us help you more effectively by answering all of the following questions as specifically and completely as you can.

What version of UAA are you running?

What output do you see from curl <YOUR_UAA>/info -H'Accept: application/json'? 75.0.0

How are you deploying the UAA?

I am deploying the UAA

  • using cf-deployment

What did you do?

We utilize UAA with SAML and IDP providers to allow other agencies sign into the platform. We currently have 6 IDP providers in our SAML providers config

What did you expect to see? What goal are you trying to achieve with the UAA?

By using idpMetadata links inside of static metadata, we hope to perform less UAA administration as our customers make changes to their IDP configurations. If the customer's metadata site was/is down, UAA would log a message and continue to work using the last known values for the metadata.

What did you see instead?

One of the 6 IDP providers metadata site was down and during a refresh, UAA would time out contacting that provider via the idpMetadata URL. That timeout caused WARN message and spring exception and UAA would go down and restart. This process kept repeating itself until the site for the metadata came back online. This would happen across all the UAA VMs and effect all users with slow logins or time outs trying to login even using the other IDP providers that where online and/or provided a static metadata config. The provider that was down, was down for almost 9 hours

Please include UAA logs if available.

IDP URL renamed and IP of IDP provider changed to 0.0.0.0

[2021-03-05T12:59:52.389571Z] uaa - 15 [https-jsse-nio-8443-exec-27] ....  WARN --- ExpiringUrlCache: Unable to fetch metadata for {0}. {1}
[2021-03-05T12:59:52.389644Z] uaa - 15 [https-jsse-nio-8443-exec-27] .... ERROR --- NonSnarlMetadataManager: Invalid SAML IDP zone[uaa] alias[someidp-provider.gov]
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://someidp-provider.gov:443/federationmetadata/2007-06/federationmetadata.xml": Connect to someidp-provider.gov:443 [someidp-provider.gov/0.0.0.0] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to someidp-provider.gov:443 [someidp-provider.gov/0.0.0.0] failed: connect timed out
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:784) ~[spring-web-5.3.3.jar:5.3.3]
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:750) ~[spring-web-5.3.3.jar:5.3.3]
	at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:351) ~[spring-web-5.3.3.jar:5.3.3]
	at org.cloudfoundry.identity.uaa.cache.ExpiringUrlCache.getUrlContent(ExpiringUrlCache.java:73) ~[cloudfoundry-identity-server-75.0.0.jar:?]
	at org.cloudfoundry.identity.uaa.cache.ExpiringUrlCache.getUrlContent(ExpiringUrlCache.java:54) ~[cloudfoundry-identity-server-75.0.0.jar:?]
	at org.cloudfoundry.identity.uaa.provider.saml.FixedHttpMetaDataProvider.fetchMetadata(FixedHttpMetaDataProvider.java:29) ~[cloudfoundry-identity-server-75.0.0.jar:?]
	at org.cloudfoundry.identity.uaa.provider.saml.SamlIdentityProviderConfigurator.configureURLMetadata(SamlIdentityProviderConfigurator.java:165) ~[cloudfoundry-identity-server-75.0.0.jar:?]
	at org.cloudfoundry.identity.uaa.provider.saml.SamlIdentityProviderConfigurator.getExtendedMetadataDelegate(SamlIdentityProviderConfigurator.java:122) ~[cloudfoundry-identity-server-75.0.0.jar:?]
	at org.cloudfoundry.identity.uaa.provider.saml.NonSnarlMetadataManager.getAvailableProviders(NonSnarlMetadataManager.java:149) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.cloudfoundry.identity.uaa.provider.saml.NonSnarlMetadataManager.getHostedSPName(NonSnarlMetadataManager.java:430) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.security.saml.metadata.MetadataGeneratorFilter.processMetadataInitialization(MetadataGeneratorFilter.java:100) [spring-security-saml2-core-1.0.10.RELEASE.jar:1.0.10.RELEASE]
	at org.springframework.security.saml.metadata.MetadataGeneratorFilter.doFilter(MetadataGeneratorFilter.java:86) [spring-security-saml2-core-1.0.10.RELEASE.jar:1.0.10.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.cloudfoundry.identity.uaa.security.web.SecurityFilterChainPostProcessor$UaaLoggingFilter.doFilter(SecurityFilterChainPostProcessor.java:259) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.cloudfoundry.identity.uaa.security.web.SecurityFilterChainPostProcessor$HttpsEnforcementFilter.doFilter(SecurityFilterChainPostProcessor.java:202) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.cloudfoundry.identity.uaa.oauth.DisableIdTokenResponseTypeFilter.doFilterInternal(DisableIdTokenResponseTypeFilter.java:86) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.cloudfoundry.identity.uaa.zone.IdentityZoneResolvingFilter.doFilterInternal(IdentityZoneResolvingFilter.java:78) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.cloudfoundry.identity.uaa.web.LimitedModeUaaFilter.doFilterInternal(LimitedModeUaaFilter.java:73) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.cloudfoundry.identity.uaa.security.web.CorsFilter.doFilterInternal(CorsFilter.java:137) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.cloudfoundry.identity.uaa.authentication.UTF8ConversionFilter.validateParamsAndContinue(UTF8ConversionFilter.java:75) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.cloudfoundry.identity.uaa.authentication.UTF8ConversionFilter.doFilter(UTF8ConversionFilter.java:59) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.cloudfoundry.identity.uaa.web.HeaderFilter.doFilter(HeaderFilter.java:52) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.cloudfoundry.identity.uaa.metrics.UaaMetricsFilter.doFilterInternal(UaaMetricsFilter.java:84) [cloudfoundry-identity-server-75.0.0.jar:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.2.jar:5.4.2]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) [spring-security-web-5.4.2.jar:5.4.2]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) [spring-security-web-5.4.2.jar:5.4.2]
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.3.3.jar:5.3.3]
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.3.3.jar:5.3.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.3.jar:5.3.3]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) [spring-session-core-2.4.1.jar:2.4.1]
	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) [spring-session-core-2.4.1.jar:2.4.1]
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.3.3.jar:5.3.3]
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.3.3.jar:5.3.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
	at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) [catalina.jar:9.0.41]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [catalina.jar:9.0.41]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [catalina.jar:9.0.41]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [catalina.jar:9.0.41]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [catalina.jar:9.0.41]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [catalina.jar:9.0.41]
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) [catalina.jar:9.0.41]
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764) [catalina.jar:9.0.41]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [catalina.jar:9.0.41]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [catalina.jar:9.0.41]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-coyote.jar:9.0.41]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-coyote.jar:9.0.41]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888) [tomcat-coyote.jar:9.0.41]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597) [tomcat-coyote.jar:9.0.41]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:9.0.41]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:9.0.41]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to someidp-provider.gov:443 [someidp-provider.gov/0.0.0.0] failed: connect timed out
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
	at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web-5.3.3.jar:5.3.3]
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.3.3.jar:5.3.3]
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-5.3.3.jar:5.3.3]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:775) ~[spring-web-5.3.3.jar:5.3.3]
	... 72 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
	at java.net.Socket.connect(Socket.java:609) ~[?:?]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
	at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web-5.3.3.jar:5.3.3]
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.3.3.jar:5.3.3]
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-5.3.3.jar:5.3.3]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:775) ~[spring-web-5.3.3.jar:5.3.3]
	... 72 more
@cf-gitbot
Copy link

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/177224767

The labels on this github issue will be updated when the story is started.

@ChrisMcGowan
Copy link
Author

Any updates or guidance on this issue ?

Thanks

@spgreenberg spgreenberg mentioned this issue Mar 16, 2021
Closed
@spgreenberg spgreenberg mentioned this issue Apr 26, 2021
Merged
@spgreenberg
Copy link
Contributor

PR opened: #1563

@strehle strehle linked a pull request Jul 1, 2021 that will close this issue
Return existing records on update failure #1563
Merged
@strehle strehle mentioned this issue Sep 22, 2022
Open
@cf-gitbot cf-gitbot added delivered accepted Accepted the issue and removed delivered labels Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
accepted Accepted the issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Return existing records on update failure
3 participants
@spgreenberg @cf-gitbot @ChrisMcGowan