Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] testKerbTicketGeneratedForDifferentServerFailsValidation fails on 6.5 #35982

Closed
jkakavas opened this issue Nov 28, 2018 · 3 comments · Fixed by #39221
Closed

[CI] testKerbTicketGeneratedForDifferentServerFailsValidation fails on 6.5 #35982

jkakavas opened this issue Nov 28, 2018 · 3 comments · Fixed by #39221
Assignees
@bizybot
Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI

Comments

@jkakavas
Copy link
Member

Build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+multijob-darwin-compatibility/67/console

failed when KDC failed to connect to the in memory ldap server. I can't reproduce this locally so it could be a transient error but I'm opening this so that we take a look at least. Initially, it felt this has something to do with #35764 but this wasn't backported to 6.5 so it should be irrelevant.

Stacktrace:

java.security.PrivilegedActionException: org.apache.kerby.kerberos.kerb.KrbException: Failed to start connection with LDAP
	at __randomizedtesting.SeedInfo.seed([5AC16FEBD70AFBA8:B0A2C0EE71270FF]:0)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer.<init>(SimpleKdcLdapServer.java:86)
	at org.elasticsearch.xpack.security.authc.kerberos.KerberosTestCase.startSimpleKdcLdapServer(KerberosTestCase.java:111)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kerby.kerberos.kerb.KrbException: Failed to start connection with LDAP
	at org.apache.kerby.kerberos.kdc.identitybackend.LdapIdentityBackend.doInitialize(LdapIdentityBackend.java:114)
	at org.apache.kerby.kerberos.kerb.identity.backend.AbstractIdentityBackend.initialize(AbstractIdentityBackend.java:67)
	at org.apache.kerby.kerberos.kerb.server.KdcUtil.getBackend(KdcUtil.java:115)
	at org.apache.kerby.kerberos.kerb.server.impl.AbstractInternalKdcServer.init(AbstractInternalKdcServer.java:65)
	at org.apache.kerby.kerberos.kerb.server.KdcServer.init(KdcServer.java:256)
	at org.apache.kerby.kerberos.kerb.server.SimpleKdcServer.init(SimpleKdcServer.java:155)
	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer.prepareKdcServerAndStart(SimpleKdcLdapServer.java:150)
	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer.init(SimpleKdcLdapServer.java:104)
	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer.access$000(SimpleKdcLdapServer.java:39)
	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer$2.run(SimpleKdcLdapServer.java:89)
	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer$2.run(SimpleKdcLdapServer.java:86)
	... 39 more
Caused by: org.apache.directory.api.ldap.model.exception.LdapException: The response queue has been emptied, no response was found.
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1327)
	at org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:130)
	at org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:114)
	at org.apache.kerby.kerberos.kdc.identitybackend.LdapIdentityBackend.startConnection(LdapIdentityBackend.java:100)
	at org.apache.kerby.kerberos.kdc.identitybackend.LdapIdentityBackend.doInitialize(LdapIdentityBackend.java:111)
	... 49 more
Caused by: org.apache.directory.api.ldap.model.exception.LdapException: TimeOut occurred
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1305)
	... 53 more
@jkakavas jkakavas added >test-failure Triaged test failures from CI :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) labels Nov 28, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@jaymode
Copy link
Member

jaymode commented Jan 10, 2019

@bizybot what is the status here?

@bizybot
Copy link
Contributor

bizybot commented Feb 12, 2019

I looked at the CI failures since 28th November and there is no other build where this has failed.
I will try to spend some time again and see if I can reproduce this.

bizybot pushed a commit to bizybot/elasticsearch that referenced this issue Feb 21, 2019
Add await busy loop for SimpleKdcLdapServer initialization
e14bb2a 
There have been intermittent failures where either
ldap server could not be started or kdc server could
not be started causing failures during test runs.

`KdcNetwork` class from Apache kerby project does not set reuse
address to `true` on socket so if the port that we found to be free
is in `TIME_WAIT` state it may fail to bind. As this is internal
class for kerby, I could not find a way to extend.

This commit adds a retry loop for initialization. It will keep
trying in a await busy loop and fail after 10 seconds if not
initialized.

Closes elastic#35982
@bizybot bizybot mentioned this issue Feb 21, 2019
Merged
bizybot added a commit to bizybot/elasticsearch that referenced this issue Feb 25, 2019
@bizybot
Add await busy loop for SimpleKdcLdapServer initialization (elastic#3…
ed20d7c 
…9221)

There have been intermittent failures where either
LDAP server could not be started or KDC server could
not be started causing failures during test runs.

`KdcNetwork` class from Apache kerby project does not set reuse
address to `true` on the socket so if the port that we found to be free
is in `TIME_WAIT` state it may fail to bind. As this is an internal
class for kerby, I could not find a way to extend.

This commit adds a retry loop for initialization. It will keep
trying in an await busy loop and fail after 10 seconds if not
initialized.

Closes elastic#35982
bizybot added a commit to bizybot/elasticsearch that referenced this issue Feb 25, 2019
@bizybot
Add await busy loop for SimpleKdcLdapServer initialization (elastic#3…
6ef7098 
…9221)

There have been intermittent failures where either
LDAP server could not be started or KDC server could
not be started causing failures during test runs.

`KdcNetwork` class from Apache kerby project does not set reuse
address to `true` on the socket so if the port that we found to be free
is in `TIME_WAIT` state it may fail to bind. As this is an internal
class for kerby, I could not find a way to extend.

This commit adds a retry loop for initialization. It will keep
trying in an await busy loop and fail after 10 seconds if not
initialized.

Closes elastic#35982
@bizybot bizybot mentioned this issue Feb 25, 2019
Merged
bizybot added a commit to bizybot/elasticsearch that referenced this issue Feb 25, 2019
@bizybot
Add await busy loop for SimpleKdcLdapServer initialization (elastic#3…
63a5583 
…9221)

There have been intermittent failures where either
LDAP server could not be started or KDC server could
not be started causing failures during test runs.

`KdcNetwork` class from Apache kerby project does not set reuse
address to `true` on the socket so if the port that we found to be free
is in `TIME_WAIT` state it may fail to bind. As this is an internal
class for kerby, I could not find a way to extend.

This commit adds a retry loop for initialization. It will keep
trying in an await busy loop and fail after 10 seconds if not
initialized.

Closes elastic#35982
@bizybot bizybot mentioned this issue Feb 25, 2019
Merged
bizybot added a commit to bizybot/elasticsearch that referenced this issue Feb 25, 2019
@bizybot
Add await busy loop for SimpleKdcLdapServer initialization (elastic#3…
c490f02 
…9221)

There have been intermittent failures where either
LDAP server could not be started or KDC server could
not be started causing failures during test runs.

`KdcNetwork` class from Apache kerby project does not set reuse
address to `true` on the socket so if the port that we found to be free
is in `TIME_WAIT` state it may fail to bind. As this is an internal
class for kerby, I could not find a way to extend.

This commit adds a retry loop for initialization. It will keep
trying in an await busy loop and fail after 10 seconds if not
initialized.

Closes elastic#35982
@bizybot bizybot mentioned this issue Feb 25, 2019
Merged
bizybot added a commit that referenced this issue Feb 25, 2019
@bizybot
Add await busy loop for SimpleKdcLdapServer initialization (#39221) (#…
802d44f 
…39344)

There have been intermittent failures where either
LDAP server could not be started or KDC server could
not be started causing failures during test runs.

`KdcNetwork` class from Apache kerby project does not set reuse
address to `true` on the socket so if the port that we found to be free
is in `TIME_WAIT` state it may fail to bind. As this is an internal
class for kerby, I could not find a way to extend.

This commit adds a retry loop for initialization. It will keep
trying in an await busy loop and fail after 10 seconds if not
initialized.

Closes #35982
bizybot added a commit that referenced this issue Feb 25, 2019
@bizybot
Add await busy loop for SimpleKdcLdapServer initialization (#39221) (#…
851fb5c 
…39343)

There have been intermittent failures where either
LDAP server could not be started or KDC server could
not be started causing failures during test runs.

`KdcNetwork` class from Apache kerby project does not set reuse
address to `true` on the socket so if the port that we found to be free
is in `TIME_WAIT` state it may fail to bind. As this is an internal
class for kerby, I could not find a way to extend.

This commit adds a retry loop for initialization. It will keep
trying in an await busy loop and fail after 10 seconds if not
initialized.

Closes #35982
bizybot added a commit that referenced this issue Feb 25, 2019
@bizybot
Add await busy loop for SimpleKdcLdapServer initialization (#39221) (#…
7021e1b 
…39342)

There have been intermittent failures where either
LDAP server could not be started or KDC server could
not be started causing failures during test runs.

`KdcNetwork` class from Apache kerby project does not set reuse
address to `true` on the socket so if the port that we found to be free
is in `TIME_WAIT` state it may fail to bind. As this is an internal
class for kerby, I could not find a way to extend.

This commit adds a retry loop for initialization. It will keep
trying in an await busy loop and fail after 10 seconds if not
initialized.

Closes #35982
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bizybot bizybot

Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add await busy loop for SimpleKdcLdapServer initialization bizybot/elasticsearch
4 participants
@bizybot @jaymode @jkakavas @elasticmachine