Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] SearchIT.testExplainWithFetchSource failed #46224

Closed
henningandersen opened this issue Sep 2, 2019 · 2 comments
Closed

[CI] SearchIT.testExplainWithFetchSource failed #46224

henningandersen opened this issue Sep 2, 2019 · 2 comments
Assignees
@henningandersen
Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >test-failure Triaged test failures from CI

Comments

@henningandersen
Copy link
Contributor

Failed during a master intake build here:
https://gradle-enterprise.elastic.co/s/rhl5wvyx4q3gk/console-log?task=:client:rest-high-level:integTestRunner

org.elasticsearch.client.SearchIT > testExplainWithFetchSource FAILED
--
org.elasticsearch.client.ResponseException: method [PUT], host [http://127.0.0.1:34715], URI [/index2], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [index2/6n_jf67PR-mE8K9GDmOL0w] already exists","index_uuid":"6n_jf67PR-mE8K9GDmOL0w","index":"index2"}],"type":"resource_already_exists_exception","reason":"index [index2/6n_jf67PR-mE8K9GDmOL0w] already exists","index_uuid":"6n_jf67PR-mE8K9GDmOL0w","index":"index2"},"status":400}
at __randomizedtesting.SeedInfo.seed([59C5F5AAFE436E4E:4AC620DD1EBE9B5]:0)
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:221)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
at org.elasticsearch.client.SearchIT.indexDocuments(SearchIT.java:159)

It looks like this could affect all rest integration tests, since the wipe and recreate pattern is used everywhere. The relevant server log snippet is here:

(start of previous test: testNonExistentSearchTemplate)
[2019-09-02T08:47:32,674][INFO ][o.e.c.m.TemplateUpgradeService] [integTest-0] Starting template upgrade to version 8.0.0, 1 templates will be updated and 0 will be removed
[2019-09-02T08:47:32,678][INFO ][o.e.c.m.MetaDataIndexTemplateService] [integTest-0] adding template [.logstash-management] for index patterns [.logstash]
[2019-09-02T08:47:32,699][INFO ][o.e.c.m.TemplateUpgradeService] [integTest-0] Templates were upgraded successfully to version 8.0.0
[2019-09-02T08:47:32,723][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [integTest-0] adding index lifecycle policy [watch-history-ilm-policy]
[2019-09-02T08:47:32,763][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [integTest-0] adding index lifecycle policy [slm-history-ilm-policy]
[2019-09-02T08:47:32,841][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [index] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-09-02T08:47:32,934][INFO ][o.e.c.m.MetaDataMappingService] [integTest-0] [index/SdzF_EaURTK8LNWKxinhtQ] create_mapping [type]
[2019-09-02T08:47:32,989][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [index1] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-09-02T08:47:33,094][INFO ][o.e.c.m.MetaDataMappingService] [integTest-0] [index1/oAEk53QpRdiC0k0e6tSDFw] create_mapping [_doc]
[2019-09-02T08:47:33,131][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [index2] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[2019-09-02T08:47:33,241][INFO ][o.e.c.m.MetaDataMappingService] [integTest-0] [index2/Qh4tbqJKQh6C8GJxSMe1tQ] update_mapping [_doc]
[2019-09-02T08:47:33,274][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [index3] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-09-02T08:47:33,382][INFO ][o.e.c.m.MetaDataMappingService] [integTest-0] [index3/v6r9wYQlR6aHXDnZFQNLsA] create_mapping [_doc]
[2019-09-02T08:47:33,428][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [index4] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[2019-09-02T08:47:33,623][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [index1/oAEk53QpRdiC0k0e6tSDFw] deleting index
[2019-09-02T08:47:33,623][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [index4/nTGvrPFRQ92RCNvssR8KyQ] deleting index
[2019-09-02T08:47:33,623][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [index3/v6r9wYQlR6aHXDnZFQNLsA] deleting index
[2019-09-02T08:47:33,624][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [index2/Qh4tbqJKQh6C8GJxSMe1tQ] deleting index
[2019-09-02T08:47:33,624][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [index/SdzF_EaURTK8LNWKxinhtQ] deleting index
[2019-09-02T08:47:33,698][INFO ][o.e.c.m.MetaDataIndexTemplateService] [integTest-0] removing template [.logstash-management]
(start of failing test testExplainWithFetchSource)
[2019-09-02T08:47:33,720][INFO ][o.e.c.m.TemplateUpgradeService] [integTest-0] Starting template upgrade to version 8.0.0, 1 templates will be updated and 0 will be removed
[2019-09-02T08:47:33,725][INFO ][o.e.c.m.MetaDataIndexTemplateService] [integTest-0] adding template [.logstash-management] for index patterns [.logstash]
[2019-09-02T08:47:33,746][INFO ][o.e.c.m.TemplateUpgradeService] [integTest-0] Templates were upgraded successfully to version 8.0.0
[2019-09-02T08:47:33,769][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [integTest-0] adding index lifecycle policy [watch-history-ilm-policy]
[2019-09-02T08:47:33,810][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [integTest-0] adding index lifecycle policy [slm-history-ilm-policy]
[2019-09-02T08:47:33,859][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [index] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-09-02T08:47:33,957][INFO ][o.e.c.m.MetaDataMappingService] [integTest-0] [index/mTMwD3oDTK6Gtmf3lUaBjw] create_mapping [type]
[2019-09-02T08:47:34,012][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [index1] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-09-02T08:47:34,156][INFO ][o.e.c.m.MetaDataMappingService] [integTest-0] [index1/-28NAAelR_a_FFaj5-5A1A] create_mapping [_doc]
[2019-09-02T08:47:35,724][INFO ][o.e.c.m.MetaDataCreateIndexService] [integTest-0] [index2] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[2019-09-02T08:48:05,767][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [index/mTMwD3oDTK6Gtmf3lUaBjw] deleting index
[2019-09-02T08:48:05,768][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [index1/-28NAAelR_a_FFaj5-5A1A] deleting index
[2019-09-02T08:48:05,768][INFO ][o.e.c.m.MetaDataDeleteIndexService] [integTest-0] [index2/6n_jf67PR-mE8K9GDmOL0w] deleting index
[2019-09-02T08:48:11,116][INFO ][o.e.c.m.MetaDataIndexTemplateService] [integTest-0] removing template [.logstash-management]

It looks like there may be some kind of retry involved based on the pause between the create index log message and the deletion messages. Also, notice that the delete index2 request from the previous test has a different UUID, so from the looks of it, that test did clean up OK.

Unable to reproduce using either of:

./gradlew :client:rest-high-level:integTestRunner --tests "org.elasticsearch.client.SearchIT.testExplainWithFetchSource" -Dtests.seed=59C5F5AAFE436E4E -Dtests.security.manager=true -Dtests.locale=ln-CG -Dtests.timezone=Europe/Ulyanovsk -Dcompiler.java=12 -Druntime.java=11
./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=59C5F5AAFE436E4E -Dtests.security.manager=true -Dtests.locale=ln-CG -Dtests.timezone=Europe/Ulyanovsk -Dcompiler.java=12 -Druntime.java=11

Though this is a search test, the problem seems to fit within the distributed area, since it is the delete and create index part that fails.

Did not mute this test, since it seems like this could hit any rest test and this test was just unlucky.
@henningandersen henningandersen added >test-failure Triaged test failures from CI :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. labels Sep 2, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@henningandersen henningandersen mentioned this issue Sep 2, 2019
Closed
@henningandersen henningandersen self-assigned this Sep 4, 2019
@henningandersen
Copy link
Contributor Author

The symptom here is the same as in #46091, so closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@henningandersen henningandersen

Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@elasticmachine @henningandersen