[CI] XPackRestIT test {p0=ml/forecast/Test forecast unknown job} failing #116150

elasticsearchmachine · 2024-11-03T14:09:08Z

Build Scans:

Reproduction Line:

./gradlew ":x-pack:plugin:yamlRestTest" --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/forecast/Test forecast unknown job}" -Dtests.seed=F1BDE02137EABDC7 -Dtests.locale=smn -Dtests.timezone=Europe/Uzhgorod -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.junit.TestCouldNotBeSkippedException: Test could not be skipped due to other failures

Issue Reasons:

[main] 2 consecutive failures in test test {p0=ml/forecast/Test forecast unknown job}
[main] 2 failures in test test {p0=ml/forecast/Test forecast unknown job} (100.0% fail rate in 2 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

The text was updated successfully, but these errors were encountered:

…st/Test forecast unknown job} #116150

elasticsearchmachine · 2024-11-03T14:09:11Z

This has been muted on branch main

Mute Reasons:

[main] 2 consecutive failures in test test {p0=ml/forecast/Test forecast unknown job}
[main] 2 failures in test test {p0=ml/forecast/Test forecast unknown job} (100.0% fail rate in 2 executions)

Build Scans:

elasticsearchmachine · 2024-11-03T14:09:32Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2024-11-04T12:06:03Z

Pinging @elastic/ml-core (Team:ML)

davidkyle · 2024-11-04T12:17:42Z

The failure is due to an assertion in the logs taking down the node

[2024-11-01T02:46:22,021][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [yamlRestTest-0] fatal error in thread [elasticsearch[yamlRestTest-0][system_critical_write][T#3]], exiting
java.lang.AssertionError: null
	at org.elasticsearch.index.mapper.IgnoredSourceFieldMapper.postParse(IgnoredSourceFieldMapper.java:161) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:190) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:136) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:113) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1043) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:984) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:928) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:378) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:237) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:305) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:153) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:80) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:220) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1575) ~[?:?]

yamlRestTest.log

elasticsearchmachine · 2024-11-04T12:18:50Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

kkrik-es · 2024-11-04T12:54:04Z

@davidkyle thanks for looking. The stack trace above seems like an issue with synthetic source indeed.

I just synced and can't reproduce the issue. Did you just use the command above, in main? If it doesn't reproduce any more, I'm tempted to unmute and see if it'll come back.

kkrik-es · 2024-11-04T13:02:26Z

Btw the first failure link above points to a different error:

REPRODUCE WITH: ./gradlew ":x-pack:plugin:yamlRestTest" --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/forecast/Test forecast unknown job}" -Dtests.seed=F1BDE02137EABDC7 -Dtests.locale=smn -Dtests.timezone=Europe/Uzhgorod -Druntime.java=23

XPackRestIT > test {p0=ml/forecast/Test forecast unknown job} FAILED
    org.junit.TestCouldNotBeSkippedException: Test could not be skipped due to other failures
        at org.junit.runners.model.MultipleFailureException.<init>(MultipleFailureException.java:36)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1014)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1575)

        Caused by:
        org.junit.AssumptionViolatedException: [ml/forecast/Test forecast unknown job] skipped, reason: [https://github.com/elastic/elasticsearch/issues/34747]

The second link is from #116049 that has changes for synthetic source, so it's irrelevant.

…st/Test forecast unknown job} elastic#116150

kkrik-es · 2024-11-05T08:41:24Z

Assigning back to @davidkyle since this doesn't seem like an issue with synthetic source. Please assign back to me if a failure outside a PR or another repro pointing to a parsing exception.

davidkyle · 2024-11-05T10:33:50Z

Thanks for the investigation @kkrik-es

davidkyle · 2024-11-05T10:43:40Z

The failing test is actually muted, the TestCouldNotBeSkippedException means that the error occurred either in the test setup or teardown. In this case it is a search_phase_execution_exception in the teardown.

    org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:35139], URI [/_ml/trained_models/_stats?size=10000], status line [HTTP/1.1 500 Internal Server Error]	
    {"error":{"root_cause":[],"type":"exception","reason":"Searching for stats for models [lang_ident_model_1] failed","caused_by":{"type":"search_phase_execution_exception","reason":"","phase":"query","grouped":true,"failed_shards":[],"caused_by":{"type":"search_phase_execution_exception","reason":"Search rejected due to missing shards [[.ml-stats-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.","phase":"query","grouped":true,"failed_shards":[]}}},"status":500}	
        at app//org.elasticsearch.client.RestClient.convertResponse(RestClient.java:351)	
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:317)	
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)	
        at app//org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.deleteAllTrainedModelIngestPipelines(MlRestTestStateCleaner.java:43)	
        at app//org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.resetFeatures(MlRestTestStateCleaner.java:34)	
        at app//org.elasticsearch.xpack.test.rest.AbstractXPackRestTest.clearMlState(AbstractXPackRestTest.java:138)	
        at app//org.elasticsearch.xpack.test.rest.AbstractXPackRestTest.cleanup(AbstractXPackRestTest.java:118)

elasticsearchmachine added :Search Relevance/Ranking Scoring, rescoring, rank evaluation. >test-failure Triaged test failures from CI labels Nov 3, 2024

elasticsearchmachine added a commit that referenced this issue Nov 3, 2024

Mute org.elasticsearch.xpack.test.rest.XPackRestIT test {p0=ml/foreca…

82b3b4d

…st/Test forecast unknown job} #116150

elasticsearchmachine added needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Nov 3, 2024

davidkyle added :ml Machine learning and removed :Search Relevance/Ranking Scoring, rescoring, rank evaluation. labels Nov 4, 2024

elasticsearchmachine added the Team:ML Meta label for the ML team label Nov 4, 2024

elasticsearchmachine removed the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Nov 4, 2024

davidkyle added :StorageEngine/Logs You know, for Logs and removed :ml Machine learning Team:ML Meta label for the ML team labels Nov 4, 2024

elasticsearchmachine added the Team:StorageEngine label Nov 4, 2024

kkrik-es added :ml Machine learning Team:ML Meta label for the ML team Team:StorageEngine :StorageEngine/Mapping The storage related side of mappings and removed Team:StorageEngine :StorageEngine/Logs You know, for Logs :ml Machine learning Team:ML Meta label for the ML team labels Nov 4, 2024

kkrik-es self-assigned this Nov 4, 2024

kkrik-es assigned davidkyle Nov 4, 2024

jfreden pushed a commit to jfreden/elasticsearch that referenced this issue Nov 4, 2024

Mute org.elasticsearch.xpack.test.rest.XPackRestIT test {p0=ml/foreca…

2dc5820

…st/Test forecast unknown job} elastic#116150

kkrik-es removed their assignment Nov 5, 2024

kkrik-es added :ml Machine learning Team:ML Meta label for the ML team and removed Team:StorageEngine :StorageEngine/Mapping The storage related side of mappings labels Nov 5, 2024

davidkyle added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Nov 5, 2024

davidkyle mentioned this issue Nov 8, 2024

[ML] Avoid the .ml-stats index in post test cleanup #116476

Merged

kkrik-es closed this as completed Nov 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] XPackRestIT test {p0=ml/forecast/Test forecast unknown job} failing #116150

[CI] XPackRestIT test {p0=ml/forecast/Test forecast unknown job} failing #116150

elasticsearchmachine commented Nov 3, 2024

elasticsearchmachine commented Nov 3, 2024

elasticsearchmachine commented Nov 3, 2024

elasticsearchmachine commented Nov 4, 2024

davidkyle commented Nov 4, 2024

elasticsearchmachine commented Nov 4, 2024

kkrik-es commented Nov 4, 2024

kkrik-es commented Nov 4, 2024

kkrik-es commented Nov 5, 2024

davidkyle commented Nov 5, 2024

davidkyle commented Nov 5, 2024

[CI] XPackRestIT test {p0=ml/forecast/Test forecast unknown job} failing #116150

[CI] XPackRestIT test {p0=ml/forecast/Test forecast unknown job} failing #116150

Comments

elasticsearchmachine commented Nov 3, 2024

elasticsearchmachine commented Nov 3, 2024

elasticsearchmachine commented Nov 3, 2024

elasticsearchmachine commented Nov 4, 2024

davidkyle commented Nov 4, 2024

elasticsearchmachine commented Nov 4, 2024

kkrik-es commented Nov 4, 2024

kkrik-es commented Nov 4, 2024

kkrik-es commented Nov 5, 2024

davidkyle commented Nov 5, 2024

davidkyle commented Nov 5, 2024