[CI] ES|QL TelemetryIT.testMetrics failure

[thecoop]

CI Link

https://gradle-enterprise.elastic.co/s/xrzv7jdcyi4ky/console-log/raw

Repro line

./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.TelemetryIT.testMetrics"

Does it reproduce?

Yes

Applicable branches

main

Failure history

No response

Failure excerpt

1> [2024-10-31T08:08:55,817][INFO ][o.e.x.e.a.TelemetryIT    ] [testMetrics] [TelemetryIT#testMetrics {TestCase=Test[query=FROM idx
  1> | EVAL ip = to_ip(host), x = to_string(host), y = to_string(host)
  1> | INLINESTATS max(id)
  1> , expectedCommands={INLINESTATS=1, STATS=1, FROM=1, EVAL=1}, expectedFunctions={TO_STRING=2, MAX=1, TO_IP=1}, success=true]}]: cleaned up after test
  1> [2024-10-31T08:08:55,818][INFO ][o.e.x.e.a.TelemetryIT    ] [testMetrics] [TestCase=Test[query=FROM idx
  1> | EVAL ip = to_ip(host), x = to_string(host), y = to_string(host)
  1> | INLINESTATS max(id)
  1> , expectedCommands={INLINESTATS=1, STATS=1, FROM=1, EVAL=1}, expectedFunctions={TO_STRING=2, MAX=1, TO_IP=1}, success=true]] after test
  2> REPRODUCE WITH: ./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.TelemetryIT.testMetrics {TestCase=Test[query=FROM idx
  2> | EVAL ip = to_ip(host), x = to_string(host), y = to_string(host)
  2> | INLINESTATS max(id)
  2> , expectedCommands={INLINESTATS=1, STATS=1, FROM=1, EVAL=1}, expectedFunctions={TO_STRING=2, MAX=1, TO_IP=1}, success=true]}" -Dtests.seed=2C24155C2F138C18 -Dtests.locale=es-PH -Dtests.timezone=America/Miquelon -Druntime.java=23
  2> REPRODUCE WITH: ./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.TelemetryIT.testMetrics {TestCase=Test[query=FROM idx
  2> | EVAL ip = to_ip(host), x = to_string(host), y = to_string(host)
  2> | INLINESTATS max(id)
  2> , expectedCommands={INLINESTATS=1, STATS=1, FROM=1, EVAL=1}, expectedFunctions={TO_STRING=2, MAX=1, TO_IP=1}, success=true]}" -Dtests.seed=2C24155C2F138C18 -Dtests.locale=es-PH -Dtests.timezone=America/Miquelon -Druntime.java=23
  2> REPRODUCE WITH: ./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.TelemetryIT.testMetrics {TestCase=Test[query=FROM idx
  2> | EVAL ip = to_ip(host), x = to_string(host), y = to_string(host)
  2> | INLINESTATS max(id)
  2> , expectedCommands={INLINESTATS=1, STATS=1, FROM=1, EVAL=1}, expectedFunctions={TO_STRING=2, MAX=1, TO_IP=1}, success=true]}" -Dtests.seed=2C24155C2F138C18 -Dtests.locale=es-PH -Dtests.timezone=America/Miquelon -Druntime.java=23
  2> java.lang.AssertionError: 
    Expected: is <true>
         but: was <false>
        at __randomizedtesting.SeedInfo.seed([2C24155C2F138C18]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
        at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2543)
        at org.elasticsearch.xpack.esql.action.TelemetryIT.assertAllUsages(TelemetryIT.java:217)
        at org.elasticsearch.xpack.esql.action.TelemetryIT.lambda$testQuery$0(TelemetryIT.java:188)

    java.lang.AssertionError: 
    Expected: is <true>
         but: was <false>
        at __randomizedtesting.SeedInfo.seed([2C24155C2F138C18]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
        at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2543)
        at org.elasticsearch.xpack.esql.action.TelemetryIT.assertAllUsages(TelemetryIT.java:217)
        at org.elasticsearch.xpack.esql.action.TelemetryIT.lambda$testQuery$0(TelemetryIT.java:188)

    java.lang.AssertionError: 
    Expected: is <true>
         but: was <false>
        at __randomizedtesting.SeedInfo.seed([2C24155C2F138C18]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
        at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2543)
        at org.elasticsearch.xpack.esql.action.TelemetryIT.assertAllUsages(TelemetryIT.java:217)
        at org.elasticsearch.xpack.esql.action.TelemetryIT.lambda$testQuery$0(TelemetryIT.java:188)

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[luigidellaquila]

I think it's related to #115813

The underlying error is a query planning problem:

java.lang.IllegalStateException: can't find input for [max(id){r}#464]

the query is

FROM idx
| EVAL ip = to_ip(host), x = to_string(host), y = to_string(host)
| INLINESTATS max(id)

The failure is quite deterministic, I'll disable that specific query from the telemetry tests to avoid too much noise.

@costin maybe you have a clue on the root cause, this test never failed before AFAIK

[luigidellaquila]

Related to #110923

[costin]

Thanks! Needs more investigation since as you said, this test passed so far...

[costin]

Backport to 8.x at #116045

[alex-spies]

This still fails (when unmuted) with current main. We currently do not work on INLINESTATS and the current work on JOINs will likely put INLINESTATS in an even worse state before we come back to it and finish it.

That is, until we pick up work on INLINESTATS again, this has to remain muted and this test failure cannot be worked on.

[nik9000]

I think we can drop the usage of INLINESTATS in the test, right?

[alex-spies]

That's already done in Luigi's mute.

Since the CI failure itself is resolved and we currently don't work on INLINESTATS, maybe it's better to add this failed query to the INLINESTATS follow-up meta issue than having a separate issue for it that'll stay open for a while.

[alex-spies]

Moved the corresponding work on INLINESTATS into its meta issue.