ESQL: ClassCastException in stats group by field that is missing in some indices

[craigtaverner]

Description

The following query:

FROM logs-* | STATS count=count(user_agent.name) BY user_agent.name

yields this error:

class org.elasticsearch.compute.data.ConstantNullBlock cannot be cast to 
    class org.elasticsearch.compute.data.BytesRefBlock
(org.elasticsearch.compute.data.ConstantNullBlock and org.elasticsearch.compute.data.BytesRefBlock are in unnamed module of loader java.net.FactoryURLClassLoader @15a8cebd)

The server logs claim the exception is thrown on line 56 of BytesRefBlockHash.java.

Data

The data used for this is the elastic/logs track in benchmarks, where many benchmark queries look at all indices, using GET /logs-*/_search, so I expect ESQL queries on logs-* should also work.

Further investigation of the data:

• Omitting the group by FROM logs-* | STATS count=count(user_agent.name) works and returns a count of 55494, which matches the number of non-null user agent names (a bit over 10%, see below)
• Omitting the aggregating function FROM logs-* | STATS by user_agent.name returns a different error ValuesSources are mismatched hinting that the issue might relate to different mappings in the different indices.
• Searching for values of user_agent.name that are not null from logs-* | keep user_agent.name | WHERE user_agent.name IS NOT NULL | LIMIT 20 yields many values like Chrome, Go-http-client, etc.
• Doing the stats with a non-null predicate also works

FROM logs-* 
| WHERE user_agent.name IS NOT NULL 
| STATS count=count(user_agent.name) BY user_agent.name 
| SORT count DESC

returns:

count,user_agent.name
33896,Other
10697,Go-http-client
6547,Chrome
1245,Firefox
993,Apache-HttpClient
658,Java
547,Safari
440,Opera
122,LINE
105,aws-sdk-nodejs
90,curl
76,IE
33,Edge
17,Zune
9,Chrome Frame
5,WebKit Nightly
5,BountyBot
3,Mobile Safari
3,Slackbot
1,Android
1,Nimbostratus-Bot
1,Chrome Mobile iOS

Counting how many are NULL vs NOT NULL:

• FROM logs-* | WHERE user_agent.name IS NOT NULL | STATS count(@timestamp) -> 406506
• FROM logs-* | WHERE user_agent.name IS NULL | STATS count(@timestamp) -> 55494

While of course, trying the same with a group by fails:

• FROM logs-* | STATS count(@timestamp) BY user_agent.name -> ValuesSources are mismatched

Index mappings

Finally, looking at the mappings files, there are 13 index mappings defined in elastic/logs:

• 8 files have no user_agent mapping at all
• 2 files have empty user_agent mappings (I’ve never seen this before!)
• 3 files have user_agent with sub-field name as keyword

The correctly mapped files contain the name like this:

"name": {
  "ignore_above": 1024,
  "type": "keyword"
}

The empty user_agent mapping looks like this:

"user_agent": {
  "properties": {}
}

[elasticsearchmachine]

Pinging @elastic/es-ql (Team:QL)

[elasticsearchmachine]

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

[dnhatn]

@craigtaverner Could you please rerun the query with error_trace=true to get a full stracktrace?

[dnhatn]

I've opened #100208