Kibana needs to handle "Fielddata access on the _id field is disallowed" #52517

bhavyarm · 2019-12-09T16:11:06Z

Kibana version: 8.0.0

Elasticsearch version: 8.0.0

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from snapshot build on 8th dec

Describe the bug: Elasticsearch has a breaking change elastic/elasticsearch#49166 and Kibana needs to handle it.

Steps to reproduce:

Download latest snapshot for 8.0.0 ES/Kibana
Start ES/Kibana
Logs about the breaking change appear all over both ES/Kibana logs

Kibana logs:

 log   [11:11:26.794] [warning][plugins][usageCollection] { Error: [illegal_argument_exception] Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled
    at respond (/Users/bhavyarajumandya/Desktop/test_master/kibana-8.0.0-SNAPSHOT-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/bhavyarajumandya/Desktop/test_master/kibana-8.0.0-SNAPSHOT-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/bhavyarajumandya/Desktop/test_master/kibana-8.0.0-SNAPSHOT-darwin-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/bhavyarajumandya/Desktop/test_master/kibana-8.0.0-SNAPSHOT-darwin-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 400,
  displayName: 'BadRequest',
  message:
   '[illegal_argument_exception] Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled',
  path: '/.kibana_task_manager/_search',
  query: { ignore_unavailable: true },
  body:
   { error:
      { root_cause: [Array],
        type: 'search_phase_execution_exception',
        reason: 'all shards failed',
        phase: 'query',
        grouped: true,
        failed_shards: [Array],
        caused_by: [Object] },
     status: 400 },
  statusCode: 400,
  response:
   '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".kibana_task_manager_1","node":"PJ3jQRajSkW-IzEAP3-5LA","reason":{"type":"illegal_argument_exception","reason":"Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled"}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled"}}},"status":400}',
  toString: [Function],
  toJSON: [Function] }

ES logs:

[2019-12-09T11:15:07,181][DEBUG][o.e.a.s.TransportSearchAction] [bhavyarajumandya] [.kibana_task_manager_1][0], node[PJ3jQRajSkW-IzEAP3-5LA], [P], s[STARTED], a[id=7l-bdd6cQ2uDl-bv06SzdQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.kibana_task_manager], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"query":{"bool":{"must":[{"term":{"type":{"value":"task","boost":1.0}}},{"bool":{"filter":[{"term":{"_id":{"value":"task:oss_telemetry-vis_telemetry","boost":1.0}}}],"boost":1.0}}],"boost":1.0}},"sort":[{"task.runAt":{"order":"asc"}},{"_id":{"order":"desc"}}]}}]
org.elasticsearch.transport.RemoteTransportException: [bhavyarajumandya][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled
	at org.elasticsearch.index.mapper.IdFieldMapper$IdFieldType$1.build(IdFieldMapper.java:172) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:134) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:197) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:358) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:766) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:591) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:351) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:688) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
[2019-12-09T11:15:07,182][DEBUG][o.e.a.s.TransportSearchAction] [bhavyarajumandya] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException$1: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:304) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:242) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:423) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1059) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1168) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1142) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:65) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:688) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.lang.IllegalArgumentException: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled
	at org.elasticsearch.index.mapper.IdFieldMapper$IdFieldType$1.build(IdFieldMapper.java:172) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:134) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:197) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:358) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:766) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:591) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:351) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	... 9 more

Any additional context:
elastic/elasticsearch#49166

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-12-09T16:11:38Z

Pinging @elastic/kibana-stack-services (Team:Stack Services)

bhavyarm · 2019-12-09T16:12:01Z

~~Please note ES change is a breaking change for 7.6.0~~

Edit - breaking change is for 8.0.0 - cc @LeeDr @timroes confirmed it in my 7.6.0 logs.

tylersmalley · 2020-01-09T19:30:37Z

This is related to elastic/elasticsearch#49166 and elastic/elasticsearch#43599

tylersmalley · 2020-01-09T19:32:29Z

We're currently pinned to an older version of Elasticsearch 8.0.0 both locally and on CI. To unpin this, we will need to remove our usage of sorting on _id which is done here. Is sorting on _id required, or would this be something easy to remove?

pmuellr · 2020-01-09T19:51:21Z

There's also potentially this:

kibana/x-pack/legacy/plugins/task_manager/server/queries/mark_available_tasks_as_claimed.ts

Lines 98 to 121 in dfac5d8

    
           const SORT_VALUE_TO_BE_FIRST = 0; 
        
           export const sortByIdsThenByScheduling = (claimTasksById: string[]): SortClause => { 
        
             const { 
        
               _script: { 
        
                 script: { source }, 
        
               }, 
        
             } = SortByRunAtAndRetryAt; 
        
             return defaultsDeep( 
        
               { 
        
                 _script: { 
        
                   script: { 
        
                     source: ` 
        
           if(params.ids.contains(doc['_id'].value)){ 
        
             return ${SORT_VALUE_TO_BE_FIRST}; 
        
           } 
        
           ${source} 
        
           `, 
        
                     params: { ids: claimTasksById }, 
        
                   }, 
        
                 }, 
        
               }, 
        
               SortByRunAtAndRetryAt 
        
             ); 
        
           };

pmuellr · 2020-01-09T20:08:12Z

Quick peek through the code, I think it would be save to change the function doing this to remove the sortById bits. Clearly the thinking was we'd get exact pagination results, and there doesn't seem to be any other field that could be used instead. OTOH, I don't think where this gets used, it's going to matter, right now. Following the references, it's up to a fetch() API on the task_manager service, and I doubt anyone making any calls that depend on stable pagination.

kibana/x-pack/legacy/plugins/task_manager/server/task_store.ts

Lines 430 to 442 in dfac5d8

    
           function paginatableSort(sort: any[] = []) { 
        
             const sortById = { _id: 'desc' }; 
        
             if (!sort.length) { 
        
               return [{ 'task.runAt': 'asc' }, sortById]; 
        
             } 
        
             if (sort.find(({ _id }) => !!_id)) { 
        
               return sort; 
        
             } 
        
             return [...sort, sortById]; 
        
           }

pmuellr · 2020-01-09T20:59:03Z

Running FTS with KBN_ES_SNAPSHOT_USE_UNVERIFIED=1 (which uses the most recent snapshot), shows an error in the mark_available_tasks_as_claimed.ts file, noted above. The one in task_store.ts is also problematic, needs to be fixed.

resolves elastic#52517 Not sure if these fixes are completely kosher, specifically if there are any down-stream effects.

peterschretlen · 2020-01-09T22:15:40Z

The only usage I can see is the telemetry usage collector, which filters for a specific _id value when it does a fetch (so it doesn't use pagination). Seems safe to remove pagination support, we can re-introduce later if/when needed.

pmuellr · 2020-01-16T19:41:51Z

I'm running FTS/FTR from master (with Gidi's fix merged), setting the following when running FTS so it uses the latest unverified ES snapshot, which is what broke with the old code (when sorting with _id):

# FTS
KBN_ES_SNAPSHOT_USE_UNVERIFIED=1 cd-x-pack.sh node scripts/functional_tests_server.js \
>     --config=test/alerting_api_integration/security_and_spaces/config.ts```

# FTR
cd-x-pack.sh node ../scripts/functional_test_runner.js --grep "Alerting" \
    --config=test/alerting_api_integration/security_and_spaces/config.ts

passed!

Sorting/Aggregating by _id was removed in elastic#52517 Retains sorting as requested by elastic#29747 Signed-off-by: Tyler Smalley <[email protected]>

This reverts commit d2b90749634c5f1d7daac61751a81b4f5057fc9b. That approach unfortunately did not work due to the problem that is described in elastic#52517.

bhavyarm added bug Fixes for quality problems that affect the customer experience Feature:Task Manager Team:Stack Services labels Dec 9, 2019

gmmorris self-assigned this Dec 10, 2019

bmcconaghy added Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) and removed Team:Stack Services labels Dec 12, 2019

tylersmalley mentioned this issue Jan 9, 2020

Revert pinned ES snapshot #53645

Closed

pmuellr added a commit to pmuellr/kibana that referenced this issue Jan 9, 2020

removes task_manager's sorting of documents by _id

28d5d3a

resolves elastic#52517 Not sure if these fixes are completely kosher, specifically if there are any down-stream effects.

pmuellr mentioned this issue Jan 9, 2020

removes task_manager's sorting of documents by _id #54407

Closed

7 tasks

gmmorris mentioned this issue Jan 14, 2020

removes usage of the _id field in Task manager #54765

Merged

7 tasks

gmmorris closed this as completed in #54765 Jan 16, 2020

tylersmalley mentioned this issue Jan 16, 2020

Sort server-side in SavedObject export #55128

Merged

kobelb added the needs-team Issues missing a team label label Jan 31, 2022

botelastic bot removed the needs-team Issues missing a team label label Jan 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kibana needs to handle "Fielddata access on the _id field is disallowed" #52517

Kibana needs to handle "Fielddata access on the _id field is disallowed" #52517

bhavyarm commented Dec 9, 2019

elasticmachine commented Dec 9, 2019

bhavyarm commented Dec 9, 2019 •

edited

Loading

tylersmalley commented Jan 9, 2020

tylersmalley commented Jan 9, 2020

pmuellr commented Jan 9, 2020

pmuellr commented Jan 9, 2020

pmuellr commented Jan 9, 2020

peterschretlen commented Jan 9, 2020

pmuellr commented Jan 16, 2020

Kibana needs to handle "Fielddata access on the _id field is disallowed" #52517

Kibana needs to handle "Fielddata access on the _id field is disallowed" #52517

Comments

bhavyarm commented Dec 9, 2019

elasticmachine commented Dec 9, 2019

bhavyarm commented Dec 9, 2019 • edited Loading

tylersmalley commented Jan 9, 2020

tylersmalley commented Jan 9, 2020

pmuellr commented Jan 9, 2020

pmuellr commented Jan 9, 2020

pmuellr commented Jan 9, 2020

peterschretlen commented Jan 9, 2020

pmuellr commented Jan 16, 2020

bhavyarm commented Dec 9, 2019 •

edited

Loading