Field data loading is forbidden on [FIELDNAME] #15267
Comments
|
Having the same issue: kibana 4.3, elasticsearch 2.1.0 |
|
Same here, had to re-index with the same data twice, because a shutdown caused my translogs to be corrupted, now, the second time, i get "Field data loading is forbidden on timestamp" when i want to do anything in kibana... Had this error before on other fields, which i circled around by using raw fields. Maybe i am managing my data in an undesired way (different indeces for different parts of our architecture each split by date again, yes templates are present and i can use raw fields as well as geoip data, so this shouldnt be a problem). Standalone usage btw. on java 1.8.0_45 |
|
I have the same issue. Just a simplest sample data set. |
|
same issue here: i've also tried restarting all services and reindexing on elasticsearch. permissions on /var/lib/elasticsearch look fine. i'll try rolling elastic back to an earlier version shortly. Kibana error: elasticsearch log error: |
|
this seems to have had a good outcome (visualizations are working, at least atm)... be aware this will wipe all previous index data: then repeat these steps to reinstall 2.1.0 with the good 2.0.1 perhaps i'd broken my indexes somehow... not sure... eitherway, its working atm. |
|
I have the same issue: EDIT: Like @mirnujAtom I also use java 1.8.0_65 |
|
@cybacolt |
|
Fixed my problem with similar to what @cybacolt suggested - moved /var/lib/elasticsearch from a working server (which was upgraded from 2.0 to 2.1) and dropped all data it used to have. @Isabaellchen there is a variable called path.data: in elasticsearch.yml, check where does it refer to in your configuration. |
PhaedrusTheGreek
changed the title
|
The problem happens when Common Variations of this error:
In default Logstash 2.1 with Kibana 4.3+, it may come down to the fact that you have visualizing on an analyzed field.
Running an aggregation on an analyzed field will produce the same error: One should be able to solve the problem by using the Wiping the /var/lib/elasticsearch directory also deletes the index templates which are installed by Logstash, so that might explain why previous comments were able to workaround by doing this. |
|
As stated above, i received the error on the timestamp field, rendering kibana completely infunctional until i reindexed. Also, i had the same data indexed before, without receiving this error, until my translogs got corrupted and i had to start from scratch. |
|
@Isabaellchen are you using Logstash? I'm wondering because Logstash uses the @timestamp field, whereas your error shows just timestamp (without the @ symbol). Did your timestamp field's mapping happen to have |
|
A similar things happen if the field is a string. In my case I have JSON that is inserted and when trying to get a unique count it fails. This is with all the latest production versions. |
|
thanks @PhaedrusTheGreek ,through using the .raw version of the field instead,i solution the problem. |
|
@PhaedrusTheGreek |
|
This is not a bug. It is a safeguard. The logstash template now disables fielddata loading where it makes sense, eg see https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json#L15 You get this message when you try to sort or run aggregations or scripts on analyzed fields. Fulfilling this request would cause massive amounts of memory usage on your cluster, and it almost certainly isn't what you want anyway, eg This is by design. |
|
the good old 'its not a bug, its a feature' :) but seriously, this kind of change breaks the application to the point where we have to reindex our data. besides the 'breaking changes' section, where else should i look before updating? |
|
Now how do we turn off this safeguard so we can keep using the solutions we had in place before without changing them? |
|
@argais you can do it by updating the mapping for string fields as follows: |
|
This is happening when I sort on a column in the Kibana Discover page. |
|
I am also getting this using: I'm just testing with basic example from here: Full error: |
|
So in the use case below, what would be the appropriate solution with this new feature? I'm using logstash to parse web logs. I use kv to capture all the query strings, which in one case is a comma separated list which gets stored as string in elasticsearch, eg layers.raw:"w,x,y,z" The results for each log entry could be any combination of w,x,y,z or all of them. I was building my vizualization off the analyzed field "layers", because I could get a count of each call of w. That is no longer possible with this issue. |
|
I've been hit by this issue for the last couple of weeks, and I still can't figure out what exactly I need to do to fix it.. I don't know what "field data" is for one thing. Am I correct to assume that, I am doing something wrong with the way I am feeding the data? Is there something I need to do on my filter? Interestingly, I am not having this problem on an ELK instance where I am loading data from AMQP (input:rabbimq, output:elasticsearch on my logstash) without any filter. |
|
OK, to explain what all this means... Analyzed string fields are for full text search. Not-analyzed string fields are for concrete values eg HTTP method, URI, status code, etc. Typically these fields are used for filtering (rather than full text search) and are often used in aggregations. It seldom makes sense to aggregate on an analyzed string field, eg an analyzed string field containing "Quick brown fox" indexes the terms "quick", "brown", and "fox", and a terms aggregation on this field would aggregate on each of these terms (instead of the full "Quick brown fox", which is probably what you wanted). On top of that, not_analyzed strings can use "doc_values", ie their values get stored on a disk in a way that they can be looked up for each document in a very efficient way while running aggregations. Analyzed strings do not support doc values. If you try to aggregate on an analyzed string, it reads the full index, "uninverts the values", and stores the result in memory. This can use an ENORMOUS amount of memory and even crash your server. You don't want to do this. The change in logstash-plugins/logstash-output-elasticsearch#309 means that:
So, instead of aggregating on field |
|
That makes sense this far. But is it intended, that the *.raw field is empty, althoug it is dynamically created? I am just starting out with ELK and doing some tests with the syslog input plugin. It imports the data, but won't let me visualize it. Now your explanation makes sense to me, but the *.raw fields are all empty on my side. Wouldn't it make sense to dynamically add the data to the fields? Or am I missing something? |
|
@installboy the |
|
@clintongormley Thanks for the explanation. Is there anyway to make Kibana use "host.raw" (for example) field instead of "host" when user tries to visualize (or "aggregate") analyzed field? Right now, Kibana doesn't even show "host.raw" under "Available Fields" list, so I didn't know I should've used that field instead. Currently, In order to visualize the hostname, our user needs to do following.
Obviously.. having to ask all of our users to this is not an option. I feel that the better approach is to tell ElasticSearch to use non-analyzed field for fields like "host". If it is, is there a documentation on how to do this? |
|
@rashidkpc see #15267 (comment) i was under the (possibly mistaken?) impression that Kibana automatically selected the |
|
@clintongormley it doesn't for me - in fact, I get a |
Not for me either. But if it did, it would solve this issue for me. |
|
The fix for me turned out to be to regenerate the index pattern. This can be done in the Kibana GUI by going to Settings, deleting the index pattern, and recreating it with default settings. The |
|
@kmoe I have the .raw fields in my index (although they won't appear under "Available Fields" list). I can query against it, generate graphs, etc.. The problem is, when a user clicks on a field (such as "host") Kibana doesn't automatically use the host.raw. It uses the (analyzed)host field instead and generates an error message.
My understanding is that, Kibana UI needs to be updated so that if user clicks on the Visualize button, it should look for .raw counter part, and if it exists, use that field to generate the URL to visualize the field. |
|
Yeah, I've done that in my fork of Kibana @soichih, but I had to change the way visualisations were loaded so I doubt I'll be able to make a PR that would get accepted upstream. |
|
@kmoe Is it possible to send pull-request for just the fix for .raw field? If you could share a fork I can try nitpicking. |
|
@clintongormley So in my use case where we were we are capturing a query string from a URI that is a comma dellimted list of options which can be unique for any call and there are hundreds of options, how would I replace the fact that I could take a w,x,y,z list and get counts for any of the individual options? With the raw field of course I an only going to get w,x,y.z as a return. |
|
@soichih Unfortunately I'm not able to share my fork at the moment, and as I say it quite dramatically changes the way visualisations are loaded (it now tries to load them from a JSON file as we need declarative template dashboards in the filesystem). However, this weekend I will see if I can make a small change in the source that would be accepted upstream to automatically use |
|
@kmoe If you could that, that will be fantastic! I see a lot of PR request / merge activities here > https://github.com/elastic/kibana/pulls |
i chatted to @rashidkpc about this and he said it used to be supported in Kibana 3 but was removed because it proved to be problematic. @kmoe 's solution of regenerating the index pattern (#15267 (comment)) I believe Kibana is going to be switching to not caching the mappings, so this issue should disappear. |
|
Thanks for the info. I am not sure what "caching the mappings" means, but do you know when this feature will be released? I tried @kmoe's solution(?) for recreating the index but it didn't solve my problem. The problem is not that the .raw fields doesn't exist, but rather Kibana doesn't automatically select those .raw field when user clicks "Visualize" button. Instead, Kibana tries to draw graphs using analyzed version of the field and throws an exception that user does not understand. |
|
As @clintongormley said, it's a must to not aggregate or sort on an analyzed field. I have some users doing that, and this can make your cluster crash and JVM going into «GC Infinity Loop». Even if you have plenty heap... I was happy to see the the new template: In our case, as a general rule we want to disable analyzed string fields at all. So, we modified a little bit the default template to make all string fields behave like generated .raw fields. Our template: I tried this new template by copying current production data to a test environment, then try our kibana dashboards on this. Then I get this error... I though that simply reindexing the same exact fields with my new template (strings not analyzed with doc values) will transparently render all dashboards work correctly without crashing my cluster. I just want to make all my strings fiels not analyzed with doc values, and as if they were .raw fields. UPDATE: |
I don't - you can ask Kibana about that.
Yes, as I mentioned above, autoselecting the
Yes, this setting should be used for analyzed strings which don't support doc values. For them it is a choice between loading all of the data into memory or not. In #15728 (comment) we're talking about providing a different API to return distinct fields which can be queried/aggregated on. I think this could greatly improve the Kibana experience. |
|
I met this problem when I upgrade es to the current version, and I found the groovy sort script not working anymore. I tried to fix mapping, |
|
Hi I have recently ingested apache logs to ES through logstash and I am analyzing them through Kibana . Few of the log lines contains the phrase- java.lang.outofmemory.....but when I search for the word "memory" in the kibana search bar it doesn't show up the document containing above phrase. looks like the default analyzer considering the whole phrase java.lang.outofmemory as one term rather than splitting it up into java,lang,outofmemory |
|
@ajay2881 You can use the Elastic Discourse forums for general questions. |
and others
when i Click “visualize” and then ,display error "Visualize:java.lang.lllegalStateException:Field daVisualize",or i used dashboard module ,before logstash auto update index,it was normal;when time is 8:00 am,dashboard module will not work,and display the error.
error details:
1.kibana page error:
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-2015.11.29","node":"ODyds4KgQsqrxczx1ana8A","reason":{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path","caused_by":{"type":"unchecked_execution_exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path","caused_by":{"type":"illegal_state_exception","reason":"Field data loading is forbidden on path"}}}}]}} KbnError@http://www.xxx.cn:80/bundles/c ... 64:30 RequestFailure@http://www.xxx.cn:80/bundles/c ... 97:19 http://www.xxx.cn:80/bundles/k ... 05:57 http://www.xxx.cn:80/bundles/c ... 91:28 http://www.xxx.cn:80/bundles/c ... 60:31 map@[native code] map@http://www.xxx.cn:80/bundles/c ... 59:34 callResponseHandlers@http://www.xxx.cn:80/bundles/k ... 77:26 http://www.xxx.cn:80/bundles/k ... 84:37 processQueue@http://www.xxx.cn:80/bundles/c ... 09:31 http://www.xxx.cn:80/bundles/c ... 25:40 $eval@http://www.xxx.cn:80/bundles/c ... 53:29 $digest@http://www.xxx.cn:80/bundles/c ... 64:37 $apply@http://www.xxx.cn:80/bundles/c ... 61:32 done@http://www.xxx.cn:80/bundles/c ... 10:54 completeRequest@http://www.xxx.cn:80/bundles/c ... 08:16 requestLoaded@http://www.xxx.cn:80/bundles/commons.bundle.js:37749:25
2.elasticsearch error log:
[2015-11-30 00:04:53,395][DEBUG][action.search.type ] [Thumbelina] [logstash-2015.11.29][2], node[acrTX4O0RciN8ppbSdfoww], [P], v[4], s[STARTED], a[id=ogwkZP0yQCCgatW0_tnvnw]: Failed to execute [org.elasticsearch.action.search.SearchRequest@aef2ffe] lastShard [true]
RemoteTransportException[[Thumbelina][192.168.1.76:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
Caused by: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:343)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:363)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:375)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:165)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:82)
at org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$FieldData.bytesValues(ValuesSource.java:195)
at org.elasticsearch.search.aggregations.bucket.terms.StringTermsAggregator.getLeafCollector(StringTermsAggregator.java:73)
at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:132)
at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:38)
at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:117)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:763)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:324)
... 11 more
Vian 00:18:36
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Field data loading is forbidden on response
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:156)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:76)
… 19 more
Caused by: java.lang.IllegalStateException: Field data loading is forbidden on response
at org.elasticsearch.index.fielddata.plain.DisabledIndexFieldData.fail(DisabledIndexFieldData.java:68)
at org.elasticsearch.index.fielddata.plain.DisabledIndexFieldData.loadDirect(DisabledIndexFieldData.java:54)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:163)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:156)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
… 23 more
The text was updated successfully, but these errors were encountered: