Cannot filter, sort, or aggregate on _type.

[xuzha]

There are a few Found customers find that they cannot filter on _type.
I could reproduce it with a simple clean index. (es 2.1, Kibana 4.3).

I think this might be a bug.

Discuss issue link : https://discuss.elastic.co/t/elasticsearch-2-0-with--type-not-indexed/35244/10

[johncollaros]

I have encountered this bug too after upgrading from ES1.7.x ->2.1 and Kibana 4.2 -> 4.3.1

All existing index patterns have _type as indexed.
Any new indices created after the upgrade sets _type indexed: false.
Looking in the .kibana index-patterns reveals this.

{
  "_index": ".kibana",
  "_type": "index-pattern",
  "_id": "newindex_2-*",
  "_version": 23,
  "found": true,
  "_source": {
    "title": "newindex_2-*",
    "timeFieldName": "@timestamp",
    "fields": "[{"name":"_id","type":"string","count":0,"scripted":false,"indexed":false,"analyzed":false,"doc_values":false},
{"name":"_type","type":"string","count":0,"scripted":false,"indexed":false,"analyzed":false,"doc_values":false},
{"name":"_score","type":"number","count":0,"scripted":false,"indexed":false,"analyzed":false,"doc_values":false}]"
  }
}

I tried updating the value using Elasticsearch API to modify .kibana index settings. After reloading kibana web page, this worked.
However, when refreshing the field list from the Settings->Indices tab, _type is updated back to false.

Refreshing field values for an old index where _type was set to indexed, _type stayed as indexed:true.

This is looking to be more of an ES bug that is affecting Kibana.

Using the _mapping API on an old index returned the following:

/oldindex-2014.12.28/_mapping/logs/field/_type?include_defaults=true

{
  "oldindex-2014.12.28": {
    "mappings": {
      "logs": {
        "_type": {
          "full_name": "_type",
          "mapping": {
            "_type": {
              "store": false,
              "index": "not_analyzed"
            }
          }
        }
      }
    }
  }
}

On the new index:
/newindex_2-2015.01.01/_mapping/logs/field/_type?include_defaults=true

{
  "newindex_2-2015.01.01": {
    "mappings": {
      "logs": {
        "_type": {
          "full_name": "_type",
          "mapping": {

          }
        }
      }
    }
  }
}

Looks like something has changed with ES 2.x. The mapping entity is empty, preventing Kibana from detecting the _type field as indexed.

So a workaround is to manually update the .kibana/index-patterns using the Elasticsearch API.

[aliostad]

Guys this is a complete bummer. I commonly use _type. Can we please get a resolution? It seems the fixing of this keeps getting postponed.

[johncollaros]

I got a reply on elasticsearch issue i raised: elastic/elasticsearch#16389 (comment)
Seems like the mapping api is never going to return any values for _metafields anymore and it was suggested that kibana needs to hardcode these values now.

There is a way around this. You need to edit the .kibana data..see my post on this discuss
https://discuss.elastic.co/t/elasticsearch-2-0-with--type-not-indexed/35244/20

[aliostad]

👍 Nice! @johncollaros

[epixa]

It sounds like same issue described in #5634

[rashidkpc]

As noted, this is being caused by a change to the Elasticsearch API that removes the returning of mappings for Elasticsearch metadata fields: elastic/elasticsearch#16389 (comment). I weighed in on that as I don't think its a good idea for us to hard code the abilities of those fields as they may change over time, as they have in the past. I'd prefer the Elasticsearch API informed us of the field's capabilities

[pemontto]

👍

[jconlon]

What's the point of having types if you can't filter on them? I went to the trouble of working out datatyping and mapping for my documents before indexing several different types of them.

Will this be fixed anytime soon in Kibana or do we have to either use the workaround (thanks for it doing it johncollaros) or redesign our our schemas by adding an extra type field?

Seems as though the workaround is also problematic as a user could reset Kibana and put it back into the broken state?

[dkulichkin]

It could be 100% done when sending a request manually, +1 for having a workaround for this feature,
Many thanks

[wusthuke]

@johncollaros　the method of eidt .kibana workd fine, but when i refresh in kibana (Settings --> Inidces --> refresh button), _type indexed change back to false. I used elasticsearch-2.3.3、kibana-4.5.2, is the kibana bug not fixed yet?

[orphaner]

👍

[johncollaros]

@wusthuke yes, if you ever use the refresh button in kibana it will undo the workaround. I've done a clean install of es 2.3.4 and kibana 4.5.4 and this is still broken.

[johncollaros]

Is anything being done to fix this issue in pre 5 releases?

[hummingV]

I could type in the filter manually in the search bar like this
_type: "typename"
It works without editing kibana index patterns. Doesn't have to worry about refresh index button either. It is indeed lame to have to type that in though.

[LeeDr]

I think this is the same issue with users trying to filter or aggregate on _index.

I documented here how a user could aggregate on _index by editing the visualizations visState query;
https://discuss.elastic.co/t/how-to-apply-different-filter-for-different-column-in-a-single-kibana-visualisation-data-table/58817/2

I understand the reluctance to hard-code _type and _index into the Kibana lists, but it seems they're pretty standard in Elasticsearch and probably won't go away any time soon (and lots of stuff would break if they did). If we don't, users will have to do hacks like this.

[qraynaud]

Yes I agree. Plus it will always be possible to deduce the capability using the ES cluster version. If it changes, it should not be hard to say, for every version < 5.x.y, then it is indexed, after it is not anymore.

I think this should be a blocker for 5.0.0 so we don't get this for the next big release.

[LeeDr]

Adding a scripted field type seems to help in a few cases.

1. Create a scripted field (I named it type)
2. Language = painless
3. Type = string
4. Format = - default -
5. Popularity = 0
6. Script = doc['_type'].value

From Discover tab I can add both fields to my document list under the histogram.

I can sort on my scripted type field but not on the _type field.

I can search on _type:logs but not on type:logs

I can filter on type:"logs" but not on _type

I get 2 warnings on Discover when I mouse over Visualize on _type;
1- Doc values are not enabled on this field. This may lead to excess heap consumption when visualizing.
2- This field is not indexed and might not be usable in visualizations.

I get 1 warning when I mouse over Visualize on my scripted type field Scripted fields can take a long time to execute.

In a Vertical Bar Chart Terms aggregation I see both _type and type and both return the same results. I can't tell any performance difference with my small data set.

Refreshing the index pattern does not remove the scripted field.

[Bargs]

Thanks to updates to the field stats API in ES 5.0 _type is now aggregatable. I think filtering and sorting should be possible as well, I'll need to look into the code that's blocking it at the moment.

[ytzlax]

@Bargs , I using kibana 5.1.2, I can create visualization of _type field, but when I filtering I get "_type field can not be used for filtering" error

[Bargs]

@ytzlax this will be fixed in 5.4