Missing field mappings - Increase the default lookBack setting?

[w33ble]

When Kibana fetches field mappings for an Index Pattern, not all of the fields show up

This is especially confusing when viewing the query results in Discover, where fields show up, but those same fields don't show up in the index mappings, and refreshing the mappings doesn't help. Because the fields don't exist in the mapping cache, they can't be used to create filters in Discover, or aggs in Visualize.

A few users mentioned this during Elasticon, and it just came up again when chatting with another user. I suspect this happens when a user has a number of indices (say, time-based indices) and not all of the indices have the same fields. One user I spoke to said they had multiple indices and it seemed like the mapping they got back were only the fields that existed in all of the indices.

I haven't yet dug into the issue yet to confirm the cause. I've added "needs details" since this bug needs some more exploration, specifically a confirmed cause and repro steps.

[spalger]

Should be fixed by elastic/elasticsearch#12817

[w33ble]

@spalger true. It doesn't appear to have a target set, but it's probably safe to assume we won't see it until 5.x or later. If this is a bug now, it would be good to at least know where the bug is and decide when and how to fix it.

[clintongormley]

See elastic/elasticsearch#15728 instead

[Bargs]

I don't think this is caused by fields missing from some indices but not others. There's no code here which does that sort of filtering: https://github.com/elastic/kibana/blob/master/src/ui/public/index_patterns/_transform_mapping_into_fields.js

Do you have any idea if the users were using wildcard or timestamp based index patterns? If timestamp based, the missing fields might be explained by an indexPattern:fieldMapping:lookBack value that's too low. Just realized that makes my first statement a bit of a contradiction. I should say I don't think it's caused solely by heterogeneous mappings :)

[w33ble]

@Bargs in the case of one user I was working with, this was indeed the problem. They are using daily indices and hadn't indexed data into the field in a while, so it went away and wasn't showing back up. Setting the loopBack higher fixed their issue.

Since I heard about this problem from some other users, I'm going to leave this open in the hopes that others can find this issue and the solution. I'd love some confirmation from more users that a higher loopBack setting fixes their problems too.

Adding the Discuss tag as I wonder if there's a way to make the lookBack more reliable, or if it's at least worth bumping up the default value to maybe 14 or 30, so users are less likely to run in to the problem.

[Bargs]

This issue should be solved by this one #6498

[w33ble]

Indeed, but it might be nice for existing users to bump the default up, and it's a really small change on our part. AFAIK, there shouldn't be any major issues using a number higher than 5.

[w33ble]

One of the things we really want to do is remove the mapping cache entirely in Kibana, and rely exclusively on the field stats API in Elasticsearch. That's the best solution to this problem, and since there's been literally no community input on this issue, this is probably a really rare edge case issue.

I'm just going to say that we rely on the removal of the mapping cache to fix the problem, so I'm closing this issue.