Rethink how to index nested documents when types are gone

[jpountz]

Currently nested documents reuse the _type field in order to identify root documents from children: the value of the type field starts with __ for children. However with types going away, we would probably like to stop indexing the _type field so we would need to find other ways to identify root/children.

For efficiency reasons, we need a fast query that identifies root documents. One way we could do that would be by adding a special field/value pair to root documents. However, we would probably only want to do that when there are nested mappings, which means we would have to reject adding nested objects to existing mappings.

[martijnvg]

I think this is a fair tradeoff. Especially since it would still allow the addition of additional nested fields, so it would only affect indices where no nested fields have been defined.

[jpountz]

Discussed in Fixit-Friday: we will benchmark the overhead of a field that always has the same value. If it does not matter, we can add a meta-data field that identifies nested documents, otherwise we can either:

• enforce nested mappings to be configured at index-creation time and only add the metadata field for root docs when there are nested mappings
• or add an option to disable the fied that identifies root documents so that users who do not need the feature can get some performance back

[s1monw]

I looked at this yesterday and after a closer look I think we can get the best of both worlds without indexing / adding anything to the documents. Today we index _version, _seq_id, and _primary_term for every nested document. While this is unnecessary and we can omit it we can use a DocValuesFieldExistsQuery once we don't index this for nested document. The values for these fields are all dummy values anyway so they have no real value for the user. Yet, when we look at the place where these dummy values are added we see that there is a reason for this:

https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/index/mapper/SeqNoFieldMapper.java#L247

// In the case of nested docs, let's fill nested docs with seqNo=1 and
// primaryTerm=0 so that Lucene doesn't write a Bitset for documents
// that don't have the field. This is consistent with the default value
// for efficiency.

This is a good reason for at least _seq_id and _version but given that we only access _primary_term rarely it might be ok to drop the value for nested docs in such a case. This would allow us to find parents very easily without paying any extra cost on indexing.

@bleskes I wonder what you think if this is a feasible solution?

[javanna]

@elastic/es-search-aggs

[jpountz]

We discussed it, this issue incorporated two things:

• the need to identify root documents efficiently, which @s1monw addressed as explained above
• the removal of the _type field, which still needs to be addressed

We agreed to rename _type to _nested to address the second point.