id_cache memory footprint grows linearly with number of parent documents

[polyfractal]

As parent documents are indexed, the size of the parent/child id_cache grows linearly. This applies even if the majority of parents do not have any associated children.

[deverton]

I think we might be seeing the same issue. I posted about it in the mailing list

[martijnvg]

Keeping track of which parent id is referenced by a child document is tricky. Right now we just simply load all parent ids in memory of the new segments being made searchable during a refresh. If we need to keep track if a parent id is actually used than in a worst case scenario we need to go over all segments that contain child documents and check if a parent id being loaded is actually referenced by a child doc. This can slowdown the refresh significantly.

To mitigate the high memory usage, it is best to add more nodes to you cluster. The id cache size will then nicely be divided amongst the nodes in the cluster.

[martijnvg]

Closing this in favor for #6107. Once _parent field has been cut over to doc values the on heap memory size will be 0.

Internally the id_cache has been removed since ES 1.1, and _parent field is now based on field data, also _parent field data is now less wasteful as before ES 1.1 with id cache.

[ostersc]

We are using ES 1.2.1 and are still seeing linear growth of the id_cache as we insert documents which can be parents (even if there are no children in the index).
Can you confirm the above statement: "Internally the id_cache has been removed since ES 1.1..." and give some insight into why this might be the case?