Follow-up improvements to field retrieval.

[jtibshirani]

In #55363 we designed a first version of a fields option for retrieving document fields during a search. This issue tracks important follow-up changes to 'complete' the feature.

Changes that could affect the API (should be resolved before removing 'beta' flag):

• Support unmapped fields. Support unmapped fields in search 'fields' option. #63690
• Finalize how to return nested content. Finalize how search 'fields' option returns nested data. #63709
• Finalize how to handle flattened fields.
• Consider a limit on the number of fields retrieved. Investigate current limit stemming from our use of automata for include_unmapped. Should we limit the number of fields that can be retrieved by the “fields” API #69983

Other changes:

• Handle meta fields like _size. Handle metadata fields in search 'fields' option #63569
• Avoid returning hidden text fields like _index_prefixes. Text fields should not expose internal field types #63446
• Support the API in inner_hits. Support the search 'fields' option in 'inner_hits'. #61949
• Move FieldMapper#valueFetcher to MappedFieldType. Move FieldMapper#valueFetcher to MappedFieldType #62974

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jtibshirani]

We discussed how flattened fields should be returned, and agreed on this proposal:

• Right now, if the parent flattened field is requested, we return its value as a JSON object. We do not flatten the value and return all the subfields as individual keys in the response. We'd like to stick with the current behavior, since it's simple and consistent with how we handle all other fields: if you request a field like my_field, then the response will contain the field's value under that same key my_field. Adding a new key for every subfield could actually be surprising behavior for users. Returning all the individual keys might also defeat the purpose of flattened fields -- they are useful when you have a large number of distinct subfields, and client code doesn't actually want to handle each subfield as its own 'column' by default.
• We don't currently support retrieving flattened subfields directly, as in "fields": [ "flattened_field.subfield"]. We would like to support this, and are tracking the enhancement in Support fetching flattened subfields #70605.

To clarify, this means that when requesting "fields": ["*"], the response contains an entry for the parent flattened field, but no entries for the subfields:

{
  "fields": [{ 
    "flattened_field": [{
      "subfield1": "value1",
      "subfield2": "value2"
    },
    "other_numeric_field": [ 42 ]
  }]
}

[jtibshirani]

Since we've addressed all issues and open questions, we can close this out and mark fields as 'GA'. I opened #71130 to remove the 'beta' flag.