Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Field API fails with _id being an alias #75836

Closed
timroes opened this issue Jul 29, 2021 · 3 comments
Closed

Field API fails with _id being an alias #75836

timroes opened this issue Jul 29, 2021 · 3 comments
Assignees
Labels
>bug :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team

Comments

@timroes
Copy link
Contributor

timroes commented Jul 29, 2021

Steps to reproduce:

  1. Create the following index with the appropriate index mapping:
PUT /my-index-000001
{
  "mappings": {
    "properties": {
      "@timestamp":    { "type": "date" },  
      "pseudonym":   { "type": "alias", "path": "_id"  }
    }
  }
}
  1. Create the following document:
POST my-index-000001/_doc/
{
  "@timestamp": "2021-07-29T13:13:00",
  "message": "GET /search HTTP/1.1 200 1070000",
  "user": {
    "id": "kimchy"
  }
}
  1. Query now for all fields:
GET my-index-000001/_search
{
  "fields": [ "*" ]
}

The last query will now fail with the following exception:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "unsupported_operation_exception",
        "reason" : "unsupported_operation_exception: Cannot fetch values for internal field [_id]."
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "field-alias-test",
        "node" : "VBJ2JqanRZG4IwpqrPS-WA",
        "reason" : {
          "type" : "unsupported_operation_exception",
          "reason" : "unsupported_operation_exception: Cannot fetch values for internal field [_id]."
        }
      }
    ]
  },
  "status" : 500
}

Expected behavior: This request should not fail, but return all fields correctly.

@timroes timroes added >bug needs:triage Requires assignment of a team area label Team:Search Meta label for search team labels Jul 29, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@dnhatn dnhatn added :Search/Search Search-related issues that do not fall into other categories and removed needs:triage Requires assignment of a team area label labels Jul 30, 2021
@cbuescher cbuescher self-assigned this Oct 7, 2021
@cbuescher
Copy link
Member

I looked into this a bit and we came up with a suggestion in the team that we might move away from generally prevent fetching internal fields via the fields option.
We would still exclude any metadata field from matching wildcard patterns like "*", but if explicitely asked for (or like in this case requested through an alias) we'd return the value. I opened #78828 as a proposal to discuss this option.

cbuescher pushed a commit that referenced this issue Oct 11, 2021
Currently we exclude metadata fields from being looked up using the `fields` option in search.
However, as issue like #75836 show, they can still be retrieved e.g. via aliases and then fetching
their values causes errors. 
With this change, we enable retrieval of metadata fields (like `_id`) using the fields option when
 the field is explicitely asked for. We still continue to exclude any metadata field from matching
wildcard patterns, but they should be retrievable via an exact name or if there is an alias definition
with a path to a metadata field. This change adds support for the `_id` field in particular.
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Oct 11, 2021
Currently we exclude metadata fields from being looked up using the `fields` option in search.
However, as issue like elastic#75836 show, they can still be retrieved e.g. via aliases and then fetching
their values causes errors.
With this change, we enable retrieval of metadata fields (like `_id`) using the fields option when
 the field is explicitely asked for. We still continue to exclude any metadata field from matching
wildcard patterns, but they should be retrievable via an exact name or if there is an alias definition
with a path to a metadata field. This change adds support for the `_id` field in particular.
cbuescher pushed a commit that referenced this issue Oct 14, 2021
Currently we don't allow retrieving metadata fields through the fields option in search but throw
an error on this case. In #78828 we started to enable this for "_id" if the field is explicitely requested.
This PR adds _index and _version metadata fields which are internally stored as doc values to
the list of fields that can be explicitely retrieved.

Relates to #75836
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Oct 14, 2021
)

Currently we don't allow retrieving metadata fields through the fields option in search but throw
an error on this case. In elastic#78828 we started to enable this for "_id" if the field is explicitely requested.
This PR adds _index and _version metadata fields which are internally stored as doc values to
the list of fields that can be explicitely retrieved.

Relates to elastic#75836
cbuescher pushed a commit that referenced this issue Oct 14, 2021
Currently we exclude metadata fields from being looked up using the fields option in search.
However, as issue like #75836 show, they can still be retrieved e.g. via aliases and then fetching
their values causes errors.
With this change, we enable retrieval of metadata fields (like `_id`, `_ignored` etc.) using the fields
option when the field is explicitely requested. We still continue to exclude any metadata field from
matching wildcard patterns, but they should be retrievable via an exact name or if there is an alias
definition with a path to a metadata field.
This change adds support for the `_id`, `_routing`, `_ignored`, `_index` and `_version` field in particular.

Backport of #78828, #78981 and #79042
@cbuescher
Copy link
Member

We implemented the above proposal for the following metadata fields: _id, _routing,
_ignored, _index and _version.
There are a few other internal fields (e.g. "_seq_no", "_nested_path", "_feature"...) that still throw an error when trying to retrieve them via an alias, but I think they are even more exotic and even less likely to be used in an alias, so I didn't include them in the work so far. I will open a follow up issue to discuss how we want to handle those (if e.g. ignoring them generally when trying to get value fetchers for them is better than throwing an error), but I think we can close this issue since requesting the above fields (directly or with an alias) should now work starting in 7.16.

cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Oct 25, 2021
Certain field types like the one for `_seq_no` or `_nested_path` only exist as
implementation details of other functionalities. However, we currently don't
reject creating an alias field with a path to these field types. This can lead
to errors like the one described in elastic#75836 where an alias to an internal field
broke the fields API.

This change adds a new "isInternalField" method to MappedFieldType that returns
"false" by default but can be overridden by implementations that wish to
announce that they are of truly internal use only. This information can then,
for example, be used in alias field validation to reject creating an alias to
those fields. This change adds `_seq_no`, `_field_names`, `_nested_path`,
`_source` and `_feature` to the field types that are rejected when an alias is
created for them.

Relates to elastic#79277
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team
Projects
None yet
Development

No branches or pull requests

4 participants