Allow default (dynamic) time for date mappings

[polyfractal]

It would be convenient if the date field could be configured to provide a default time (likely now()). It would make sense if this was an extension to null_value which has specific, dynamic functionality for a date field.

[colings86]

+1

[clintongormley]

it would also remove the need for the dedicated _timestamp field

[colings86]

So I just had a look at how we might do this and thought of something relating to us doing dynamic null values for any type.

The null_value is obviously used for when there is no supplied value so adding a dynamic value to index is relatively straight forward in this respect. However, the null_value is also used in AbstractFieldMapper.nullValueFilter() to generate a filter for the MissingFilter and here is where the question arises.

We could say that if you set the null_value to be dynamic then the MissingFilter will essentially be a NoOP and not match any documents (since every document with a null value gets a dynamic value added and therefore doesn't have a missing value).

This could cause issues in a system where, for example, you set your null_value to 'NO VALUE' and some of your documents come in with the field set to null and some with the field set to 'NO VALUE' (e.g. for different systems). At the moment we treat both these as the same in the MissingFilter (they will both have the null_value in the index and will be returned by the MissingFilter. But if we make the null_value dynamic then the 'NO VALUE' fields won't be returned by the filter add won't be dynamically given a value either. I can see two options to solve this:

• Keep null_value as is and add a new parameter for dynamic value setting with an option to replace anything matching null_value with the dynamic value.
• Require that if you are using a dynamic null_value you ensure that all documents either have a valid value or null (i.e. we will neither replace things like 'NO VALUE' or return them from the MissingFilter)

[polyfractal]

Hmm, I agree..it conflates concerns and is confusing. null_value is mainly used when people need to differentiate "explicitly null" from "missing". Adding a default on top of that could lead to hairy situations as you described.

How about this:

• If configured, null_value will replace explicit NULLs with the placeholder
• If configured, missing_value will add missing fields to the document using the placeholder

Both null_value and missing_value can be configured for a static placeholder, or a dynamic value if the type makes sense (e.g. now() for dates)

That would allow everyone to specify exactly what they want with no surprises. The two settings won't interact, since a field is either null or missing.

It does break the case where a field was originally "missing" but made "not missing" by usage of missing_field, but I think that will be desired behavior and not a bug, since the user is wanting to replace a missing field with a particular value.

[clintongormley]

Another thing to consider - if we generate a value for a date field, should it alter the _source to include the generated timestamp? I think probably not. In which case the only way it would be retrievable would be to make the field stored.

[bobbyhubbard]

With _timestamp being deprecated this is a must have moving forward. We have usecases were we can't trust (or more likely dont have) the timestamp of the original document. We want the timestamp when ES last indexed it.

Maybe instead of adding confusion to null_value, maybe you could add a new type? For example, a new type called timestamp would always default the current timestamp into a stored field outside of the _source

This would update the stored field whenever the document is indexed. Another possible usecase might be to insert the timestamp only the FIRST time the document is indexed...which means if its set, dont update it on subsequent updates of the document. No idea how to name that...maybe timestamp_once?

[martijnvg]

@bobbyhubbard I think the way forward is by using the ingest node (#14049) when it gets available.

Adding a timestamp to a document would look like this:

1. Setup a pipeline with a set processor that uses the ingest timestamp:

PUT _ingest/pipeline/1
{
  "processors" : [
    {
      "set" : {
        "your_timestamp_field" : "{{_ingest.timestamp}}"
      }
    }
  ]
}

2. Make index/bulk requests use this pipeline:

PUT /index/type/id?pipeline_id=1
{
...
}

[PhaedrusTheGreek]

+1

[clintongormley]

Closing in favour of #14049