_ttl counting down faster than expected

[nithyanv]

Hi,

I believe _ttl is counting down faster than it should.

I have the following resultset for an index.

{
_index: some_index
_type: some_type
_id: some_id
_score: 1
fields: {
my_id: some_Id
my_timestamp: [
2015-03-26 16:05:00+0530
]
_ttl: 1581673831
}

Since today is 01/04/15, the document is around 5 days old. However, if we were to check _ttl (1581673831 milliseconds = 18.3 days left). How is that possible? It should be around 25 days to go The index was mapped with the _template mapping given below. I have also given the _mapping that was set up dynamically by Elastic when data was fed into it. Please note that _ttl is set correctly as 2592000000 milliseconds = 30 days as per _template mapping.

For the following template mapping,

"my_template" : {
"order" : 1,
"template" : "my_index*",
"settings" : {
"index.mapping.allow_type_wrapper" : "true",
"index.refresh_interval" : "5s",
"index.store.compress.stored" : "true",
"index.cache.field.type" : "soft"
},
"mappings" : {
"default" : {
"_timestamp" : {
"enabled" : true,
"path" : "my_timestamp",
"format" : "YYYY-MM-dd HH:mm:ssZ"
},
"_ttl" : {
"enabled" : true,
"default" : "30d"
},
"properties" : {
"my_timestamp" : {
"format" : "YYYY-MM-dd HH:mm:ssZ",
"type" : "date"
},
"my_id" : {
"index" : "not_analyzed",
"type" : "string"
}
},
"_all" : {
"enabled" : false
}
}
},
"aliases" : { }
}
}

It has the following index mapping,

"my_index_2015.03.26" : {
"mappings" : {
"mytype1" : {
"_all" : {
"enabled" : false
},
"_timestamp" : {
"enabled" : true,
"path" : "my_timestamp",
"format" : "YYYY-MM-dd HH:mm:ssZ"
},
"_ttl" : {
"enabled" : true,
"default" : 2592000000
},
"properties" : {
"my_id" : {
"type" : "string",
"index" : "not_analyzed"
},
"my_timestamp" : {
"type" : "date",
"format" : "YYYY-MM-dd HH:mm:ssZ"
}
}
}
}
}
}

[clintongormley]

Hi @nithyanv

When I try out your examples, the _ttl looks correct. I wonder if you ran an update request on this document?

[nithyanv]

We are populating using bulk input. Sometimes, we see overwrites (i.e) version is incremented some of our records may have same _id in the test data. But timestamp is the same even in these records. How does update affect _ttl?

[clintongormley]

@nithyanv an update request could mess with the TTL, depending on what you set.

Perhaps check the version of the document with the bad TTL value? See if it is 1 or higher. Also try inserting a new document with the same timestamp, and see if it is also reflecting the wrong TTL?

It seems to work for me, so I'm trying to figure out what is different about your setup.

[nithyanv]

@clintongormley I believe it is related to Update. We were also getting _version conflict as multiple threads were updating the same record over and over again. That was resolved with setting retryOnconflict(). But now we still get "org.elasticsearch.index.mapper.MapperParsingException: failed to parse [_ttl]" when writing over the same record. This is even though document is well within the ttl.

[clintongormley]

Closing in favour of #11809

[clintongormley]

I was incorrect - #11809 doesn't fix this issue. Reopening.

Related to #11802

[clintongormley]

Closing in favour of #18280