Do not allow users to set TTL larger than internal storage window #190
Labels
Comments
beevee
changed the title
beevee
added a commit
that referenced
this issue
Mar 25, 2020
lastCheckTimestamp denotes a most recent point in time when a trigger was checked. While checking trigger, we are interested only in metric points after lastCheckTimestamp, because all previous points should have been checked during the previous check. There is no obvious reason to fetch more historical data points. At least, there is no test that shows why we should do that. Why is this bad? Because we don't have any limits on TTL value set by user. Trying to fetch 1e15 metric points from a data source leads to OOM death of all checker instances. Close #190
beevee
added a commit
that referenced
this issue
Mar 25, 2020
lastCheck.Timestamp denotes a most recent point in time when a trigger was checked. While checking trigger, we are interested only in metric points after lastCheck.Timestamp, because all previous points should have been checked during the previous check. There is no obvious reason to fetch more historical data points. At least, there is no test that shows why we should do that. Why is this bad? Because we don't have any limits on TTL value set by user. Trying to fetch 1e15 metric points from a data source leads to OOM death of all checker instances. Close #190
beevee
added a commit
that referenced
this issue
Mar 25, 2020
lastCheck.Timestamp denotes a most recent point in time when a trigger was checked. While checking trigger, we are interested only in metric points after lastCheck.Timestamp, because all previous points should have been checked during the previous check. There is no obvious reason to fetch more historical data points. At least, there is no test that shows why we should do that. Why is this bad? Because we don't have any limits on TTL value set by user. Trying to fetch 1e15 metric points from a data source leads to OOM death of all checker instances. Close #190
beevee
added a commit
that referenced
this issue
Mar 26, 2020
beevee
added a commit
that referenced
this issue
Mar 26, 2020
beevee
added a commit
that referenced
this issue
Mar 26, 2020
beevee
added a commit
that referenced
this issue
Mar 26, 2020
beevee
added a commit
that referenced
this issue
Mar 26, 2020
beevee
added a commit
that referenced
this issue
Mar 29, 2020
beevee
added a commit
that referenced
this issue
Mar 29, 2020
beevee
added a commit
that referenced
this issue
Mar 30, 2020
beevee
added a commit
that referenced
this issue
Mar 30, 2020
beevee
added a commit
that referenced
this issue
Mar 30, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Сейчас при чеке триггера берется интервал с учетом TTL, тоесть есть TTL = 10 min, то мы возьмем промежуток равный:
from: время последнего чека - TTL
to: now.
Соответсвенно если TTL слишком большой то мы берем неоправданно большой промежуток, даже если за такой промежуток нет. Самое плохое, к чему это может привести это утечка всей памяти (оооочень большой TTL).
Самое простое решение: ограничить это время на фронте: максимальный промежуток - время хранения метрик.
Посложнее: разобраться, зачем мы чекаем за такой большой промежуток? Надеемся что метрики появятся? Может вообще не стоит чекать за такой большой промежуток?
The text was updated successfully, but these errors were encountered: