rule: Fix bug when rules were out of sync

[lilic]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Note: The commits were taken from this PR #1439, as that PR was closed too early but the bug still exists and we hit it in our setup. I added the original author as the co-commiter, as they were not responsive on the original PR.

Changes

From original PR:

Thanos rule loading alerts about out-of-sync rule files
Specifically, the following scenario:
Rule file rule. Yaml, which contains some rules, through the reload command, thanos rule load rules.
If you clear the contents of rule. Yaml, but leave the file in, the old rules in thanos rule will not be cleared.
This bug leads to a common situation (all rules in one file)
If thanos rule has 100 rules, it can only clear 99 rules at most.

Note this also fixes when you have one rule loaded only, and you delete that rule it doesn't get updated, the number or rules is not important, it's the last rule.

Verification

There is a unit test added, I also tested this manually in our cluster setup by loading the rule via yaml file as the OP suggested and deleting the rule from the file, the rule was now cleared correctly.

[bwplotka]

NIce, LGTM, just some questions.

Also how this was fixed?

Note this also fixes when you have one rule loaded only, and you delete that rule it doesn't get updated, the number or rules is not important, it's the last rule.

[bwplotka]

Is that necessary?

[bwplotka]

Also don't see you describe this being a bug in PR desc 🤔 am I missing something?

[lilic]

I was just picking up from the previous PR which went through review from what I gathered, as mentioned in the description, if you feel like it's not needed let me know! :)

[simonpasquier]

IIUC the idea is to remove duplicate rule files (e.g. thanos rule --rule-file foo.yaml --rule-file foo.yaml) but in practice it shouldn't matter as the rule manager will only load the file once eventually.

[GiedriusS]

That's true but I don't see a problem in avoiding processing the same file twice or more. I would say we should keep this change.

[lilic]

Note this also fixes when you have one rule loaded only, and you delete that rule it doesn't get updated, the number or rules is not important, it's the last rule.

This was fixed as the consequence of this PR, or more like the bug scope was previously thought to be only when you have a lot of rules from what I gather.

[GiedriusS]

LGTM, well tested 🏅