Revisit some TODO items around keys_changed_at consistency checks.

[rfk]

Past-rfkelly helpfully left a TODO item in the code for checking consistency of keys_changed_at, which I went to revisit in light of Bug 1613032.

Unfortunately for Past-rfkelly, his reasoning that we could add this check once all servers are running the latest code is not sound. There's a (very very small) chance that a user could have:

• Synced once against a new server, recording a keys_changed_at timestamp
• Reset their account password, then synced once against an older server during the rollout of the keys_changed_at feature, recording an updated generation but leaving the old keys_changed_at.
• Never synced again after that.

When this user suddenly comes back and syncs against a server with the proposed additional consistency checks, we would see an update to keys_changed_at but no corresponding update to generation, and incorrectly lock them out of their sync data.

I briefly toyed with the idea of adding a check that says "for records created after X date, ensure that a change to keys_changed_at is accompanied by a change to generation. But it seemed like too much effort for too little gain, so instead I have:

• Added some more documentation about the consistency relationships we actually expect to see between these values.
• Remove the "TODO" about adding more checks once all servers are deployed.
• Added a different, less restrictive check that ensures we never see a keys_changed_at that is outright ahead of the corresponding generation.

[tublitzed]

hey, @rfk are you still wanting to merge this? We're starting the port to Rust here in a few weeks so probably good if we get it in before then if so.

[rfk]

Yes please, I think it'll be worth having for completeness and comparison when you do the rust version.

[fzzzy]

I'm going to add this to the services engineering board so we don't keep forgetting it.

[fzzzy]

This looks good. Thanks for writing the extensive explanatory comment.