Flags have poor defaults for new deployments

[leth]

When you start cortex for the first time, you end up with a poor set of default flags which do not reflect the ideal way to run cortex. e.g. #489

We also see this when running cortex locally for development, I think I heard @meccanorak discussing it recently.

Obviously, changing the defaults values of flags is a recipie for a bad time, and many sad ops teams, however I think there's a way we can work around this.

• Add a mandatory "deployment start date" flag which denotes when a deployment began
  • Probably needs to start as optional, and not do anything, then transition to mandatory and functional
  • For ease of local development, accept 'today' as a magic value
• For flags which improve behaviour, select a date from which they change default value
  • We should not backdate these
• On startup, compare the "deployment start date" with each flag date, and set the default

What do people think of this idea?
cc @aaron7, @jml, @tomwilkie, @meccanorak

[jml]

Could we instead enable the latest schema/table system if there are no explicit date flags, and keep current behaviour if there are?

[leth]

That would certainly be simpler! Sounds worth exploring!

…

On Thu, 10 Aug 2017, 18:44 Jonathan Lange, ***@***.***> wrote: Could we instead enable the latest schema/table system if there are no explicit date flags, and keep current behaviour if there are? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#538 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADMkuU1PoIWDM-SgVkU7bgJokcMKKsDks5sW0FfgaJpZM4Ozc2C> .

[tomwilkie]

Id like to move scheme definitions to a config file - it's getting to much for flags. This could also enable scheme reloads without restarts. Probably a different issue though. For flag default changes, can you share what you currently have in Dev/prod? I'll shared what I use and have we can make sure we don't break anything. On Thu, 10 Aug 2017 at 14:23, Marcus Cobden <[email protected]> wrote:

…

That would certainly be simpler! Sounds worth exploring! On Thu, 10 Aug 2017, 18:44 Jonathan Lange, ***@***.***> wrote: > Could we instead enable the latest schema/table system if there are no > explicit date flags, and keep current behaviour if there are? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#538 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AADMkuU1PoIWDM-SgVkU7bgJokcMKKsDks5sW0FfgaJpZM4Ozc2C > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#538 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhRJTevBso8juNBgl3_nF1Ymw9muPks5sW1iTgaJpZM4Ozc2C> .

[meccanorak]

Perhaps digressing....
Must confess to puzzlement over reasoning behind changing default (from true to false) for -dynamodb.use-periodic-tables
Maybe @tomwilkie (or anyone else) could enlighten?

[aaron7]

Perhaps digressing....
Must confess to puzzlement over reasoning behind changing default (from true to false) for -dynamodb.use-periodic-tables
Maybe @tomwilkie (or anyone else) could enlighten?

I believe it is because BigTable doesn't require weekly tables.

If you remember, we switch tables weekly for DynamoDB to avoid the number of shards growing too large for a table where throughput is distributed equally among them. This causes problems where we can't optimise throughput capacity if we are hitting one shard more than others. I don't believe BigTable suffers from the same problem, so the default was changed to false, otherwise it would have to be set explicitly to false for BigTable. I believe BigTable still uses the same schema config.

[bboreham]

I quite like Marcus' suggestion, but it seems rather complex in practice.

Maybe we could add one big -legacy-flags setting which gives current settings, and if you don't say that then you get a "recommended" set of defaults ?

[tomwilkie]

We’ve got config files now and ship a much more reasonable set of defaults for local dev. I think we can close this now.