Remove ability to start components without sharding

[pstibrany]

Mimir currently has several components which don't require sharding:

• compactor (-compactor.sharding-enabled)
• store-gateway (-store-gateway.sharding-enabled)
• alertmanager (-alertmanager.sharding-enabled)
• ruler (-ruler.enable-sharding)

Mimir also has components which always require sharding configuration:

• distributor
• ingesters

I'd like to suggest to remove ability to start compactor, store-gateway, alertmanager and ruler with no sharding.

Implication is that these components would require a ring configuration to run properly. This would increase the initial configuration complexity, but would also make scaling the components much easier (just start another one).

To reduce the configuration complexity, we could introduce "global ring configuration", that would be used if component-specific ring was not configured.

[pstibrany]

Related to discussion in #814.

[09jvilla]

This seems reasonable to me - if some components already require a sharding config, its actually nice to be consistent and just make them all do it. The global ring configuration totally makes sense as well - ideally we can mimic what already exists today in Loki with the common_config, that lets you do exactly this.

The only thing I would like to check -- is there some way we can set the default sharding config such that the user doesn't have to do any extra configuration if all they are doing is running 1 Mimir process with target=all? Meaning that the default assumes sharding is enabled but you're only sharding to 1 replica (namely, itself).

I'm trying to see if we can get the benefit of consistency (requiring the sharding config for everything) while at the same time not adding configuration overhead for the simplest deployment architecture.

[pstibrany]

The only thing I would like to check -- is there some way we can set the default sharding config such that the user doesn't have to do any extra configuration if all they are doing is running 1 Mimir process with target=all? Meaning that the default assumes sharding is enabled but you're only sharding to 1 replica (namely, itself).

We can do this by using “inmemory” KV store to be the default for all components. Then if someone tries to start multiple instances, they all will be isolated.

Another possibility would be to use “memberlist” KV store. Then running multiple instances would require bit of configuration (port to listen on and port to join the cluster), but it would allow user to scale up quickly using only those two parameters. Running single instance would not require any extra options.

[Logiraptor]

Is it possible to have an experience like the following?

Single-instance, single-binary works with zero configuration
Multiple-instance only requires providing an address for the other instances (either directly or DNS)

I think memberlist can work without peers with zero configuration, is that right? If so, I would vote for memberlist as the default instead of inmemory.

[09jvilla]

If that's the case @Logiraptor , I also would vote for memberlist instead of inmemory. Seems like based on what you and @pstibrany said, there's a way we can do memberlist to make single-instance, single-binary require 0 configuration changes by the user (handled all with defaults) and make scaling up require only small config changes.

I believe Loki defaults to memberlist now for this same reason.

[09jvilla]

If I understand correctly, the default now for the rings is consul so regardless of whether we do memberlist or inmemory, its still a change from the existing default.

[pstibrany]

Single-instance, single-binary works with zero configuration Multiple-instance only requires providing an address for the other instances (either directly or DNS)

I think memberlist can work without peers with zero configuration, is that right? If so, I would vote for memberlist as the default instead of inmemory.

Yes, that’s correct. (With the caveat that if instances run on the same machine, they need to use different listening port for memberlist connections, so it requires changing of two options — one for listening port, one to find other peers.)

If I understand correctly, the default now for the rings is consul so regardless of whether we do memberlist or inmemory, its still a change from the existing default.

Correct.

[pstibrany]

Nice thing about memberlist is that it also provides “single ring” config – there is only single memberlist client configuration in Mimir, no matter how many services use it. This is in contrast to other ring clients.

[09jvilla]

Neat. Alrighty well it sounds like default sharding to on and using memberlist as the kvstore is the way forward then. @pstibrany @Logiraptor are yall in agreement? anyone else we should be running this by?

[Logiraptor]

Sounds good to me @09jvilla.

[pracucci]

I agree on having memberlist with zero config required for the 1 replica case (I wasn't much in favour of the in-memory KV store, which I believe shouldn't even be configurable).

[09jvilla]

@pstibrany -- not sure what other opinions are needed, but when ready, I think this is ready to move from "proposal" to "let's do this / no longer a proposal". If/when you feel the same, can you update the name of the issue?

[pstibrany]

I think we have consensus here. Changing the title.

[pstibrany]

Plan:

• Dskit update: Update dskit #935
• Ruler: Removed -ruler.enable-sharding and made memberlist the default value -ruler.ring.store. #943
• Compactor: Removed -compactor.sharding-enabled, changed default backend for compactor ring to memberlist. #956
• Store-gateway: Removed optional sharding from store gateway. #976
• Alertmanager, in progress
  • Alertmanager: Remove ability to run without clustering. #1044
  • Alertmanager: Remove upstream clustering (gossip) operation.  #1126
  • Alertmanager: Change default ring store from consul to memberlist. #1171
• Documentation
  • Remove mentions of obsolete sharding-enabled flags. #1123

[pracucci]

I think we can consider this work done, so I'm going to close the issue. Please reopen if there's anything else left.