receive: Add ability to consume contents of the hashrings file directly #3121

kakkoyun · 2020-09-03T15:13:27Z

Signed-off-by: Kemal Akkoyun [email protected]

I added CHANGELOG entry for this change.

Fixes #3026.

Changes

Added --receive.hashrings alternative to receive.hashrings-file flag (lower priority). Content of JSON file that contains the hashring configuration.

Verification

make test-local
make test-e2e

cmd/thanos/receive.go

squat · 2020-09-03T15:52:59Z

Looking forward to taking a close look at this 👍

kakkoyun · 2020-09-17T14:18:37Z

Friendly ping @squat
Also somewhat related #3138

bwplotka

Thanks! LGTM 💪 perfect.

bwplotka

LGTM from my side.

pkg/extflag/pathorcontent.go

cmd/thanos/receive.go

pkg/receive/config.go

kakkoyun · 2020-12-09T13:55:58Z

@squat Thanks for the great review as always. I have addressed your concerns, could please take another look at it?

bwplotka

Love it especially extra tests. But I think we can do better on pathOrContent API 🤗 Let's do it properly as it's quite flexible flag, so would be nice to have clean API.

bwplotka · 2020-12-18T10:25:35Z

cmd/thanos/receive.go

 			}, func(error) {
 				cancel()
 			})
 		} else {
+			// The Hashrings config file path is not given, so initialize using content..


Suggested change

// The Hashrings config file path is not given, so initialize using content..

// The Hashrings config file path is not given, so initialize using content.

bwplotka · 2020-12-18T10:37:18Z

pkg/extflag/pathorcontent.go

+// Path returns the path of the given file that has passed to the flag.
+// It returns error if both a path and content are given.
+// It returns error if the required flag is set to true and path is empty.
+func (p *PathOrContent) Path() (string, error) {


Hmmmm...

So Content() is a bit different semantically. We aim for Content to be always there unless flag is optional.

With Path.. we have one more case, of user not specifying path at al, but specifying content. While technically this method should be understandable there are so many cases to tests to check which of those 6 cases we have (content passed, path passed, filag not used) x2 (required or not)

While all of this we have also this important message that is hidden in terms of hashring file:

IF you specify via content it will be used. If specify via path it will be dynamically reloadable. There are 2 problems with this:

This is surprising. We never reload content. How user can know we do it now? And let's say we are explicit we reload hashring. How user will know that other flags then hashrings are not dynamically reloadable?

What if I want via path but don't want to dynamically reload? For example I want to put file, start receive and remove (for some reason). Maybe rare, but someone could be surprised this is not possible. Might other usecases as well.

I think we need to think this through. I have some ideas, but before I state them, maybe you have as well? 🤗 Do you see the problem?

This is surprising. We never reload content. How user can know we do it now? And let's say we are explicit we reload hashring. How user will know that other flags then hashrings are not dynamically reloadable?

I understand the concern. Maybe overloading the PathOrContent for the Receive's use case is not the best way.
Receive has a special way to handle the given path as you already mentioned. It initializes a watcher for the given path.

What if I want via path but don't want to dynamically reload? For example I want to put file, start receive and remove (for some reason). Maybe rare, but someone could be surprised this is not possible. Might other usecases as well.

This is a problem we already have. This PR shouldn't be aiming to solve this. However, I can add more documentation to clarify this behaviour.

How about rather than overloading the PathOrContent, we implement dedicated logic in the receive. Specific to receive? Would this work? WDYT?

hm what about, creating a second constructor?

So we can have:

something or something-file as one constructor

something and something-file and something-reloadable-file if you use another constructor

So receive will use latter (: Then we can actually embed file watching in the second constructor, directly in pathOrContent package 🤗

WDYT?

I like the idea. I would love to work on it on a separate PR to introduce this feature in a more generalized form.
How about merging this PR as is and then creating an issue for it? As I told you I can work on it. WDYT?

@bwplotka As we talked off-line, do we have a consensus? Can we proceed with this?

YES! love it. Thanks!

Signed-off-by: Kemal Akkoyun <[email protected]>

Simplify hasring configuration Signed-off-by: Kemal Akkoyun <[email protected]>

bwplotka

Well done with short term thing 👍🏽

It will have the same flag spec 🤗

kakkoyun requested review from bwplotka and squat September 3, 2020 15:13

kakkoyun force-pushed the hashring_file branch from facacc8 to cfb0bda Compare September 3, 2020 15:14

kakkoyun changed the title ~~receive: Add ability to consume content of the hashring directly~~ receive: Add ability to consume contents of the hashring file directly Sep 3, 2020

kakkoyun changed the title ~~receive: Add ability to consume contents of the hashring file directly~~ receive: Add ability to consume contents of the hashrings file directly Sep 3, 2020

kakkoyun force-pushed the hashring_file branch from cfb0bda to f4b7c33 Compare September 3, 2020 15:31

squat reviewed Sep 3, 2020

View reviewed changes

cmd/thanos/receive.go Outdated Show resolved Hide resolved

kakkoyun mentioned this pull request Sep 17, 2020

receive: Use PathOrContent file for the hashring #3026

Closed

kakkoyun requested a review from squat September 17, 2020 14:18

bwplotka approved these changes Sep 23, 2020

View reviewed changes

kakkoyun force-pushed the hashring_file branch from a2222ce to b2f7761 Compare September 24, 2020 07:34

bwplotka approved these changes Sep 24, 2020

View reviewed changes

kakkoyun force-pushed the hashring_file branch from b2f7761 to cedcac5 Compare October 29, 2020 10:44

squat reviewed Nov 17, 2020

View reviewed changes

pkg/extflag/pathorcontent.go Outdated Show resolved Hide resolved

squat reviewed Nov 17, 2020

View reviewed changes

cmd/thanos/receive.go Outdated Show resolved Hide resolved

squat reviewed Nov 17, 2020

View reviewed changes

pkg/receive/config.go Show resolved Hide resolved

kakkoyun force-pushed the hashring_file branch 2 times, most recently from 7e21dfd to 81dbb29 Compare December 9, 2020 13:43

bwplotka requested changes Dec 18, 2020

View reviewed changes

kakkoyun force-pushed the hashring_file branch from 9915bc7 to 3dad8ba Compare December 18, 2020 11:28

kakkoyun added 5 commits December 18, 2020 14:32

Add ability to consume content of the hashring directly

f8636ab

Signed-off-by: Kemal Akkoyun <[email protected]>

Remove file format

c49a8cf

Signed-off-by: Kemal Akkoyun <[email protected]>

Address review issues

7095c35

Signed-off-by: Kemal Akkoyun <[email protected]>

Address review issues

207b945

Signed-off-by: Kemal Akkoyun <[email protected]>

Address review issues

996dad9

Simplify hasring configuration Signed-off-by: Kemal Akkoyun <[email protected]>

kakkoyun force-pushed the hashring_file branch from 3dad8ba to 996dad9 Compare December 18, 2020 11:33

bwplotka approved these changes Dec 28, 2020

View reviewed changes

bwplotka merged commit dd670a3 into thanos-io:master Dec 28, 2020

kakkoyun deleted the hashring_file branch January 4, 2021 10:39

kakkoyun mentioned this pull request Mar 24, 2021

receiver: Split - Upstream the changes from the PR #3580 #3845

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

receive: Add ability to consume contents of the hashrings file directly #3121

receive: Add ability to consume contents of the hashrings file directly #3121

kakkoyun commented Sep 3, 2020 •

edited

Loading

squat commented Sep 3, 2020

kakkoyun commented Sep 17, 2020

bwplotka left a comment

bwplotka left a comment

kakkoyun commented Dec 9, 2020

bwplotka left a comment

bwplotka Dec 18, 2020

bwplotka Dec 18, 2020

kakkoyun Dec 18, 2020

bwplotka Dec 18, 2020

kakkoyun Dec 18, 2020

kakkoyun Dec 18, 2020

bwplotka Dec 28, 2020

bwplotka left a comment

	// The Hashrings config file path is not given, so initialize using content..
	// The Hashrings config file path is not given, so initialize using content.

receive: Add ability to consume contents of the hashrings file directly #3121

receive: Add ability to consume contents of the hashrings file directly #3121

Conversation

kakkoyun commented Sep 3, 2020 • edited Loading

Changes

Verification

squat commented Sep 3, 2020

kakkoyun commented Sep 17, 2020

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

kakkoyun commented Dec 9, 2020

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka Dec 18, 2020

Choose a reason for hiding this comment

bwplotka Dec 18, 2020

Choose a reason for hiding this comment

kakkoyun Dec 18, 2020

Choose a reason for hiding this comment

bwplotka Dec 18, 2020

Choose a reason for hiding this comment

kakkoyun Dec 18, 2020

Choose a reason for hiding this comment

kakkoyun Dec 18, 2020

Choose a reason for hiding this comment

bwplotka Dec 28, 2020

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

kakkoyun commented Sep 3, 2020 •

edited

Loading