operator: Add option to act as cluster wide #1777

guilhem · 2017-12-14T23:33:46Z

for a cluster to be managed widely, user have to add annotation:
etcd.database.coreos.com/scope: clusterwide
And admin have to run an operator with option -cluster-wide.

Current implementation have a lack of locking if many operators with cluster-wide run on different namespaces.

etcd-bot · 2017-12-14T23:34:01Z

Can one of the admins verify this patch?

etcd-bot · 2017-12-14T23:34:03Z

Can one of the admins verify this patch?

guilhem · 2017-12-15T11:41:09Z

JFYI: a docker image exist with this patch guilhem/etcd-operator

xiang90 · 2017-12-15T18:56:57Z

The idea seems fine.

But we need to add tests and related documentation to move this forward.

guilhem · 2017-12-15T21:20:13Z

Good to know! :)
That's the news I waited before putting more effort on tests and documentation.
I will do this this weekend or next week :)

xiang90 · 2017-12-15T21:21:56Z

@guilhem

Out of curiosity, what is your use case of having a cluster wide etcd operator? Why a per namespace one will not work for you?

guilhem · 2017-12-17T12:19:06Z

@xiang90

I want to save resources (pod/memory/watchers in API) and operations when having a lot of namespaces.

guilhem · 2017-12-17T16:51:47Z

My main concern is were to put configuration:

as I do in annotation
in Spec part
nowhere, can't mix etcd-operators

raoofm · 2017-12-18T17:32:28Z

@xiang90 @guilhem Can you please expand on the usecase? I didn't get it.

guilhem · 2017-12-18T17:38:42Z

@raoofm just want to avoid having to manage many etcdoperators
My operator need etcdoperator and I don't want to manage another operator in it.
I just want to install it as external dependency :)

xiang90 · 2018-01-17T22:33:08Z

@raoofm

more details here: #859

guilhem · 2018-01-18T14:09:33Z

@xiang90 I added some tests and documentation.
Don't hesitate to comment, or better, correct any mistake I made ;)

raoofm · 2018-01-18T14:10:39Z

@xiang90 awesome, makes sense. Thanks @guilhem

hongchaodeng · 2018-01-18T17:56:00Z

@etcd-bot ok to test

guilhem · 2018-01-31T10:07:02Z

I know news are quite huge...
But it's there anything I can do to make this PR merged?

xiang90 · 2018-02-01T18:18:36Z

The idea looks good for me. Defer to @hongchaodeng or @fanminshi or @hasbro17 for a actual code review.

hongchaodeng · 2018-02-01T21:59:17Z

example/deployment-clusterwide.yaml

+        image: quay.io/coreos/etcd-operator:v0.8.1
+        command:
+        - etcd-operator
+        - -all-namespaces


cluster-wide?

hongchaodeng · 2018-02-01T23:28:19Z

pkg/util/constants/constants.go

-	EnvOperatorPodNamespace = "MY_POD_NAMESPACE"
+	EnvOperatorPodName            = "MY_POD_NAME"
+	EnvOperatorPodNamespace       = "MY_POD_NAMESPACE"
+	EnvRestoreOperatorServiceName = "SERVICE_ADDR"


remove this EnvRestoreOperatorServiceName

oops, bad merge

hongchaodeng

Sorry for the review delay.
When you have change, please ping me on github.

hongchaodeng · 2018-02-04T19:04:12Z

doc/user/clusterwide.md

+
+## Install etcd operator
+
+etcd operator have to run with `-all-namespaces` arg option.


flag should be "cluster-wide"?

I try to be like kubectl option --all-namespaces=false.
But I wasn't really happy with this solution and will change it to cluster-wide :)

hongchaodeng · 2018-02-04T19:08:30Z

doc/user/clusterwide.md

+
+etcd operator have to run with `-all-namespaces` arg option.
+
+For example you can run:


There is no need to mention the command and install guide here.
Just give a reference to the file.

hongchaodeng · 2018-02-04T19:09:40Z

example/deployment-clusterwide.yaml

+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: etcd-operator-cluster-wide


There is too much duplicate and it is just hard to maintain for every upgrade.
I would recommend to add a comment to existing deployment file: https://github.com/coreos/etcd-operator/blob/master/example/deployment.yaml

hongchaodeng · 2018-02-04T19:12:05Z

pkg/controller/controller.go

@@ -64,6 +65,10 @@ func New(cfg Config) *Controller {
 func (c *Controller) handleClusterEvent(event *Event) error {
 	clus := event.Object

+	if !c.managed(clus) {
+		return fmt.Errorf("cluster (%s) isn't managed", clus.Name)


This will introduce unnecessary errors/warnings in the log for unmanaged EtcdCluster CRs. Should just ignore it.

Make this func return (ignored bool, err error) ?

@hongchaodeng tell me if my last commit is ok for you.
I will rebase if ok.

hongchaodeng · 2018-02-04T20:19:55Z

Test failed:

--- FAIL: TestHandleClusterEventClusterwideIgnored (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x16cc8bc]

goroutine 56 [running]:
testing.tRunner.func1(0xc4201605a0)
	/usr/local/go/src/testing/testing.go:711 +0x5d9
panic(0x17fab80, 0x23564a0)
	/usr/local/go/src/runtime/panic.go:491 +0x2a2
github.com/coreos/etcd-operator/pkg/controller.TestHandleClusterEventClusterwideIgnored(0xc4201605a0)
	/go/src/github.com/coreos/etcd-operator/pkg/controller/controller_test.go:112 +0x1ac
testing.tRunner(0xc4201605a0, 0x19ccc58)
	/usr/local/go/src/testing/testing.go:746 +0x16d
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x569
FAIL	github.com/coreos/etcd-operator/pkg/controller	0.553s

hongchaodeng · 2018-02-04T20:22:11Z

doc/user/clusterwide.md

+kind: "EtcdCluster"
+metadata:
+  name: "example-etcd-cluster"
+  annotations:


Can you add the comment in example file for this annotation instead of here?
We have plans to show the code files in the doc later: #1904

hongchaodeng · 2018-02-04T20:22:39Z

pkg/controller/controller.go

@@ -61,28 +62,32 @@ func New(cfg Config) *Controller {
 	}
 }

-func (c *Controller) handleClusterEvent(event *Event) error {
+func (c *Controller) handleClusterEvent(event *Event) (bool, error) {


add a comment for the returned value?

use named values instead (as you previously suggest :) )

hongchaodeng · 2018-02-04T20:24:05Z

Can you also rebase against master? Your commit is too old.

NatsuSir · 2018-02-05T00:04:38Z

I don't know what you say😂😂

hongchaodeng · 2018-02-05T04:31:39Z

Which part you don't understand?

hongchaodeng

CI failed due to unknown issue:

denied: Access denied.

hongchaodeng · 2018-02-12T17:19:57Z

example/example-etcd-cluster-clusterwide.yaml

+  name: "example-etcd-cluster"
+  # Adding this annotation make this cluster managed by clusterwide operators
+  # namespaced operators ignore it
+  annotations:


Can we combine this to example-etcd-cluster.yaml and add a comment for this?

oops my bad, was what I wanted to do

hongchaodeng · 2018-02-12T17:21:48Z

pkg/controller/controller.go

@@ -61,28 +62,37 @@ func New(cfg Config) *Controller {
 	}
 }

-func (c *Controller) handleClusterEvent(event *Event) error {
+func (c *Controller) handleClusterEvent(event *Event) (ignored bool, err error) {


add a comment for this func what returned values are

hongchaodeng · 2018-02-12T17:22:17Z

pkg/controller/controller.go

 	clus := event.Object

+	if !c.managed(clus) {
+		ignored = true


Prefer explicit return value:

return true, nil

hongchaodeng · 2018-02-12T17:26:59Z

pkg/util/constants/constants.go

@@ -30,4 +30,7 @@ const (

 	EnvOperatorPodName      = "MY_POD_NAME"
 	EnvOperatorPodNamespace = "MY_POD_NAMESPACE"
+
+	AnnotationScope       = "etcd.database.coreos.com/scope"


Can you also move these const into k8sutil:
https://github.com/coreos/etcd-operator/blob/master/pkg/util/k8sutil/k8sutil.go#L46

We want to get rid of this file.

remove AnnotationScope?

guilhem · 2018-02-13T14:40:22Z

@hongchaodeng any news about why tests are failing?
I don't see anything on my side to do to fix it :/

fanminshi · 2018-02-13T16:19:48Z

via https://jenkins-etcd-public.prod.coreos.systems/job/etcd-operator/3228/

17:48:25 Short-lived access for ['gcr.io', 'us.gcr.io', 'eu.gcr.io', 'asia.gcr.io', 'b.gcr.io', 'bucket.gcr.io', 'l.gcr.io', 'launcher.gcr.io', 'appengine.gcr.io', 'us-mirror.gcr.io', 'eu-mirror.gcr.io', 'asia-mirror.gcr.io', 'mirror.gcr.io'] configured.
17:48:25 building test-pod container...
17:48:39 unable to find image "sha256:c8c15988d8f837cf1318568c8eaf4a20915b1ae7db32158505f48a7adb670396"
17:48:40 namespace "e2e-etcd-operator-3228" deleted
17:48:40 Build step 'Execute shell' marked build as failure

fanminshi · 2018-02-13T16:20:24Z

@etcd-bot re-test this please reason: build image not found might due to jenkin flake.

fanminshi · 2018-02-13T16:21:34Z

@guilhem have you rebase this pr on top of the latest master?

guilhem · 2018-02-13T16:56:19Z

@fanminshi yes :/

fanminshi · 2018-02-13T17:01:28Z

@guilhem test passed. :)

fanminshi · 2018-02-13T17:02:55Z

README.md


- The etcd operator only manages the etcd cluster created in the same namespace. Users need to create multiple operators in different namespaces to manage etcd clusters in different namespaces.
+### Limitations


probably no need to have Limitations section if it is blank.

fanminshi · 2018-02-13T17:07:25Z

cmd/operator/main.go

@@ -66,6 +68,7 @@ func init() {
 	flag.BoolVar(&printVersion, "version", false, "Show version and quit")
 	flag.BoolVar(&createCRD, "create-crd", true, "The operator will not create the EtcdCluster CRD when this flag is set to false.")
 	flag.DurationVar(&gcInterval, "gc-interval", 10*time.Minute, "GC interval")
+	flag.BoolVar(&clusterWide, "cluster-wide", false, "The operator will watch clusters in all namespaces")


The operator will watch clusters in all namespaces => Enable operator to watch clusters in all namespaces?

Avoid the word Will ref: https://developers.google.com/style/tense

thanks for this link :)

fanminshi · 2018-02-13T17:08:53Z

pkg/util/constants/constants.go

@@ -30,4 +30,7 @@ const (

 	EnvOperatorPodName      = "MY_POD_NAME"
 	EnvOperatorPodNamespace = "MY_POD_NAMESPACE"
+
+	AnnotationScope       = "etcd.database.coreos.com/scope"


remove AnnotationScope?

fanminshi · 2018-02-13T17:09:02Z

pkg/util/constants/constants.go

@@ -30,4 +30,7 @@ const (

 	EnvOperatorPodName      = "MY_POD_NAME"
 	EnvOperatorPodNamespace = "MY_POD_NAMESPACE"
+
+	AnnotationScope       = "etcd.database.coreos.com/scope"
+	AnnotationClusterWide = "clusterwide"


remove AnnotationClusterWide?

hongchaodeng · 2018-02-13T17:05:47Z

doc/user/clusterwide.md

+
+More information in [install guide](doc/user/install_guide.md).
+
+## [Cluster resource example](example/example-etcd-cluster-clusterwide.yaml)


Probably not use a header?
Add a sentence:

See the example in ...

@guilhem fix this?

@fanminshi hehe was in another batch ;)

hongchaodeng · 2018-02-13T17:06:22Z

pkg/controller/controller.go

@@ -61,28 +62,37 @@ func New(cfg Config) *Controller {
 	}
 }

-func (c *Controller) handleClusterEvent(event *Event) error {
+// handleClusterEvent returns ignored true if cluster isn't managed by this instance.


returns ~~ignored~~ true if cluster is ignored (not managed) by this instance.

@guilhem fix this?

hongchaodeng · 2018-02-13T17:10:51Z

pkg/controller/controller.go

@@ -61,28 +62,37 @@ func New(cfg Config) *Controller {
 	}
 }

-func (c *Controller) handleClusterEvent(event *Event) error {
+// handleClusterEvent returns ignored true if cluster isn't managed by this instance.
+func (c *Controller) handleClusterEvent(event *Event) (ignored bool, err error) {


Sorry I wasn't clear before.
Prefer ALL returns to be explicit about return values.

handleClusterEvent(event *Event) (bool, error) { ... return false, nil ... return false, fmt.Errorf(...)

hehe no problem :)

hongchaodeng · 2018-02-13T17:11:34Z

@guilhem Thanks for your patience and persistence. Only a few minor comments. Overall it looks good.

guilhem · 2018-02-13T17:38:45Z

I "think" I fixed your comments :)
But no problem for all the time this PR take me, I know how it's hard to accept PR with new feature from community.
And it is my first commit in your project... I also have a lot to learn.
Thank you for your patience.

If all is ok, I will squash all my PR to have something clean ;)

hongchaodeng · 2018-02-13T17:40:55Z

I know how it's hard to accept PR with new feature from community.

Thanks again for your patience.
We try to embrace the community so there are a lot of things for us to learn. We will be more open and let us know what we did not do well.

If all is ok, I will squash all my PR to have something clean ;)

No worries. We will do squash when we merge.

hongchaodeng · 2018-02-13T17:42:41Z

LGTM

fanminshi · 2018-02-13T17:43:08Z

doc/user/clusterwide.md

+
+More information in [install guide](doc/user/install_guide.md).
+
+See the example in [example/example-etcd-cluster.yaml](example/example-etcd-cluster.yaml)


the link in [example/example-etcd-cluster.yaml](example/example-etcd-cluster.yaml) doesn't work. you might want to try ~~~[example/example-etcd-cluster.yaml](../../blob/master/example/example-etcd-cluster.yaml)~~~

edit: use [example/example-etcd-cluster.yaml](../../example/example-etcd-cluster.yaml)

Right. Should be

[example/example-etcd-cluster.yaml](../../example/example-etcd-cluster.yaml)

@hongchaodeng ~~~this [example/example-etcd-cluster.yaml](../../example/example-etcd-cluster.yaml) didn't work for me.~~~

edit: use [example/example-etcd-cluster.yaml](../../example/example-etcd-cluster.yaml)

I'm trying with ../../

yeah, ../../ is the right approach.

fanminshi · 2018-02-13T17:53:18Z

lgtm

guilhem · 2018-02-13T17:55:06Z

Last thing you may add is a "BETA" mention in cmd help.

But you can do it directly on my own branch, feel free ;)

guilhem · 2018-02-13T18:28:48Z

🎉 👍

Thank you all :)

guilhem force-pushed the clusterwide branch 2 times, most recently from ad657e7 to dde9475 Compare January 18, 2018 14:01

guilhem changed the title ~~[WIP] operator: Add option to act as cluster wide~~ operator: Add option to act as cluster wide Jan 31, 2018

hongchaodeng reviewed Feb 1, 2018

View reviewed changes

guilhem force-pushed the clusterwide branch 2 times, most recently from 6065fd2 to d06117a Compare February 3, 2018 16:19

hongchaodeng reviewed Feb 4, 2018

View reviewed changes

guilhem force-pushed the clusterwide branch from d06117a to 0b71f61 Compare February 4, 2018 19:43

hongchaodeng reviewed Feb 4, 2018

View reviewed changes

hongchaodeng reviewed Feb 12, 2018

View reviewed changes

guilhem added 3 commits February 12, 2018 18:40

Mix examples

1bc23a1

Explicit return and doc for handleClusterEvent

605ef99

Move constants to k8sutil

8ebe2f1

fanminshi reviewed Feb 13, 2018

View reviewed changes

hongchaodeng reviewed Feb 13, 2018

View reviewed changes

guilhem added 6 commits February 13, 2018 18:15

Remove unused constants

2b2920b

Remove useless limitation section

18c0731

Command help rephrasing

e03e637

Clusterwide example

a4f4b06

handleClusterEvent help

fd0e59b

Explicit return values for handleClusterEvent

cc6cd9e

fanminshi reviewed Feb 13, 2018

View reviewed changes

Fix relative link to example file

11ce31f

hongchaodeng merged commit 0a4c2c1 into coreos:master Feb 13, 2018

gjcarneiro mentioned this pull request Dec 10, 2018

Support cross-namespace cluster operation #859

Open


		## Install etcd operator

		etcd operator have to run with `-all-namespaces` arg option.


		etcd operator have to run with `-all-namespaces` arg option.

		For example you can run:


		- The etcd operator only manages the etcd cluster created in the same namespace. Users need to create multiple operators in different namespaces to manage etcd clusters in different namespaces.
		### Limitations


		More information in [install guide](doc/user/install_guide.md).

		## [Cluster resource example](example/example-etcd-cluster-clusterwide.yaml)


		More information in [install guide](doc/user/install_guide.md).

		See the example in [example/example-etcd-cluster.yaml](example/example-etcd-cluster.yaml)

operator: Add option to act as cluster wide #1777

operator: Add option to act as cluster wide #1777

Conversation

guilhem commented Dec 14, 2017

etcd-bot commented Dec 14, 2017

etcd-bot commented Dec 14, 2017

guilhem commented Dec 15, 2017

xiang90 commented Dec 15, 2017

guilhem commented Dec 15, 2017

xiang90 commented Dec 15, 2017

guilhem commented Dec 17, 2017 • edited Loading

guilhem commented Dec 17, 2017

raoofm commented Dec 18, 2017

guilhem commented Dec 18, 2017

xiang90 commented Jan 17, 2018

guilhem commented Jan 18, 2018

raoofm commented Jan 18, 2018

hongchaodeng commented Jan 18, 2018

guilhem commented Jan 31, 2018

xiang90 commented Feb 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guilhem Feb 3, 2018 • edited Loading

Choose a reason for hiding this comment

hongchaodeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hongchaodeng commented Feb 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hongchaodeng commented Feb 4, 2018

NatsuSir commented Feb 5, 2018

hongchaodeng commented Feb 5, 2018

hongchaodeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guilhem commented Feb 13, 2018

fanminshi commented Feb 13, 2018

fanminshi commented Feb 13, 2018

fanminshi commented Feb 13, 2018

guilhem commented Feb 13, 2018

fanminshi commented Feb 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hongchaodeng commented Feb 13, 2018

guilhem commented Feb 13, 2018

hongchaodeng commented Feb 13, 2018

hongchaodeng commented Feb 13, 2018

fanminshi Feb 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanminshi Feb 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanminshi commented Feb 13, 2018

guilhem commented Dec 17, 2017 •

edited

Loading

guilhem Feb 3, 2018 •

edited

Loading

fanminshi Feb 13, 2018 •

edited

Loading

fanminshi Feb 13, 2018 •

edited

Loading