Create an example of a python custom controller

[dimberman]

I am attempting to create an example custom controller using the python-client, but I cant seem to find any Informer (https://github.com/kubernetes/client-go/tree/master/informers) endpoint in client python. Am I missing this endpoint or has this not been added to the library yet?

If it does exist, I would very much like to contribute an example to encourage creation of custom controllers.

cc: @foxish @mbohlool

[mbohlool]

Informer is a client-go thing not a kubernetes API thing (as it is in client-go repo). An informer is basically a watch (example in python). Actually as far as I can tell, there is no Informer in kubernetes but sharedInformer that is a more advanced concept that we do not have in python client and if anybody wants to implement that, I would spend dedicated review time for them :)

[karmab]

is it enough to leverage the watch code ? what is specific to the shared informer?

[mbohlool]

shared informers has cache and connection sharing.

[mattmoor]

FYI I am adding a little sample in this PR to bazelbuild/rules_k8s.

[mattmoor]

Also I'm very open to feedback :)

[karmab]

@mattmoor thanks, this is exactly what i had in mind and was coming with (testing with tpr instead of crd but crds are the way to go)!

[karmab]

@mbohlool ok, if i were to implement shared informers in this python library, what would be the way to go ?

[mbohlool]

It would be basically an extension to Watch class to cache and share watch calls. It would be amazing if the interface of the class does not change and it would be smart enough to share an existing watch connection.

[karmab]

@mattmoor @dimberman i've also created https://github.com/karmab/samplecontroller if you wanna have a look

[mattmoor]

So my controller deployments aren't living terribly long. I was playing with a derivative of my example, and it is very slowly crash looping.

I'm not sure of the exact symptom, but I have seen messages about connection exhaustion (maybe a leak in the watch logic?).

Is there a more stable pattern than the one I'm using, or any useful debugging trick to better identify what's going on?

[karmab]

the following does the trick for me: ``` while True: if resource_version is None: stream = watch.Watch().stream(crds.list_cluster_custom_object, DOMAIN, "v1", "guitars") else: stream = watch.Watch().stream(crds.list_cluster_custom_object, DOMAIN, "v1", "guitars", resource_version=resource_version) for event in stream: obj = event["object"] operation = event['type'] spec = obj.get("spec") if not spec: print("No 'spec' in object, skipping event: %s" % json.dumps(obj, indent=1)) continue metadata = obj.get("metadata") resource_version = metadata['resourceVersion'] name = metadata['name'] print("Handling %s on %s" % (operation, name)) done = spec.get("review", False) if done: continue review_guitar(crds, obj) ```

[mattmoor]

@karmab I assume you mean the combination of the outer while True and the resource_version logic to avoid reprocessing? Is resource_version from a single object a reasonable thing to rely upon?

[karmab]

i was told by mehdi this was the way to go :)

[mattmoor]

@karmab I think that is more verbose than you need, try:

    resource_version = ''
    while True:
        stream = watch.Watch().stream(crds.list_cluster_custom_object,
                                      DOMAIN, VERSION, PLURAL,
                                      resource_version=resource_version)
        for event in stream:
            try:
                obj = event["object"]
                # Normal processing...

                # Configure where to resume streaming.                                                                                                                                           
                metadata = obj.get("metadata")
                if metadata:
                    resource_version = metadata["resourceVersion"]
            except:
                logging.exception("Error handling event")

I'm going to leave that running for a while and see if it does the trick :)

[karmab]

@mattmoor you are right, i changed my controller

[marcellodesales]

@karmab @mattmoor @mbohlool

My Requirement

I'm writing a controller that:

• It watches for the instances of itself (A)
• When (A) is created, it must create a related CRD for controller (B), (C), and (D)
• Beyond creation, (A) must watch the Status changes of (B), (C), (D)
• When (B), (C), (D) reaches the desirable state, (A) needs to report on that

Questions

• So, should I create 4 different watchers?

  • I was brought here because of the informers API as well (suggested by @liggitt on sig-api-machinery)

• Is there anything like the following?

                          watch.Watch().stream(crds.list_cluster_custom_object,
                                      [ 
                                        { DOMAIN_A, VERSION_A, PLURAL_A}, 
                                        { DOMAIN_B, VERSION_B, PLURAL_B},
                                        { DOMAIN_C, VERSION_C, PLURAL_C}, 
                                        { DOMAIN_D, VERSION_D, PLURAL_D},
                                      ])```

         for event in stream:
            try:
                type = object.type()
                switch (type)
                   case (A): A_handler(obj)
                   case (B): B_handler(obj)
                   case (C): C_handler(obj)
                   case (D): D_handler(obj)                  
            except:
                logging.exception("Error handling event")

[mattmoor]

I'm not aware of a way to do that.

You can certainly use ~4 watches to accomplish this, but I'm also curious if there's a better way :)

[sebgoa]

@mattmoor I went back to your example controller listed above, it does not seem to restart at the correct resourceVersion. It keeps restarting from the first CRD object created.

Have you seen this behavior before ?

[mattmoor]

No, and I've been predominantly working with Go-based controllers lately.

[mbohlool]

dup of #30?

[akaihola]

Tip: the Metacontroller framework may help in some cases when writing custom controllers.

With Metacontroller, you can write your controller logic in any language.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.