Allow adding container labels for kind nodes

[hellt]

Hi @aojea @BenTheElder
I would like to see if you consider the enhancement request to be able to provide container metadata via container labels a feasible request.

In containerlab (containerlab.dev) we would like to add support for spinning up kind clusters as part of a more extensive containerlab lab. In containerlab, the containers that are "orchestrated" by it, tagged accordingly so that additional tools around containerlab can filter containers and provide some automation around it.

When looking into kind API we didn't find a way to add metadata to kind containers (these are worker and master nodes). Is is possible to enhance API and allow providing tags to kind containers?

[aojea]

If by container metadata via container labels you refer to the kindnode containers, they already have information necessary to identify them

            "Labels": {
                "io.x-k8s.kind.cluster": "kind",
                "io.x-k8s.kind.role": "control-plane"
            },

does it work for you?

[hellt]

@aojea, the enhancement request is about being able to assign custom labels to the ones you do by default.

[aojea]

I can see how that can be difficult to support with docker/podman duality
Do you have time next week and we talk over this, this way I can use the opportunity to learn more about the project

[hellt]

@aojea yes, we can sync up in twitter DMs to choose some time for a sync

[BenTheElder]

@aojea what came of this?

Previously we've rejected this feature. We're trying to keep the options to minimal essential ones as the project is tricky enough to maintain already and there's existing labels for identifying nodes from any particular kind cluster, as well as exported APIs and commands for listing nodes which can then be interacted with via docker/podman.

[logikone]

I have a use case where we're creating multiple kind clusters in a CI container for testing. The CI container has the hosts docker.sock mounted in the container. Sometimes when something fails and the job exits you can end up having some dangling kind clusters remaining on the CI host. The job containers that are created by the runner have a bunch of labels on them identifying which repo, job, etc the container belongs to. I'd like to be able to put some of those same labels on the containers that kind creates so I can better identify dangling containers.

I'd be happy to contribute in my free time if this is a feature that you'd like to see added. Thanks!

[BenTheElder]

can you use a computed cluster name instead? (container names are computed from cluster name and are stable + predictable and labeled with the cluster name already)

[logikone]

Ya, that's what I'm doing currently 👍

[icefed]

this feature has been rejected, try my commit if you really want this, https://github.com/icefed/kind/tree/container-label

[whg517]

I integrated kind into the project for testing, and in the test environment, I wanted the container created by kind to support adding container label parameters. But what we're seeing now is the community rejecting this feature.

I tried to research other tools and found that minikube does not have this feature. But k3d (which deploys k3s in docker) has implemented this similar capability as far back as 2021. Try replacing the toolchain if possible.

eg:

  runtime: # runtime (docker) specific options
    gpuRequest: all # same as `--gpus all`
    labels:
      - label: bar=baz # same as `--runtime-label 'bar=baz@agent:1'` -> this results in a runtime (docker) container label
        nodeFilters:
          - agent:1
    ulimits:
      - name: nofile
        soft: 26677
        hard: 26677

I hope my information will be helpful to anyone who searches for this question in the future.

What I would say to this question is this: according to BenTheElder, the community should only add basic features, not useful ones. However, for the user, there is no entry to pass docker parameters, which makes the user helpless. Like the k8s label, kind provides nodes.labels and labels can also be added using kubeadmConfigPatches. On this question, obviously, there are some absolutes.

[BenTheElder]

I wanted the container created by kind to support adding container label parameters.

For what purpose?

What I would say to this question is this: according to #3709 (comment), the community should only add basic features, not useful ones.

No, we add features that are essential, because every feature is another thing to support.

So far we have not seen any usage where you must add labels to the node containers. There are other alternatives.

Regarding GPUs: https://github.com/klueska/kind-with-gpus-examples

We've been talking to folks in SIG Node / DRA / Nvidia (primarily in #3290 but also in other venues).

However, for the user, there is no entry to pass docker parameters, which makes the user helpless.

Passing arbitrary docker parameters is a support nightmare and generally not required to use kind succesfully.

We generally aim to support portable standard Kubernetes APIs wherever possible (e.g. recently NetworkPolicy, instead of only conformant APIs)

Like the k8s label, kind provides nodes.labels and labels can also be added using kubeadmConfigPatches. On this question, obviously, there are some absolutes.

Right ... there is support for setting node labels, and nodes are 1:1 with container names, so you can list them with kubectl

[whg517]

First of all thank you for your positive response and in-depth discussion. Whether this feature is accepted or not in the end, I think it makes sense.

Secondly, I would like to express some of my own views. Admittedly, kind is primarily focused on the k8s, making it easier for users to manage the k8s. But kind is deployed based on docker (and has experimental support for other runtimes, of course), and in the case of deep use, it is hoped that you can better manage the running kind container.

For example, my current needs:

I have a lot of docker services running in my environment right now, based on docker compose.

In the docker environment, I chose traefik as the reverse proxy tool for the docker container service. It takes 80 and 443. traefik In a docker environment, service discovery can be performed based on labels of docker containers and proxy traffic can be dynamically configured. This feature is similar to how k8s ingress now uses cert-manage to automatically generate certificates.

This way I can proxy traefik traffic into the ingress created by kind, and I can easily manage traffic access by creating multiple kinds.

Since I was unable to dynamically configure traefik for the kind environment through the tag traefik, I could only manage traefik configuration through the profile. This is very inflexible.

summary:

Maybe my needs are very niche, even to say that no one else does what I do. You may also say that the k8s completely meets my needs, why use a container. There are different technical cost considerations in different environments.

I just use docker container labels to complete a small transformation of the existing technology stack, just like traefik uses labels to realize the service discovery function of docker environment.

docker labels may also have other applications, including basic management of grouping, filtering, etc.

Finally, I put forward my ideas on perpetuating someone else's problem, and I realize that this is a niche need, so I have now resorted to manually adding profiles to manage my existing environment.