✨ Label interruptible nodes #3668

alexander-demicev · 2020-09-21T12:35:28Z

What this PR does / why we need it:

Transfer machine labels to the node. The use-case is described here #3504.

The transfer of machine labels to the node is similar to setting Noderef. Important thing to mention is that only labels with "cluster.x-k8s.io" prefix are transferred.

This PR should help with implementing the termination handler for spot instances - #3528

Happy to hear what others think about this change.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3504

k8s-ci-robot · 2020-09-21T12:35:36Z

Hi @alexander-demichev. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

JoelSpeed

Looks good, couple of small comments but I have nothing major to add

JoelSpeed · 2020-09-21T12:50:08Z

controllers/machine_controller_nodelabels.go

@@ -0,0 +1,79 @@
+/*
+Copyright 2019 The Kubernetes Authors.


Suggested change

Copyright 2019 The Kubernetes Authors.

Copyright 2020 The Kubernetes Authors.

JoelSpeed · 2020-09-21T12:51:41Z

controllers/machine_controller_nodelabels.go

+
+	for key, value := range machineLabels {
+		// sync only labels with "cluster.x-k8s.io" prefix, so users can understand where labels come from
+		if strings.HasPrefix(key, "cluster.x-k8s.io") {


Is there a constant that can be used for this somewhere already?

enxebre · 2020-09-21T13:04:16Z

is this intending to supersede kubernetes-sigs/cluster-api-provider-aws#1876?
Overall seems reasonable to me.

ncdc · 2020-09-21T13:06:03Z

We have had previous requests about setting labels on nodes - see #458 and #493. The earlier consensus was that Cluster API will not reconcile node labels, but the kubeadm bootstrapper can set the initial values for a node's labels using KubeadmConfig.spec.{initConfiguration,joinConfiguration}.nodeRegistration.kubeletExtraArgs["node-labels"]. Continuing maintenance of a node's labels is better left to some other actor.

I do think it could be useful to adjust Machine.spec to add a field such as initialNodeLabels and then update the contract for bootstrap providers to use the values in that field to set the node's initial labels.

We will need to have further discussion on this before we can proceed, given the previous decision not to do this.

/hold

cc @kubernetes-sigs/cluster-api-maintainers

alexander-demicev · 2020-09-21T13:19:42Z

@ncdc Will list of labels be additive if we use kubeadm boostraper?

ncdc · 2020-09-21T13:23:50Z

@alexander-demichev they're the initial set of labels. See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ (search for --node-labels) for more information.

enxebre · 2020-09-21T13:29:12Z

@ncdc @alexander-demichev is there a way to intrinsically infer a given machine is for preemptible so the node can be labeled based on this as an implementation detail rather than generically syncing labels?
E.g Couldn't this be done by each specific provider implementation when they see their preemptible API fields set?

ncdc · 2020-09-21T13:49:51Z

@enxebre ah ok now I see what the desired behavior is here around labels & preemptible/spot instances. If we're looking for a behavior that is consistent across infrastructure providers, I'd say it's most appropriate to implement that behavior in core Cluster API, or at the very least, define the contract portion in the core.

A behavior such as "if a Machine is preemptible, make sure its Node has the label cluster.x-k8s.io/interruptible-instance" could be implemented

as one of the initial node labels set by the bootstrapper when registering the node
via reconciliation in the core Cluster API machine controller
via something outside of / on top of Cluster API

Option 1 (bootstrapper) requires every Cluster API bootstrap provider to support setting a node's initial labels. This seems reasonable to me. One downside is that if the interruptible label were somehow removed from the Node, nothing would add it back.

Option 2 (reconciliation) should probably only support syncing a very specific set of labels to a Node, namely this interruptible label for the time being. I wouldn't want to open this up to user-supplied labels, for the reasons previously stated in the other issues I linked above. But I do think it makes sense to sync labels defined by Cluster API that influence the behavior of the overall system. One downside to this option is that there would be a short period of time during which the interruptible label would be absent from the new Node. One upside is that the interruptible label would be re-added to the Node if it were somehow removed.

Option 3 (some external entity) I mentioned for completeness, but I don't think it's appropriate because Cluster API should be or is knowledgable that this is an interruptible machine.

For this specific use case -- ensuring a specific interruptible label is present on a Node -- I think it's appropriate to do both 1 and 2 together. 1 is already available in the kubeadm bootstrap provider and may be available in others. I do think a future API improvement could be what I mentioned in a previous comment - adding an initialNodeLabels spec field to Machine and having bootstrap providers use that. Option 2 would be new, and restricted solely to syncing this new interruptible label.

I also wonder if it would make sense to consider adding another new spec field to Machine - interruptible (name TBD) - to indicate the user understands this could be deleted by the cloud provider. This would simplify the burden on the user: they'd set spec.interruptible, and the machine reconciler would take care of setting the appropriate interruptible label (in other words, the user wouldn't have to spell out the full key name for the label and hope they don't have a typo). Adding a new interruptible field to the Machine spec has an added benefit for infrastructure providers - they could validate the corresponding InfraMachine (AWSMachine/AzureMachine/GCPMachine/etc) is configured with the interruptible field(s) appropriately. This idea could be implemented later, however - it's not required for the label syncing to function as desired.

enxebre · 2020-09-21T14:09:20Z

This all make sense to me @ncdc. As a user to define an interruptible machine you just need to set it in one place today, the infra template. I think we should aim to keep it that way so there's no room for partial supported scenarios. To that end I wonder if it'd make sense to let interruptible to be in the status rather than the spec (so we own it, not the user) and let option 2 to reconcile the node label based on this field being populated.

ncdc · 2020-09-21T14:16:44Z

I wonder if it'd make sense to let interruptible to be in the status rather than the spec (so we own it, not the user) and let option 2 to reconcile the node label based on this field being populated.

I like this idea. As I was writing my spec.interruptible suggestion, I was wondering if it would be an unnecessary extra step. Brainstorming on flow:

User creates InfraMachine with whatever spec field(s) are required for that provider to indicate it's interruptible
Infra provider sets InfraMachine.status.interruptible=true
Machine controller syncs InfraMachine.status.interruptible to Machine.status.interruptible
(optional) Bootstrap provider somehow waits for the above sync so it can use Machine.status.interruptible when creating the bootstrap data secret (this needs more thinking)
Machine controller ensures the interruptible label is always present on the Node if Machine.status.interruptible is true

CecileRobertMichon · 2020-09-21T15:26:57Z

If we do move forward with this approach, can we also make sure that the same can be applied to MachinePools? I think the approach @ncdc describes above should work at first glance but I want to make sure we explicitly include it in the design (and even maybe implementation if it's straight forward).

vincepri · 2020-09-21T16:01:57Z

I'd hold off on adding more status syncing between infrastructure provider and the machine controller. From a high level perspective, Cluster API itself doesn't know that a Machine is preemptible.

The approach described above could potentially cause a dependency loop between the controllers. One alternative approach, which should be discussed in a proposal, is to add a condition to the InfrastructureMachine that communicates "ready to receive bootstrap data". Bootstrap providers would have to wait for this condition to appear before generating the bootstrap configuration and continue execution.

This would be a breaking change, which definitely requires some more thinking / design proposal. I'd suggest to open an RFE issue and gather use cases and proposals.

How does that sound?

/milestone v0.4.0

enxebre · 2020-09-22T08:59:39Z

One alternative approach, which should be discussed in a proposal, is to add a condition to the InfrastructureMachine that communicates "ready to receive bootstrap data"...

I see this as something orthogonal and just a nice to have for the interruptible instances story. To cover the interruptible instances story end to end as a first iteration I see no concerns to proceed as with 1, 2, 3 and 5.
In parallel we can have a separate wider discussion for 4 and other use cases. wdyt?

vincepri · 2020-09-22T12:55:40Z

In parallel we can have a separate wider discussion for 4 and other use cases. wdyt?

If the only goal is to update node labels after the Node is up, I think that could be doable. That said, adding a new synced status field goes somewhat against our different goal to move completely to conditions, and remove the current use of boolean fields.

vincepri · 2020-09-22T13:11:02Z

Is this something you'd like to do in v0.3.x, or can we wait until v0.4.0 and have an RFE / small proposal in an issue?

ncdc · 2020-09-22T17:22:24Z

I'd like to revise my previous suggestion and remove the part about adding interruptible to machine.status and syncing it from the infra machine to the machine. We can still achieve what we need by having the machine controller look at the infra machine's status (for interruptible), and avoid syncing that information to the machine.

remove the current use of boolean fields.

Have we stated as a goal to never use any more booleans in status? Something like status.interruptible=true/false feels different than a condition, as it's data that's derived from the spec and isn't something that will ever change for the life of the infra machine. It seems ok to have a status boolean for this?

re timing, I propose we do this once we open main for new features & breaking changes for v0.4.0. Do you want a CAEP for this (contract change for infra providers to add status.interruptible (name/location TBD), machine controller updated to sync just this one label to interruptible nodes), or can we repurpose #3504 for the details?

alexander-demicev · 2020-09-25T13:24:27Z

@ncdc I think changing the description of #3504 and giving more context should be fine.

alexander-demicev · 2020-09-30T13:42:01Z

@ncdc @vincepri PTAL #3504 and let's move the discussion there.

alexander-demicev · 2020-11-03T19:04:23Z

/test pull-cluster-api-test-main

k8s-ci-robot · 2020-11-03T19:04:37Z

@alexander-demichev: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-cluster-api-test-main

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

alexander-demicev · 2020-11-03T19:06:13Z

@ncdc Hi, I tried to address all comments, PTAL

vincepri · 2020-11-06T16:36:37Z

controllers/machine_controller_node_labels.go

+	// Check that the Machine hasn't been deleted or in the process.
+	if !machine.DeletionTimestamp.IsZero() {
+		return ctrl.Result{}, nil
+	}
+
+	// Check that the Machine has a NodeRef.
+	if machine.Status.NodeRef == nil {
+		return ctrl.Result{}, nil
+	}


Can we merge these two together?

vincepri · 2020-11-06T16:37:05Z

controllers/machine_controller_node_labels.go

+	err = r.setInterruptibleNodeLabel(ctx, remoteClient, machine.Status.NodeRef.Name)
+	if err != nil {


Suggested change

err = r.setInterruptibleNodeLabel(ctx, remoteClient, machine.Status.NodeRef.Name)

if err != nil {

if err := r.setInterruptibleNodeLabel(ctx, remoteClient, machine.Status.NodeRef.Name); err != nil {

vincepri · 2020-11-06T16:37:10Z

controllers/machine_controller_node_labels.go

+	if err != nil {
+		return ctrl.Result{}, fmt.Errorf("failed to get interruptible status from infrastructure provider for Machine %q in namespace %q: %w", machine.Name, machine.Namespace, err)
+	}
+


Suggested change

vincepri · 2020-11-06T16:37:38Z

controllers/machine_controller_node_labels.go

+		return ctrl.Result{}, err
+	}
+
+	logger.Info("Set interruptible label to Machine's Node", "nodename", machine.Status.NodeRef.Name)


Suggested change

logger.Info("Set interruptible label to Machine's Node", "nodename", machine.Status.NodeRef.Name)

logger.V(3).Info("Set interruptible label to Machine's Node", "nodename", machine.Status.NodeRef.Name)

vincepri · 2020-11-06T16:38:16Z

controllers/machine_controller_node_labels.go

+
+	err := remoteClient.Get(ctx, client.ObjectKey{Name: nodeName}, node)
+	if err != nil {


Suggested change

err := remoteClient.Get(ctx, client.ObjectKey{Name: nodeName}, node)

if err != nil {

if err := remoteClient.Get(ctx, client.ObjectKey{Name: nodeName}, node); err != nil {

vincepri · 2020-11-06T16:38:42Z

controllers/machine_controller_node_labels.go

+	if err := patchHelper.Patch(ctx, node); err != nil {
+		return err
+	}
+
+	return nil
+}


Suggested change

if err := patchHelper.Patch(ctx, node); err != nil {

return err

}

return nil

}

return patchHelper.Patch(ctx, node)

}

controllers/machine_controller_node_labels.go

ncdc · 2020-11-16T16:19:34Z

controllers/machine_controller_node_labels.go

+	// Get interruptible instance status from the infrastructure provider.
+	interruptible, _, err := unstructured.NestedBool(infra.Object, "status", "interruptible")
+	if err != nil {
+		return ctrl.Result{}, fmt.Errorf("failed to get interruptible status from infrastructure provider for Machine %q in namespace %q: %w", machine.Name, machine.Namespace, err)


@vincepri what do you think about logging here and returning early instead of returning an error? If we return an error, it means we will start exponential backoff retries. I assume that unless the infra machine changes, continuing to retry the same object will yield the same failure each time. And, given we are already watching for infra machine changes, the machine will get requeued whenever the infra machine is updated.

We definitely can. That said, these errors are uncommon code paths where we shouldn't get into unless something really wrong happened. Most of the time, functions like NestedBool are going to return regardless if the value exists or not, and in this case we use the default value for the boolean if the json path (status.interruptible) doesn't exists. The only way for this call to fail is if the infra.Object isn't well formed, which is unlikely in most cases, and it would probably hint to data corruption

Exactly, and if it's malformed data, exponential backoff isn't going to be useful here.

That makes sense, thank you.

ncdc · 2020-11-16T16:20:28Z

controllers/machine_controller_node_labels.go

+)
+
+func (r *MachineReconciler) reconcileInterruptibleNodeLabel(ctx context.Context, cluster *clusterv1.Cluster, machine *clusterv1.Machine) (ctrl.Result, error) {
+	logger := ctrl.LoggerFrom(ctx)


nit: consider moving this down to immediately before you first use the logger

ncdc · 2020-11-16T16:22:31Z

controllers/machine_controller_node_labels.go

+		return err
+	}
+
+	patchHelper, err := patch.NewHelper(node, r.Client)


Can you move this down so we only instantiate if we know we need to patch?

controllers/machine_controller_node_labels.go

controllers/machine_controller_node_labels_test.go

ncdc · 2020-11-16T16:25:10Z

controllers/machine_controller_node_labels_test.go

+	// Check if node gets interruptible label
+	g.Eventually(func() bool {
+		updatedNode := &corev1.Node{}
+		err := testEnv.Get(ctx, client.ObjectKey{Name: node.Name, Namespace: ns.Name}, updatedNode)


Suggested change

err := testEnv.Get(ctx, client.ObjectKey{Name: node.Name, Namespace: ns.Name}, updatedNode)

err := testEnv.Get(ctx, client.ObjectKey{Name: node.Name}, updatedNode)

vincepri · 2020-11-16T18:11:49Z

/ok-to-test

ncdc · 2020-11-20T16:02:44Z

controllers/machine_controller_node_labels_test.go

+		g.Expect(testEnv.Cleanup(ctx, do...)).To(Succeed())
+	}(cluster, node, infraMachine, machine)
+
+	g.Expect(clusterv1.AddToScheme(scheme.Scheme)).To(Succeed())


nit: this line isn't necessary because it's already done in

cluster-api/test/helpers/envtest.go

Lines 76 to 84 in ace4e24

// Calculate the scheme.

utilruntime.Must(apiextensionsv1.AddToScheme(scheme.Scheme))

utilruntime.Must(clusterv1.AddToScheme(scheme.Scheme))

utilruntime.Must(bootstrapv1.AddToScheme(scheme.Scheme))

utilruntime.Must(expv1.AddToScheme(scheme.Scheme))

utilruntime.Must(crs.AddToScheme(scheme.Scheme))

utilruntime.Must(addonv1.AddToScheme(scheme.Scheme))

utilruntime.Must(kcpv1.AddToScheme(scheme.Scheme))

utilruntime.Must(admissionv1.AddToScheme(scheme.Scheme))

.

vincepri · 2020-11-20T17:22:43Z

Can we rename this PR and change the description to better capture that it only supports interruptible label?

ncdc · 2020-11-20T19:38:23Z

/retitle ✨ Label interruptible nodes
/lgtm
/hold cancel
/assign @vincepri

ncdc · 2020-11-20T19:38:55Z

/retest

vincepri

One minor comment, otherwise LGTM

vincepri · 2020-11-20T20:10:52Z

controllers/machine_controller_node_labels.go

+		return ctrl.Result{}, err
+	}
+
+	logger := ctrl.LoggerFrom(ctx)


Suggested change

logger := ctrl.LoggerFrom(ctx)

log := ctrl.LoggerFrom(ctx)

To be consistent with the rest of the codebase

vincepri · 2020-11-20T20:11:16Z

controllers/machine_controller_node_labels.go

+
+func (r *MachineReconciler) setInterruptibleNodeLabel(ctx context.Context, remoteClient client.Client, nodeName string) error {
+	node := &apicorev1.Node{}
+


Suggested change

alexander-demicev · 2020-11-23T12:34:57Z

@ncdc @vincepri All fixed, thanks for reviews!

vincepri

/approve
/lgtm

k8s-ci-robot · 2020-11-30T03:56:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

alexander-demicev · 2020-12-01T11:29:53Z

/retest

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 21, 2020

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 21, 2020

k8s-ci-robot requested review from JoelSpeed and vincepri September 21, 2020 12:35

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 21, 2020

JoelSpeed reviewed Sep 21, 2020

View reviewed changes

alexander-demicev mentioned this pull request Sep 21, 2020

[WIP] Add ability to sync aws machine and node labels kubernetes-sigs/cluster-api-provider-aws#1876

Closed

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 21, 2020

k8s-ci-robot added this to the v0.4.0 milestone Sep 21, 2020

randomvariable mentioned this pull request Sep 21, 2020

Spot Instance interruption notice support kubernetes-sigs/cluster-api-provider-aws#1899

Closed

vincepri mentioned this pull request Sep 24, 2020

📖 Windows Support CAEP #3616

Merged

Arvinderpal mentioned this pull request Sep 28, 2020

Support syncing a set of labels from MachineDeployment/MachineSet/Machine to Nodes #493

Closed

alexander-demicev mentioned this pull request Sep 30, 2020

Set interruptible label to nodes #3504

Closed

vincepri requested changes Nov 6, 2020

View reviewed changes

ncdc reviewed Nov 16, 2020

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 16, 2020

ncdc reviewed Nov 20, 2020

View reviewed changes

k8s-ci-robot assigned vincepri Nov 20, 2020

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 20, 2020

k8s-ci-robot changed the title ~~✨ Transfer machine labels to node~~ ✨ Label interruptible nodes Nov 20, 2020

k8s-ci-robot assigned ncdc Nov 20, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 20, 2020

vincepri reviewed Nov 20, 2020

View reviewed changes

Add interruptible label to machine node

515a30a

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 23, 2020

vincepri approved these changes Nov 30, 2020

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 30, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 30, 2020

k8s-ci-robot merged commit c04d34e into kubernetes-sigs:master Dec 1, 2020

fabriziopandini mentioned this pull request Feb 5, 2021

How to add label on the worker nodes on the fly? #4147

Closed

alexander-demicev mentioned this pull request Mar 8, 2021

REQUEST: New membership for alexander-demichev kubernetes/org#2547

Closed

6 tasks

alexander-demicev deleted the labels branch April 6, 2021 10:49

	Copyright 2019 The Kubernetes Authors.
	Copyright 2020 The Kubernetes Authors.

		err = r.setInterruptibleNodeLabel(ctx, remoteClient, machine.Status.NodeRef.Name)
		if err != nil {

	err = r.setInterruptibleNodeLabel(ctx, remoteClient, machine.Status.NodeRef.Name)
	if err != nil {
	if err := r.setInterruptibleNodeLabel(ctx, remoteClient, machine.Status.NodeRef.Name); err != nil {

	logger.Info("Set interruptible label to Machine's Node", "nodename", machine.Status.NodeRef.Name)
	logger.V(3).Info("Set interruptible label to Machine's Node", "nodename", machine.Status.NodeRef.Name)


		err := remoteClient.Get(ctx, client.ObjectKey{Name: nodeName}, node)
		if err != nil {

	err := testEnv.Get(ctx, client.ObjectKey{Name: node.Name, Namespace: ns.Name}, updatedNode)
	err := testEnv.Get(ctx, client.ObjectKey{Name: node.Name}, updatedNode)

	// Calculate the scheme.
	utilruntime.Must(apiextensionsv1.AddToScheme(scheme.Scheme))
	utilruntime.Must(clusterv1.AddToScheme(scheme.Scheme))
	utilruntime.Must(bootstrapv1.AddToScheme(scheme.Scheme))
	utilruntime.Must(expv1.AddToScheme(scheme.Scheme))
	utilruntime.Must(crs.AddToScheme(scheme.Scheme))
	utilruntime.Must(addonv1.AddToScheme(scheme.Scheme))
	utilruntime.Must(kcpv1.AddToScheme(scheme.Scheme))
	utilruntime.Must(admissionv1.AddToScheme(scheme.Scheme))


		func (r *MachineReconciler) setInterruptibleNodeLabel(ctx context.Context, remoteClient client.Client, nodeName string) error {
		node := &apicorev1.Node{}

✨ Label interruptible nodes #3668

✨ Label interruptible nodes #3668

Conversation

alexander-demicev commented Sep 21, 2020

k8s-ci-robot commented Sep 21, 2020

JoelSpeed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enxebre commented Sep 21, 2020

ncdc commented Sep 21, 2020

alexander-demicev commented Sep 21, 2020

ncdc commented Sep 21, 2020

enxebre commented Sep 21, 2020 • edited Loading

ncdc commented Sep 21, 2020

enxebre commented Sep 21, 2020

ncdc commented Sep 21, 2020

CecileRobertMichon commented Sep 21, 2020

vincepri commented Sep 21, 2020

enxebre commented Sep 22, 2020

vincepri commented Sep 22, 2020

vincepri commented Sep 22, 2020

ncdc commented Sep 22, 2020

alexander-demicev commented Sep 25, 2020

alexander-demicev commented Sep 30, 2020

alexander-demicev commented Nov 3, 2020

k8s-ci-robot commented Nov 3, 2020

alexander-demicev commented Nov 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincepri commented Nov 16, 2020

Choose a reason for hiding this comment

vincepri commented Nov 20, 2020 • edited Loading

ncdc commented Nov 20, 2020

ncdc commented Nov 20, 2020

vincepri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexander-demicev commented Nov 23, 2020

vincepri left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 30, 2020

alexander-demicev commented Dec 1, 2020

enxebre commented Sep 21, 2020 •

edited

Loading

vincepri commented Nov 20, 2020 •

edited

Loading