Add support for optional cluster type labels to cluster deployment and cluster provision metrics #1925

suhanime · 2022-12-02T17:22:24Z

In addition to the cluster-type label, it would be helpful for users to be able to provide their own set of labels (like STS, Private Link, Managed VPC etc) to look for while logging metrics. Allow support for such optional labels for metrics of ClusterDeployment and ClusterProvision controller

xref: https://issues.redhat.com/browse/HIVE-2045

/assign @2uasimojo

suhanime · 2022-12-02T17:31:53Z

/retest

pkg/constants/constants.go

pkg/controller/clusterdeployment/clusterdeployment_controller.go

suhanime

Configs to hiveConfig can be made as following:

    "spec": {
        "logLevel": "debug",
        "metricsConfig": {
            "metricsWithClusterTypeLabels": {
                "managed_vpc": "api.openshift.com/managed-vpc",
                "private_link": "api.openshift.com/private-link",
                "sts": "api.openshift.com/sts"
            }
        },
        "targetNamespace": "hive"
    },

Screenshot of example metric to ensure it works well with and without optional labels being provided:

@2uasimojo ready for review

pkg/controller/metrics/metrics.go

codecov · 2022-12-14T19:11:17Z

Codecov Report

Merging #1925 (c34936b) into master (a04d71c) will increase coverage by 0.20%.
The diff coverage is 84.69%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1925      +/-   ##
==========================================
+ Coverage   40.50%   40.71%   +0.20%     
==========================================
  Files         366      367       +1     
  Lines       34090    34239     +149     
==========================================
+ Hits        13809    13939     +130     
- Misses      19077    19091      +14     
- Partials     1204     1209       +5

Impacted Files	Coverage Δ
pkg/constants/constants.go	`100.00% <ø> (ø)`
...shift/hive/apis/hive/v1/clusterdeployment_types.go	`33.33% <ø> (ø)`
...er/clusterprovision/clusterprovision_controller.go	`48.53% <20.00%> (-0.94%)`	⬇️
.../clusterdeployment/clusterdeployment_controller.go	`62.47% <44.44%> (-0.22%)`	⬇️
pkg/controller/metrics/metrics.go	`33.51% <62.50%> (ø)`
.../controller/metrics/metrics_with_dynamic_labels.go	`84.61% <84.61%> (ø)`
...kg/controller/clusterdeployment/clusterinstalls.go	`75.23% <100.00%> (-0.12%)`	⬇️
.../controller/clusterdeployment/clusterprovisions.go	`61.21% <100.00%> (-0.09%)`	⬇️
pkg/controller/clusterdeployment/metrics.go	`96.87% <100.00%> (+6.39%)`	⬆️
pkg/controller/clusterprovision/metrics.go	`100.00% <100.00%> (ø)`
... and 2 more

suhanime · 2022-12-14T19:45:38Z

/retest

2uasimojo · 2022-12-14T21:25:30Z

/test e2e

2uasimojo

I didn't delve into the low-level details yet -- can do that once we've agreed on the more major things discussed inline.

apis/hive/v1/metricsconfig/metricsConfig.go

pkg/controller/clusterdeployment/clusterdeployment_controller.go

pkg/controller/clusterprovision/metrics.go

pkg/controller/metrics/metrics.go

suhanime

@2uasimojo I think this addresses all of your comments. PTAL

pkg/controller/metrics/metrics.go

pkg/controller/clusterprovision/metrics.go

apis/hive/v1/metricsconfig/metricsConfig.go

pkg/controller/metrics/metrics.go

docs/hive-metrics.md

pkg/controller/clusterdeployment/metrics.go

2uasimojo

Partial review, will continue tomorrow.

apis/hive/v1/metricsconfig/metricsConfig.go

pkg/controller/metrics/metricsWithDynamicLabels.go

2uasimojo

Another chunk of comments. Still not done.

pkg/controller/metrics/metricsWithDynamicLabels.go

apis/hive/v1/metricsconfig/metricsConfig.go

docs/hive-metrics.md

2uasimojo

Okay, finally done! Sorry it took so long!

pkg/controller/clusterdeployment/clusterdeployment_controller.go

pkg/controller/metrics/metricsWithDynamicLabels.go

2uasimojo · 2023-01-09T21:59:16Z

pkg/controller/clusterdeployment/clusterdeployment_controller_test.go

+	// Register the metrics to avoid panics during testing
+	registerMetrics(mapClusterTypeLabelToValue)


So actually, only TestClusterDeploymentReconcile (the first suite) panics if metrics aren't registered.

and

You can register them via this call at the top of that function and it works fine.

so

If you intend to add any kind of UT around the metrics configuration/labeling, you would want to be able to tailor the config accordingly in that test -- which you could then do by passing in a custom map (or, if taking the suggestion above, a custom MetricsConfig).

If you're not planning on adding that UT now, perhaps just beef up the comment here so future-we don't have to rediscover the above.

That said, I think you could -- and should -- add some unit tests for this PR. This would entail generating mocks for the following interfaces:

Observer (for HistogramVec.Observe())

ObserverVec (for HistogramVec.CurryWith())

Counter (for CounterVec.Inc())

Aww, shiiit, really? CounterVec.CurryWith() isn't associated with an interface??

Welp, I guess you can't UT the CounterVecs 🤷 (I even checked whether a more recent version of the prom golang client lib might have added that interface, but nope. This seemed such a silly omission that I opened an issue for it.)

I mean, I guess we could write our own interface in the hive project. We do this e.g. for the AWS client.

The metrics do not need to be registered, but they have to be defined at least - because code never checks whether a metric is registered, it always attempts to observe it, and prometheus libraries handle the rest.
I've changed the wording a bit, and I do intend to implement the unit testing in a follow-up PR

pkg/controller/clusterdeployment/metrics.go

pkg/controller/clusterprovision/clusterprovision_controller.go

pkg/controller/clusterprovision/clusterprovision_controller_test.go

suhanime

@2uasimojo Ready for review!

apis/hive/v1/metricsconfig/metricsConfig.go

docs/hive-metrics.md

pkg/controller/clusterdeployment/clusterdeployment_controller.go

docs/hive-metrics.md

suhanime · 2023-01-25T19:03:03Z

pkg/controller/clusterdeployment/clusterdeployment_controller_test.go

+	// Register the metrics to avoid panics during testing
+	registerMetrics(mapClusterTypeLabelToValue)


The metrics do not need to be registered, but they have to be defined at least - because code never checks whether a metric is registered, it always attempts to observe it, and prometheus libraries handle the rest.
I've changed the wording a bit, and I do intend to implement the unit testing in a follow-up PR

2uasimojo

Let me know if you want to live chat about the threading concern.

apis/hive/v1/metricsconfig/metrics_config.go

docs/hive_metrics.md

pkg/controller/metrics/metrics_with_dynamic_labels.go

pkg/controller/clusterdeployment/metrics.go

suhanime

@2uasimojo Ready for review!

PS: I squashed some of the commits, but I did push my changes before the squash - for ease of review

pkg/controller/clusterdeployment/metrics.go

apis/hive/v1/metricsconfig/metricsConfig.go

pkg/controller/metrics/metricsWithDynamicLabels.go

pkg/controller/metrics/metrics_with_dynamic_labels.go

2uasimojo · 2023-02-03T20:43:47Z

/test e2e

...while I review this

2uasimojo

This is very, very close.

I was gonna say we can land it as is and address my comments in a followon... but the pointer-to-the-map thing would actually affect the API, so we have to get it right out of the gate.

pkg/controller/metrics/metrics_with_dynamic_labels.go

apis/hive/v1/metricsconfig/metrics_config.go

pkg/controller/metrics/metrics_with_dynamic_labels.go

suhanime

@2uasimojo Ready for review!

apis/hive/v1/metricsconfig/metrics_config.go

pkg/controller/metrics/metrics_with_dynamic_labels.go

suhanime · 2023-02-09T20:28:36Z

pkg/controller/metrics/metrics_with_dynamic_labels.go

+	}
+}
+
+// getLabelList combines fixed and optional labels and returns the list without duplicates.


While doing this, I thought about how it would be easier to have an early feedback - if a user tries to cause an overlap between these labels. However, there is not simpler way to do it, and we wouldn't want to error out because of metrics.

Right, so a couple things:

I think we should do this check. We could do it during Register(), or we could do it from within New*VecWithDynamicLabels() itself.

...and I think we should panic if there is an overlap (MustRegister() panics if something is wrong, so it fits the pattern).

Since it is going to be possible for a hive upgrade to trigger this panic (we add a fixed label that the user already had configured) we should make the message very clear: "HiveConfig.Spec.AdditionalClusterDeploymentLabels[%q] conflicts with a fixed label for metric %s. Please rename your label." kind of thing.

You may want to use the convenient Set class provided by apimachinery rather than rolling your own via map[string]bool.

suhanime · 2023-02-09T22:21:10Z

/retest

2uasimojo

This is a lovely iteration. I think we're really getting somewhere with the design of the objects/methods.

Most of my comments suggest refactoring, consolidation, simplification, cleanup. However, the couple of things still holding this from merging are:

We need to decide whether we ever allow the empty string as a label value. You have previously asserted that that somehow risks defining a metric with too few labels, and we agreed to always default.
I like the idea of panicking if the user tries to configure an optional label with the same key as a fixed one.

See inline for details.

Thanks!

pkg/controller/clusterdeployment/clusterdeployment_controller.go

2uasimojo · 2023-02-09T22:04:11Z

pkg/controller/metrics/metrics_with_dynamic_labels.go

+	}
+}
+
+// getLabelList combines fixed and optional labels and returns the list without duplicates.


Right, so a couple things:

I think we should do this check. We could do it during Register(), or we could do it from within New*VecWithDynamicLabels() itself.

...and I think we should panic if there is an overlap (MustRegister() panics if something is wrong, so it fits the pattern).

Since it is going to be possible for a hive upgrade to trigger this panic (we add a fixed label that the user already had configured) we should make the message very clear: "HiveConfig.Spec.AdditionalClusterDeploymentLabels[%q] conflicts with a fixed label for metric %s. Please rename your label." kind of thing.

You may want to use the convenient Set class provided by apimachinery rather than rolling your own via map[string]bool.

pkg/controller/metrics/metrics_with_dynamic_labels.go

2uasimojo · 2023-02-09T22:37:55Z

pkg/controller/metrics/metrics_with_dynamic_labels.go

+func NewCounterVecWithDynamicLabels(counterOpts *prometheus.CounterOpts,
+	labelOptions ...func(labels *dynamicLabels)) *CounterVecWithDynamicLabels {
+	counterVecMetric := &CounterVecWithDynamicLabels{
+		CounterOpts:   counterOpts,
+		dynamicLabels: &dynamicLabels{},
+	}
+	for _, o := range labelOptions {
+		o(counterVecMetric.dynamicLabels)
+	}


This paradigm seems overcomplicated to me. Why not just

Suggested change

func NewCounterVecWithDynamicLabels(counterOpts *prometheus.CounterOpts,

labelOptions ...func(labels *dynamicLabels)) *CounterVecWithDynamicLabels {

counterVecMetric := &CounterVecWithDynamicLabels{

CounterOpts: counterOpts,

dynamicLabels: &dynamicLabels{},

}

for _, o := range labelOptions {

o(counterVecMetric.dynamicLabels)

}

func NewCounterVecWithDynamicLabels(counterOpts *prometheus.CounterOpts,

fixedLabels []string, optionalLabels map[string]string) *CounterVecWithDynamicLabels {

counterVecMetric := &CounterVecWithDynamicLabels{

CounterOpts: counterOpts,

dynamicLabels: &dynamicLabels{

fixedLabels: fixedLabels,

optionalLabels: optionalLabels,

},

}

Then we don't need the With*() functions.

I'm confused - didn't you want the Opts pattern here so we can allow for future expansion of this interface if needed? Or did I just completely misunderstand why Opts pattern was needed at all?
Nevertheless, I'm reverting to the original way which you suggest here.

I checked the original ask, you only asked for more of a CounterOpts and HistogramOpts, and when I was learning about the Opts pattern and how to implement it, I decided to allow for future expansion by implementing a more flexible option. Anyways, I'm over-explaining here. Implemented your change.

Okay, I see. I hadn't been thinking about the options pattern. Now that I've looked at it in that light, I get the advantages. In particular, if we wanted to add fields to our fancy structure, we would have to add arguments to the constructor and then change all the calls to it. In this case, short term loss (more, and more complicated, code) but long term gain (easier to enhance in the future). If you want to put it back in a fup, I'm down. But for now this is good.

pkg/controller/metrics/metrics_with_dynamic_labels.go

This commit adds labels to indicate if cluster is of type STS, Private-Link or Managed-VPC for ClusterProvision controller metrics

This commit adds labels to indicate if cluster is of type STS, Private-Link or Managed-VPC for ClusterDeployment controller metrics. These labels are added only to those metrics which already report "cluster_type" label.

Allow users to add more cluster-type labels for metrics logging. These labels should be applied to all ClusterDeployment and ClusterProvision metrics that are logged per cluster-type. Optional labels can be provided via metricsConfig section in the hiveConfig. Users can name the label on the metric and assign it to a corresponding clusterDeployment (or provision) label. - While working on these changes, 2 metrics from ClusterProvision controller (metricClusterProvisionsTotal and metricInstallErrors) were moved to clusterProvision\metrics to collate all the relevant metrics and support was added for optional labels. - Registering of metrics was moved later instead of init() time because the latter is executed before we could read the hiveConfig. Consequently the metrics needed to be explicitly registered for unit test purposes to avoid nil panics

- Create a new interface for metrics with dynamic/optional labels and implement observing methods to simplify declaring and logging of the metric - Make cluster_type label a part of optional label logic, so it can eventually be made opt-in via hiveconfig - Rename MetricsWithClusterTypeLabels to AdditionalClusterDeploymentLabels - Always add optional labels (while currying) in sorted order - Remove DefaultClusterType constant from clusterdeployment_types because it was only used at one place that was removed as a part of this commit - Add documentation for metrics - Rename files and package names as per Golang naming conventions - Add Hive Operator metrics in documentation - Add omitempty to metricsConfig fields - Set cluster_type only if config is empty - Ensure metric writes/observes are protected from race conditions

This commit implements some of the changes recommended based on PR comments. Main change is applying Opts pattern to the interface for metrics with dynamic labels and apply a buildLabels helper to ensure there is no overlap between fixed and optional labels as we define and observe the corresponding metrics. A side effect to the buildLabels helper is that we no longer need to build a curried vector for a metric before observing it - we instead build the map for labels to value and observe them via `With` (or `GetMetricWith`) method, which is similar to how we observe metrics without optional label support.

suhanime

@2uasimojo Ready for review

suhanime · 2023-02-15T17:52:04Z

pkg/controller/metrics/metrics_with_dynamic_labels.go

+func NewCounterVecWithDynamicLabels(counterOpts *prometheus.CounterOpts,
+	labelOptions ...func(labels *dynamicLabels)) *CounterVecWithDynamicLabels {
+	counterVecMetric := &CounterVecWithDynamicLabels{
+		CounterOpts:   counterOpts,
+		dynamicLabels: &dynamicLabels{},
+	}
+	for _, o := range labelOptions {
+		o(counterVecMetric.dynamicLabels)
+	}


I'm confused - didn't you want the Opts pattern here so we can allow for future expansion of this interface if needed? Or did I just completely misunderstand why Opts pattern was needed at all?
Nevertheless, I'm reverting to the original way which you suggest here.

suhanime · 2023-02-15T17:56:23Z

pkg/controller/metrics/metrics_with_dynamic_labels.go

+func NewCounterVecWithDynamicLabels(counterOpts *prometheus.CounterOpts,
+	labelOptions ...func(labels *dynamicLabels)) *CounterVecWithDynamicLabels {
+	counterVecMetric := &CounterVecWithDynamicLabels{
+		CounterOpts:   counterOpts,
+		dynamicLabels: &dynamicLabels{},
+	}
+	for _, o := range labelOptions {
+		o(counterVecMetric.dynamicLabels)
+	}


I checked the original ask, you only asked for more of a CounterOpts and HistogramOpts, and when I was learning about the Opts pattern and how to implement it, I decided to allow for future expansion by implementing a more flexible option. Anyways, I'm over-explaining here. Implemented your change.

suhanime · 2023-02-16T10:04:39Z

pkg/controller/metrics/metrics.go

-	if typeStr, ok := obj.GetLabels()[hivev1.HiveClusterTypeLabel]; ok {
+// GetLabelValue returns the value of the label if set, otherwise a default value.
+func GetLabelValue(obj metav1.Object, label string) string {
+	if typeStr, ok := obj.GetLabels()[label]; ok {


Right, so I do not remember when I discovered a problem - I distinctly remember us debating what a default value should be (like N/A or unknown). Nevertheless, I decided to track down the actual problem in the library because just 1 person remembering something like this is not ideal for future-proofing.
Okay, so With or WithLabelValues internally calls GetMetricWith from vec.go, and the problem comes when getOrCreateMetricWithLabels creates the map with size equivalent to labels passed instead of reading from metric.Desc to determine the correct size (Note - even the extractLabelValues only relies on labels provided)

TL;DR
When the metric is created, it relies on the labels provided instead of metric definition as the dimension for creating the map. So if you provide empty string as label, it ignores it and creates the resultant metric map with less dimensions, consequently leading to always dropping some labels while observing.
Maybe With might not lead to such a problem - WithLabelValues definitely does. But if we know that happens, we don't want to treat them differently as a safety.

PS: These are the closest relates issues/discussions I found related to this topic:
variable labels
missing labels
flexible metric exposition

2uasimojo · 2023-02-16T21:30:24Z

pkg/controller/metrics/metrics.go

-	if typeStr, ok := obj.GetLabels()[hivev1.HiveClusterTypeLabel]; ok {
+// GetLabelValue returns the value of the label if set, otherwise a default value.
+func GetLabelValue(obj metav1.Object, label string) string {
+	if typeStr, ok := obj.GetLabels()[label]; ok {


Okay; so then my comment still applies. If the label exists but happens to have an empty value, this code as written will cause it to be observed with an empty value rather than our default.

Now, based on the code you pointed to, it seems to me like that would be okay, because we would actually be passing in the empty string as an argument, as opposed to omitting the argument entirely, so the number of dimensions would still be correct. And if that's true, I would argue it's the appropriate thing to do: i.e. there's a difference in behavior for the user between the CR lacking the label ("unspecified") and the label existing with an empty value (the value is specified, and it's "").

Whichever way we decide to do it, it should be consistent. Right now the fixed labels always convert "" to the default, but the dynamic ones don't.

2uasimojo · 2023-02-16T21:33:41Z

pkg/controller/metrics/metrics_with_dynamic_labels.go

+		},
+	}
+	if repeatedLabel := counterVecMetric.getRepeatedLabel(); repeatedLabel != "" {
+		log.Errorf("HiveConfig.Spec.AdditionalClusterDeploymentLabels[%q] conflicts with a fixed label for the metric %s. Please rename your label.", repeatedLabel, counterOpts.Name)


We agreed to make this fatal, right?. (And if you turn this into a panic, you don't need the log arg.)

2uasimojo · 2023-02-16T21:37:36Z

pkg/controller/metrics/metrics_with_dynamic_labels.go

+	return finalList
+}
+
+// getRepeatedLabel detects and returns the first optional label it encounters, that overlaps with another label


nit: This would be more useful if it returned all repeated labels so the consumer doesn't have to iterate. Can be done in a fup though.

2uasimojo · 2023-02-17T03:27:23Z

Beautiful!

/lgtm

openshift-ci · 2023-02-17T03:27:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 2uasimojo, suhanime

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [2uasimojo,suhanime]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-02-17T04:39:43Z

/retest-required

Remaining retests: 0 against base HEAD a04d71c and 2 for PR HEAD c34936b in total

2uasimojo · 2023-02-17T14:03:25Z

/retest

suhanime · 2023-02-17T16:01:36Z

/retest

openshift-ci · 2023-02-17T17:34:49Z

@suhanime: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci bot assigned 2uasimojo Dec 2, 2022

openshift-ci bot requested review from 2uasimojo and dlom December 2, 2022 17:22

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 2, 2022

2uasimojo reviewed Dec 5, 2022

View reviewed changes

pkg/constants/constants.go Outdated Show resolved Hide resolved

pkg/controller/clusterdeployment/clusterdeployment_controller.go Outdated Show resolved Hide resolved

suhanime force-pushed the HIVE-2045 branch from 2ef0bb0 to fa9de93 Compare December 14, 2022 18:31

suhanime commented Dec 14, 2022

View reviewed changes

pkg/controller/metrics/metrics.go Outdated Show resolved Hide resolved

pkg/controller/metrics/metrics.go Outdated Show resolved Hide resolved

suhanime changed the title ~~Add more cluster type labels to cluster deployment and cluster provision metrics~~ Add support for optional cluster type labels to cluster deployment and cluster provision metrics Dec 14, 2022

2uasimojo reviewed Dec 14, 2022

View reviewed changes

suhanime commented Dec 22, 2022

View reviewed changes

2uasimojo reviewed Jan 4, 2023

View reviewed changes

2uasimojo reviewed Jan 6, 2023

View reviewed changes

2uasimojo reviewed Jan 9, 2023

View reviewed changes

suhanime force-pushed the HIVE-2045 branch from 2475e02 to 756f500 Compare January 25, 2023 19:22

suhanime commented Jan 25, 2023

View reviewed changes

suhanime force-pushed the HIVE-2045 branch from 756f500 to d009d75 Compare January 25, 2023 19:32

2uasimojo reviewed Jan 25, 2023

View reviewed changes

suhanime force-pushed the HIVE-2045 branch 2 times, most recently from d210af8 to ab05f82 Compare February 2, 2023 01:24

suhanime commented Feb 2, 2023

View reviewed changes

suhanime mentioned this pull request Feb 3, 2023

feat: add cluster_id to metricProvisionFailedTerminal #1953

Closed

2uasimojo reviewed Feb 3, 2023

View reviewed changes

suhanime force-pushed the HIVE-2045 branch from ab05f82 to 25bb4e9 Compare February 9, 2023 20:26

suhanime commented Feb 9, 2023

View reviewed changes

2uasimojo reviewed Feb 9, 2023

View reviewed changes

suhanime added 7 commits February 14, 2023 15:21

Add ClusterType labels to ClusterProvision metrics

b380075

This commit adds labels to indicate if cluster is of type STS, Private-Link or Managed-VPC for ClusterProvision controller metrics

Add ClusterType labels to ClusterDeployment metrics

20fc542

This commit adds labels to indicate if cluster is of type STS, Private-Link or Managed-VPC for ClusterDeployment controller metrics. These labels are added only to those metrics which already report "cluster_type" label.

Refactor

aba011e

Some more changes per review comments

16d5c80

suhanime force-pushed the HIVE-2045 branch from 25bb4e9 to 16d5c80 Compare February 16, 2023 10:05

suhanime commented Feb 16, 2023

View reviewed changes

2uasimojo reviewed Feb 16, 2023

View reviewed changes

Last(?) iteration

c34936b

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 17, 2023

openshift-merge-robot merged commit e2fccdf into openshift:master Feb 17, 2023

suhanime deleted the HIVE-2045 branch February 17, 2023 20:30

hongkailiu mentioned this pull request Mar 3, 2023

hive metrics: add an owner label openshift/release#36902

Merged

		// Register the metrics to avoid panics during testing
		registerMetrics(mapClusterTypeLabelToValue)

Add support for optional cluster type labels to cluster deployment and cluster provision metrics #1925

Add support for optional cluster type labels to cluster deployment and cluster provision metrics #1925

Conversation

suhanime commented Dec 2, 2022 • edited Loading

suhanime commented Dec 2, 2022

suhanime left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Dec 14, 2022 • edited Loading

Codecov Report

suhanime commented Dec 14, 2022

2uasimojo commented Dec 14, 2022

2uasimojo left a comment

Choose a reason for hiding this comment

suhanime left a comment

Choose a reason for hiding this comment

2uasimojo left a comment

Choose a reason for hiding this comment

2uasimojo left a comment

Choose a reason for hiding this comment

2uasimojo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suhanime left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2uasimojo left a comment

Choose a reason for hiding this comment

suhanime left a comment

Choose a reason for hiding this comment

2uasimojo commented Feb 3, 2023

2uasimojo left a comment

Choose a reason for hiding this comment

suhanime left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suhanime commented Feb 9, 2023

2uasimojo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suhanime left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suhanime Feb 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2uasimojo commented Feb 17, 2023

openshift-ci bot commented Feb 17, 2023

openshift-ci-robot commented Feb 17, 2023

2uasimojo commented Feb 17, 2023

suhanime commented Feb 17, 2023

openshift-ci bot commented Feb 17, 2023

suhanime commented Dec 2, 2022 •

edited

Loading

suhanime left a comment •

edited

Loading

codecov bot commented Dec 14, 2022 •

edited

Loading

suhanime Feb 16, 2023 •

edited

Loading