Scalability and Performance Test Cases for Ephemeral Volumes #601

mucahitkurt · 2019-07-03T23:33:25Z

Related issue: #600

Different override files are used to test different ephemeral volume types.
For test cases and parameters you can check README file.

/sig storage
/assign @msau42

Signed-off-by: Mucahit Kurt <[email protected]>

k8s-ci-robot · 2019-07-03T23:33:26Z

Welcome @mucahitkurt!

It looks like this is your first PR to kubernetes/perf-tests 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/perf-tests has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2019-07-03T23:33:32Z

Hi @mucahitkurt. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msau42 · 2019-07-03T23:37:07Z

/ok-to-test
cc @hantaowang

msau42 · 2019-07-03T23:38:02Z

cc @wojtek-t

wojtek-t · 2019-07-05T10:27:04Z

@mucahitkurt - thanks a lot for this PR
Sorry, I'm a bit overloaded - I will take a look into it early next week. Sorry for delay.

/assign

mm4tt · 2019-07-05T13:34:41Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

+{{$GROUP := "pod-with-ephemeral-volume-startup-latency"}}
+
+#Variables
+# TODO think about this!


Ownerless/issue-less todo, please fix

It's not clear in comments but I added this and other TODOs because I need feedbacks about these parameters and their values.

See my comment above.
I actually believe that we need two separate tests:

first focus only on the first two cases from the KEP (which by design are running on a very small cluster)

separately address the third case, which I think has to be written a bit differently

Assuming we're focuing on (1) here, I would vote for simplifying this case and not think about namespaces, etc. at all here. I would be much easier to understand then,

I'll delete the parameters that I need for the (2) tests

mm4tt · 2019-07-05T13:37:08Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

+name: pod-with-ephemeral-volume-startup-latency
+automanagedNamespaces: {{$namespaces}}
+tuningSets:
+# TODO different options?


uniformqps make sense to me

mm4tt · 2019-07-05T13:44:09Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

+    Params:
+      desiredPodCount: {{$totalPods}}
+      labelSelector: group = {{$GROUP}}
+      # TODO how can i determine this?


Same here

Good approach is to start with something big enough, run the test a few times and see what is the distribution of podWithStorage startup-latency and then adjust the timeout making it big enough that it should never time out if everything goes well but small enough that it would catch issues in a reasonable time.

I think this one is also determined with the test rollout process.

mm4tt · 2019-07-05T13:54:38Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

@@ -0,0 +1,97 @@
+#Constants
+{{$NODES_PER_NAMESPACE := DefaultParam .NODES_PER_NAMESPACE 1}}


QQ, should the test scale with the size of the cluster and what was the cluster size you had in your mind when designing this test?

We're running scalability tests on clusters up to 5K nodes, if this test was run on such cluster as is, it would create 5K namespaces with 1 pod in each namespace? Is it want you wanted to achieve?

Actually there will be three different cluster usage scenarios, 2 of them will use 1 node and one of them will be cluster wide.

There will be a test rollout phase before these tests are run regularly, to determine the thresholds, but I don't know how that rollout process will be executed, before PR merge or after?, maybe @msau42 knows. I thought as these parameters can be changed manually during rollout phase, so I didn't create different override files for 1 node case and cluster wide case. Because the same thing is also valid for parameters like how many volumes will be created per node, or how many volumes will be created per node. Of course I can create different override files with the max limits for each volume type.

There will be a test rollout phase before these tests are run regularly, to determine the thresholds, but I don't know how that rollout process will be executed, before PR merge or after?

We should mark this test as more experimental (maybe under experimental/ directory) and we can do that after the PR is merged. But I'm fine either way too.

@mm4tt - the first two cases, are more about performance than scalability, and will be run against very small (1-node?) cluster. That is probably important part of the answer to the question.

@mucahitkurt - I think that we should focus on the first 2 case first (as the usecase there is completely different). And leave the third case (where the tests may potentially be visibly different) as a second thing. WDYT?

Covering only first 2 cases makes sense to me, I can create a different PR for the 3rd case after this PR is merged. I'll put these test cases under an experimental directory and 1_node directory.

wojtek-t · 2019-07-08T12:06:27Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

@@ -0,0 +1,97 @@
+#Constants
+{{$NODES_PER_NAMESPACE := DefaultParam .NODES_PER_NAMESPACE 1}}


There will be a test rollout phase before these tests are run regularly, to determine the thresholds, but I don't know how that rollout process will be executed, before PR merge or after?

We should mark this test as more experimental (maybe under experimental/ directory) and we can do that after the PR is merged. But I'm fine either way too.

@mm4tt - the first two cases, are more about performance than scalability, and will be run against very small (1-node?) cluster. That is probably important part of the answer to the question.

@mucahitkurt - I think that we should focus on the first 2 case first (as the usecase there is completely different). And leave the third case (where the tests may potentially be visibly different) as a second thing. WDYT?

wojtek-t · 2019-07-08T12:08:48Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

+{{$GROUP := "pod-with-ephemeral-volume-startup-latency"}}
+
+#Variables
+# TODO think about this!


See my comment above.
I actually believe that we need two separate tests:

first focus only on the first two cases from the KEP (which by design are running on a very small cluster)

separately address the third case, which I think has to be written a bit differently

Assuming we're focuing on (1) here, I would vote for simplifying this case and not think about namespaces, etc. at all here. I would be much easier to understand then,

wojtek-t · 2019-07-08T12:09:42Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

+name: pod-with-ephemeral-volume-startup-latency
+automanagedNamespaces: {{$namespaces}}
+tuningSets:
+# TODO different options?


uniformqps make sense to me

wojtek-t · 2019-07-08T12:11:09Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

+    Method: APIResponsiveness
+    Params:
+      action: reset
+{{ if $PROVISION_VOLUME }}


Why do you want to gate provisioning volumes?

I added the gate because for some volumes, like configmap and secret, I should pre-provision the volumes to use it inside the pod, but for some volumes, like downwardapi and emptyDir, I don't need volume pre-provisioning, volume settings are directly written inside pod template.

wojtek-t · 2019-07-08T12:11:39Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/config.yaml

+    - basename: vol-{{$APP_NAME}}
+      objectTemplatePath: {{$VOLUME_TYPE}}.yaml
+{{ end }}
+# Start measurements


I vote for moving this up and doing measurement operations together.

I'm not sure about this, because KEP says that assuming that the volumes have already been provisioned.

If I move this up, then measurement is started before volume provisioning.

I think we want to separate out provisioning time measurements from pod startup measurements because provisioning is generally a one-time bootstrapping operation (for an application).

If I move this up, then measurement is started before volume provisioning.

yes - but the measurement only measures pod startup time. the fact that you create some other objects doesn't affect that at all.

We want to separate out provisioning time - but that means creating those volumes before we create a pod - which is happening in this test no matter when you start the measurement.

When does pod startup measurement begin? For ephemeral volumes, kubelet cannot mount API volumes until the API object is created, and containers cannot start until all volumes are mounted. For PVCs, scheduler cannot schedule the pod until the pvs are provisioned and bound to the PVC.

The "measurement start" action in the config is basically "start watching for pod creations and for those measure pod startup time".
Measuring pod startup time for individual pod starts when that particular pod is created, which in case of this config are created after all volumes already exist.

I moved it up, thanks!

wojtek-t · 2019-07-08T12:12:54Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/Secret.yaml

+type: Opaque
+data:
+  username: dXNlcm5hbWVfCg==
+  password: cGFzc3dvcmRfCg==


nit: can you add end-of-lines in all files?

wojtek-t · 2019-07-08T12:14:52Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/pod.yaml

+  - name: {{.Name}}
+    image: k8s.gcr.io/pause:3.1
+    imagePullPolicy: IfNotPresent
+    ports:


nit: do we need ports?

no, I'll delete

wojtek-t · 2019-07-08T12:21:14Z

clusterloader2/testing/storage/pod-startup/ephemeral-volumes/pod.yaml

+      mountPath: /usr/share/{{$volumeIndex}}
+    {{ end }}
+  volumes:
+  {{ range $volumeIndex, $vol := Seq .VolumesPerPod }}


I'm wondering if this really makes it easier to understand.

I agree that it would result in some duplication of code (or I should say config), but wouldn't having:
pod_with_secrets, pod_with_configmaps, pod_with_emptydirs as separate yamls, be simpler?

Longer-term, I filed: #619, but I don't know how easy that is to do.

I'll do it, also will move the volume specific yaml files under the specific volume directory

mucahitkurt · 2019-07-08T21:22:07Z

@wojtek-t thanks a lot for your comments! I tried to reply & fix them, PTAL.

mucahitkurt · 2019-07-08T21:42:37Z

/test pull-perf-tests-clusterloader2

…or different volume types Signed-off-by: Mucahit Kurt <[email protected]>

wojtek-t · 2019-07-09T13:04:46Z

clusterloader2/testing/experimental/storage/pod-startup/ephemeral-volumes/1_node/config.yaml

+    Params:
+      action: start
+      labelSelector: group = {{$GROUP}}
+      threshold: {{$podStartupTimeout}}s


Medium-term (or maybe even not that medium), we should merge the logic of detecting whether the pod is stateful or stateless into the measurement and use thresholds automatically.

@mm4tt @oxddr - do you know if we have an issue opened for that?

Signed-off-by: Mucahit Kurt <[email protected]>

wojtek-t

I mostlly have couple more cosmetic comments, but I'm generally happy with how it looks like now.

wojtek-t · 2019-07-10T14:16:02Z

clusterloader2/testing/experimental/storage/pod-startup/ephemeral-volumes/1_node/config.yaml

+{{ if $PROVISION_VOLUME }}
+# Provision volumes
+- phases:
+  - namespaceRange:


@krzysied - I was thinking about it in the past, but I don't know if we ever did that (and if I ever shared with you that I was thinking about it).
In majority of cases, we're setting "namespaceRange" to basically all automanaged-namespaces.
What I would like to do is to remove all of those and say:
"if you don't specify namespaceRange, we default to all automanaged namespaces"

Is that done? If not, would you be able to add that (should be quite simple to do).

perf-tests/clusterloader2/pkg/test/simple_test_executor.go

Lines 312 to 315 in 459486e

if namespaceRange == nil {

// Returns "" which represents cluster level.

return []string{""}

}

If namespaceRange is unset, the cluster level object will be created. IIRC currently we are not using this feature. The question is if we want to create non-namespace objects in the future...

Ahh ok - we definitely would like to be able to create non-namespaced objects (e.g. CRD is non-namespaced).

We should probably come up with some "allNamespaces" thing to reduce amount of boilerplate...

wojtek-t · 2019-07-10T14:17:56Z

clusterloader2/testing/experimental/storage/pod-startup/ephemeral-volumes/1_node/config.yaml

+      labelSelector: group = {{$GROUP}}
+      # TODO decide this after test roll-out phase
+      timeout: 15m
+- measurements:


Similarly as for starting measurement, can you also move it to the bottom?

[As I mentioned somewhere else, medium term, we will simply bake all the necessary thresholds into the measurement itself and we will bundle all SLO-related ones together to limit boilerplate. It will just be simpler to do then :)]

ok., I think deleting pods before the gather action of PodStartupLatency measurement has no effect?

PodStartupLatency measurement can only change if when pods are being created - so yes, that will have no effect.

wojtek-t · 2019-07-10T14:20:45Z

...r2/testing/experimental/storage/pod-startup/ephemeral-volumes/1_node/configmap/override.yaml

@@ -0,0 +1,3 @@
+PROVISION_VOLUME: true
+POD_TEMPLATE_PATH: "configmap/pod_with_ConfigMaps.yaml"
+VOLUME_TEMPLATE_PATH: "configmap/ConfigMap.yaml"


Do you somewhere use the assumption that the file names are capitalized ConfigMap (I don't see that, but maybe I'm missing something).
If not, it would be more natural to have:
configmap/config_map.yaml
configmap/pod_with_config_map.yaml

[Same for secrets and others]

There was but no longer, I'll rename them, but volume names are not used as separate words generally, usages are like ConfigMap, EmptyDir etc. I'll rename them as like configmap/configmap.yaml, configmap/pod_with_configmap.yaml etc.

Sure that sounds fine, thanks!

Signed-off-by: Mucahit Kurt <[email protected]>

mucahitkurt · 2019-07-12T19:19:11Z

Hi @wojtek-t, @msau42 ,

Is there anything that I should do with this PR, including test roll out phase, I don't know how that will be executed and there will be anything that I can contribute?

I'll open another PR for the ephemeral volume test cases that will be run on cluster wide. Thanks!

msau42 · 2019-07-12T23:18:23Z

/lgtm
Thanks for taking this on! @wojtek-t once this merges, how do you want to proceed on establishing a baseline?

wojtek-t · 2019-07-15T09:21:19Z

@msau42

Thanks for taking this on! @wojtek-t once this merges, how do you want to proceed on establishing a baseline?

I think we should create a test-suite similar to this one:
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-scalability/sig-scalability-periodic-jobs.yaml#L2
[It should probably live in the same place, we may want to add it under experimental tab for sig-scalability.]
After couple days, we should go over tests and see how the results look like (may be useful to expose them in perf-dash, @krzysied may help with that).

Can someone some sig-storage pick it?

/approve
/hold

Holding to get agreement on next steps - feel free to cancel if you ~agree with the above.

mucahitkurt · 2019-07-15T19:09:02Z

@msau42

Thanks for taking this on! @wojtek-t once this merges, how do you want to proceed on establishing a baseline?

I think we should create a test-suite similar to this one:
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-scalability/sig-scalability-periodic-jobs.yaml#L2
[It should probably live in the same place, we may want to add it under experimental tab for sig-scalability.]
After couple days, we should go over tests and see how the results look like (may be useful to expose them in perf-dash, @krzysied may help with that).

Can someone some sig-storage pick it?

/approve
/hold

Holding to get agreement on next steps - feel free to cancel if you ~agree with the above.

@wojtek-t , @msau42

I have some questions about periodic job configuration, we have 2 test cases, one of them is max volumes for a pod case, other one is max pod-volume(1-1) for a node case. Currently I have one config file, and it seems for first test case, and volumes for pod value is 4(I added randomly).

First question is, should I create two different override files for the 2 different test cases? And periodic job definitions will have two different override files, one for test case specific(max volume for a pod or max volume for a node), other one for volume type? So there are 4 ephemeral volume types, 2 test cases, and there will be 8 periodic job definitions?

Second question is, what should my initial values be for the max volumes for a pod case(current value is 4), and max pod-volume(1-1) for a node case? Max limits is 100 pods per node, can I use this for both scenarios?

msau42 · 2019-07-15T20:00:46Z

This plan sounds good to me. For max pod-volume, 100 pods with 1 volume each sounds good to me. For max volumes in 1 pod, 100 volumes also sounds fine. Although, for the 2nd case, how is it measured? Does the test only measure 1 pod? Or does it measure multiple pods serially?

wojtek-t · 2019-07-16T11:26:29Z

First question is, should I create two different override files for the 2 different test cases? And periodic job definitions will have two different override files, one for test case specific(max volume for a pod or max volume for a node), other one for volume type? So there are 4 ephemeral volume types, 2 test cases, and there will be 8 periodic job definitions?

I think that sounds most reasonable for now.
[Medium term, (assuming we already don't - @krzysied - what is the current state?) we should allow running the same test with different overrides as part of the same test job.]

This plan sounds good to me. For max pod-volume, 100 pods with 1 volume each sounds good to me. For max volumes in 1 pod, 100 volumes also sounds fine. Although, for the 2nd case, how is it measured? Does the test only measure 1 pod? Or does it measure multiple pods serially?

+1
I think that for now we only create 1 pod and measure startup for it. This has a drawback that it's not statistically valuable. We should try changing to "create+delete" N times and compute percentiles from that. But I'm fine with moving those changes to subsequent PR.

krzysied · 2019-07-16T11:41:42Z

First question is, should I create two different override files for the 2 different test cases? And periodic job definitions will have two different override files, one for test case specific(max volume for a pod or max volume for a node), other one for volume type? So there are 4 ephemeral volume types, 2 test cases, and there will be 8 periodic job definitions?

I think that sounds most reasonable for now.
[Medium term, (assuming we already don't - @krzysied - what is the current state?) we should allow running the same test with different overrides as part of the same test job.]

Yes, currently it requires 8 periodic job definitions. Overrides are passed to the clusterloader as flags so it applies to the job scope, not the test scope.

mucahitkurt · 2019-07-16T19:33:22Z

@wojtek-t @msau42 I added a new override file for the max volumes per node case, and also updated the default volume number to 100.

I'll open a PR to test-infra for the 8 periodic job definitions.

…ault value for volumes per pod Signed-off-by: Mucahit Kurt <[email protected]>

msau42 · 2019-07-16T23:35:15Z

/lgtm
/hold cancel

k8s-ci-robot · 2019-07-16T23:35:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msau42, mucahitkurt, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~clusterloader2/OWNERS~~ [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wojtek-t · 2019-07-17T11:07:25Z

@mucahitkurt - next time please squash the commits before merging
[other than that lgtm]

mucahitkurt · 2019-07-17T19:09:12Z

@mucahitkurt - next time please squash the commits before merging
[other than that lgtm]

Ok., sorry about that, I forgot.

pod with ephemeral volumes test cases

e914fd1

Signed-off-by: Mucahit Kurt <[email protected]>

k8s-ci-robot assigned msau42 Jul 3, 2019

k8s-ci-robot added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Jul 3, 2019

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 3, 2019

k8s-ci-robot requested review from mborsz and mm4tt July 3, 2019 23:33

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 3, 2019

k8s-ci-robot assigned wojtek-t Jul 5, 2019

mm4tt reviewed Jul 5, 2019

View reviewed changes

wojtek-t mentioned this pull request Jul 8, 2019

Make it possible to source yamls in yamls #619

Closed

wojtek-t reviewed Jul 8, 2019

View reviewed changes

mucahitkurt force-pushed the storage/scalability-performance-tests-pod-startup branch from 4d2c1e9 to 6c39713 Compare July 8, 2019 21:18

simplify test cases only 1-node test cases and separate pod configs f…

185997a

…or different volume types Signed-off-by: Mucahit Kurt <[email protected]>

mucahitkurt force-pushed the storage/scalability-performance-tests-pod-startup branch from 6c39713 to 185997a Compare July 9, 2019 04:18

wojtek-t reviewed Jul 9, 2019

View reviewed changes

move podstartupmeasurements to up

bf31435

Signed-off-by: Mucahit Kurt <[email protected]>

wojtek-t reviewed Jul 10, 2019

View reviewed changes

move podstartupmeasurements gather to bottom, rename volume config files

30effd8

Signed-off-by: Mucahit Kurt <[email protected]>

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 12, 2019

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 15, 2019

wojtek-t mentioned this pull request Jul 16, 2019

Allow for running the same ClusterLoader2 test with different overrides as part of the same job #641

Closed

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 16, 2019

mucahitkurt mentioned this pull request Jul 16, 2019

Add Periodic Job Definitions for the Sig Storage Performance Tests kubernetes/test-infra#13470

Closed

override file is added for max volumes per node test case, update def…

377ab09

…ault value for volumes per pod Signed-off-by: Mucahit Kurt <[email protected]>

mucahitkurt force-pushed the storage/scalability-performance-tests-pod-startup branch from fa76e6e to 377ab09 Compare July 16, 2019 20:22

mucahitkurt mentioned this pull request Jul 16, 2019

add periodic job definitions for ephemeral volumes perf/scalability test cases kubernetes/test-infra#13471

Merged

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jul 16, 2019

k8s-ci-robot merged commit 934323e into kubernetes:master Jul 16, 2019

mucahitkurt deleted the storage/scalability-performance-tests-pod-startup branch July 17, 2019 19:15

This was referenced Jul 17, 2019

Update override files for ephemeral volumes perf test cases #653

Merged

Cluster Wide Scalability and Performance Test Cases for Ephemeral Volumes #662

Merged

mucahitkurt mentioned this pull request Sep 7, 2019

REQUEST: New membership for mucahitkurt kubernetes/org#1166

Closed

6 tasks

		@@ -0,0 +1,97 @@
		#Constants
		{{$NODES_PER_NAMESPACE := DefaultParam .NODES_PER_NAMESPACE 1}}

	if namespaceRange == nil {
	// Returns "" which represents cluster level.
	return []string{""}
	}

Scalability and Performance Test Cases for Ephemeral Volumes #601

Scalability and Performance Test Cases for Ephemeral Volumes #601

Conversation

mucahitkurt commented Jul 3, 2019

k8s-ci-robot commented Jul 3, 2019

k8s-ci-robot commented Jul 3, 2019

msau42 commented Jul 3, 2019

msau42 commented Jul 3, 2019

wojtek-t commented Jul 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mucahitkurt Jul 5, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mucahitkurt commented Jul 8, 2019

mucahitkurt commented Jul 8, 2019

Choose a reason for hiding this comment

wojtek-t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mucahitkurt Jul 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mucahitkurt commented Jul 12, 2019 • edited Loading

msau42 commented Jul 12, 2019

wojtek-t commented Jul 15, 2019

mucahitkurt commented Jul 15, 2019 • edited Loading

msau42 commented Jul 15, 2019

wojtek-t commented Jul 16, 2019

krzysied commented Jul 16, 2019

mucahitkurt commented Jul 16, 2019

msau42 commented Jul 16, 2019

k8s-ci-robot commented Jul 16, 2019

wojtek-t commented Jul 17, 2019

mucahitkurt commented Jul 17, 2019

mucahitkurt Jul 5, 2019 •

edited

Loading

mucahitkurt Jul 10, 2019 •

edited

Loading

mucahitkurt commented Jul 12, 2019 •

edited

Loading

mucahitkurt commented Jul 15, 2019 •

edited

Loading