-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
backpack and OCS #6
Comments
@bengland2 I have created PR cloud-bulldozer/benchmark-operator#335 which would allow you to run the metadata daemonset on only specifically labeled nodes. You would have to run it in addition to the normal fio/etc cr. Something as simple as this would work for nodes labeled foo=bar
|
looks good to me, just have to try it out. 1 label is probably enough. |
If needed i could probably make it loop on a list but I haven't tried that with it yet |
@bengland2 I updated the PR to take a list of labels. Let me know your thoughts. The cr would now look like:
The node(s) would have to match all the labels given as its an implied AND functionality. |
my original post wasn't clear, but I was proposing an OR not an AND. For a workload like CNV+OCS, there might be multiple sets of nodes to deal with (i.e. CNV compute nodes + OCS storage nodes). |
haven't tested the PR with 2 labels yet but it certainly works with 1 label. |
@dry923 with 2 labels it failed, backpack never runs, here's the benchmark operator log, pretty-printed using, around line 340 there is an error.
and the parse-kube-pod-log.py is here, I just can't read a raw log from benchmark operator, sorry. The CR that I used is here. Again, with just 1 label it works, and is usable for me, just not quite as general purpose. This is low-priority, but someday would be nice to have. I'm using quay.io/benchmark-operator/benchmark-operator:master as of 20m ago. |
@bengland2 Thanks for the heads up. I'll take a look at why its failing and let you know. |
@bengland2 So my initial thought was that it might be failing on the empty value with the label
However, I did a similar test and it worked fine. I also re-checked it with the fio cr we use for ci but adding in the labels and it also succeeded. The logs we're too helpful unfortunately, is there any other information you can give me? I.e. did backpack launch then error, did fio error, did the operator throw an error and it simply never started anything, etc? Thanks! |
Is this still an issue @bengland2 ? |
the may 9th post is still an issue in my mind, I've just been preoccupied with other issues. The workaround is to just use one label for backpack that explicitly identifies which nodes you want to collect on, and that should be good enough for now. |
For some OCS tests, I want to run the workload pods on separate physical machines from the OCS (OpenShift Container Storage) cluster - I think this is a problem that other folks might have too, with more than OCS. At present, backpack will be blissfully unaware that there are other nodes involved in the test and will not collect metadata about them. It would be great if there was a way to hand backpack a list of labels that mean "collect metadata about any of the nodes that have one of these labels". For example, OCS nodes are all labelled. Doing it this way would make it applicable to much more than just OCS.
The text was updated successfully, but these errors were encountered: