feat(controllers): add support for instanceMatchSelector

[Ronan-WeScale]

Fix #1401

Tested on multiple dashboard, used cases are in PR and I tested with my personal use case too :

instanceSelector:
    matchExpressions:
      - key: dashboards
        operator: In
        values:
           - a
           - b

[CLAassistant]

All committers have signed the CLA.

[Ronan-WeScale]

Hi @NissesSenap !

Can you tell me your process for PR validation and testing ? I'm really interested to become active contributor on this and other Grafana projects.

[NissesSenap]

Hi @Ronan-WeScale , I can only answer for grafana-operator.
Sadly we are not the best on writing tests in this project, we have written some unit tests but no integration tests and we have some e2e tests.
I see that you added examples in the hack folder, which is great for testing things manually, but it would be great if you can update this PR to also include e2e tests.
I recommend that you look at #1496 where new tests have been added for another use case.

We are always looking for contributors to the project, there are a number of issues that are open with help wanted flag, but to honest we want help with most of things ;)
So feel free to write in any issues that you think would fit you, and we can assign it to you.

You are also welcome to join our meetings that have once a week, for more info see: https://github.com/grafana/grafana-operator?tab=readme-ov-file#get-in-touch

I have kind of much to do at work right now, but I hope me or some other maintainer will have time to review this PR soon.

[weisdd]

@Ronan-WeScale I'm happy to see this functionality implemented, though I think it's a bit early for this PR to be merged. There are two points of improvement that I'd like to highlight.

First of all, I believe the code can and should be simplified. - Ideally, I want to be able to understand the behaviour by reading the code once. In its current form, even after reading it multiple times, I'm not entirely sure I fully understand it:

• There are many if-else blocks:
  • e.g. when filtering by operator, I would probably go either with switch block or just with multiple if blocks - you have break logic anyway;
• default values should be explicit and static:
  • e.g. there's valueMatched := matchExpression.Operator == "NotIn" and a for-loop afterwards - it's hard to guess the exact effect of that comparison;
  • I think it would make sense to set selected := false by default and also explicitly change it to either false or true in if blocks;
• I'm not sure if it's worth having logic from labelMatchedExpression in a separate function, it'd be easier to say once PR is rewritten;
• Since we're using golang 1.21, you can rely on slices.Contains when searching for elements.

Second, I see that you added some CRs to prove that the code works as expected, but, in general case, everything that can be covered by unit tests should be done with unit tests, especially with such complex functions :) - It takes a fraction of a second to run those, and refactoring becomes super-easy (any manual step has potential to turn into a mistake). Plus, tests not only prove that something works well, but also explain the behaviour.

[weisdd]

@Ronan-WeScale Just to give you an example on how code can change (not saying that the piece of code is entirely correct, haven't tested it - it's just to give you a direction):

With if:

selected := false
for _, matchExpression := range labelSelector.MatchExpressions {
	if label, ok := instance.Labels[matchExpression.Key]; ok {
		if matchExpression.Operator == "DoesNotExist" {
			selected = false
		}

		if matchExpression.Operator == "Exists" {
			selected = true
		}

		if matchExpression.Operator == "In" {
			selected = slices.Contains(matchExpression.Values, label)
		}

		if matchExpression.Operator == "NotIn" {
			selected = !slices.Contains(matchExpression.Values, label)
		}

                // All matchExpressions must evaluate to true in order to satisfy the conditions
		if !selected {
			break
		}
	}
}

With switch:

selected := false
for _, matchExpression := range labelSelector.MatchExpressions {
	if label, ok := instance.Labels[matchExpression.Key]; ok {
		switch matchExpression.Operator {
		case "DoesNotExist":
			selected = false
		case "Exists":
			selected = true
		case "In":
			selected = slices.Contains(matchExpression.Values, label)
		case "NotIn":
			selected = !slices.Contains(matchExpression.Values, label)
		}

                // All matchExpressions must evaluate to true in order to satisfy the conditions
		if !selected {
			break
		}
	}
}

So, my suggestion would be to extract the match selector logic to a separate function, so you can write some tests without the need for mocking k8s API-server and then prepare something similar to what I showed above (it doesn't have to be exactly the same code, I'm simply looking for something with low complexity, so it'd be easy to further maintain).

I hope that helps :)

Upd: One more thing that is missing (at least, per my understanding) is that all expressions must evaluate to true, so we should not really run break upon any match, we should rather evaluate the results of all expressions (can either be done through an extra slice or through break when there are no match, updated the code samples with that).

[Ronan-WeScale]

Hi @weisdd !

Thanks for your answer, I worked on e2e test last week.
I will try to simplify the code, initially I want to reduce use of If/else and/or case but I understand the complexity of my code.

[Ronan-WeScale]

Hi @weisdd @NissesSenap,

I changed the code, and run e2e test, sorry but I don't know how to right unit test on "labelMatchedExpression" func. If you can help me on this ?

[Ronan-WeScale]

Also I keep default "selected" value to true, because that broke matchLabels func.

[Ronan-WeScale]

file-checks :
Hum , I had run make all before commit but seems to forget some newlines in files. fixed but not pushed.

End-To-End :
I have used ingress in e2e because my local kind and operator running can't resolve local Kubernetes domain. It's possible to use ingress in your pipeline ? Maybe it's just a timeout ? I need logs to know, I can't reproduce this.

[Ronan-WeScale]

Hi,
I fixed my local env for running without ingress the e2e test, but now I got new problem for templating assert, dashboard list is loaded randomly, so sometimes the list doesn't match same order as the asserts file.

[weisdd]

As I mentioned before, I think we should rather focus on unit-tests, not e2e tests.

The functionality you're working on is implemented through a separate function that doesn't depend on API-server.
Unit-tests would allow to test it within a fraction of a second without the need to spin up a cluster.
When it comes to e2e-tests, they depend on image builds & cluster provisioning & operator deployment & grafana deployment, thus they offer a feedback loop that is thousands of times slower.

To get an idea on how unit-tests can be written, you can simply explore the existing code base. We heavily rely on Testify (specifically, assert package), so it's documentation would be very useful for you.
Essentially, you'll need to prepare a set of the same Golang structs for v1beta1.Grafana(it doesn't have to be a full spec, just relevant fields) and *v1.LabelSelector that cover at least the same cases as in your e2e-tests, pass them to the function and assert result. It's certainly not more difficult than usual Go code :)

[weisdd]

@Ronan-WeScale Thanks, I'll try to take a look at this later this week.

[andrejshapal]

Hello,
Very interested in this to be delivered. Please, let know if any help needed.

[NissesSenap]

@andrejshapal you can help out to verify with your local setup and make sure that this works as intended.

[andrejshapal]

Local k8s manifest

``` apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: grafana-operator namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default sources: - repoURL: ghcr.io/grafana/helm-charts chart: grafana-operator targetRevision: v5.9.0 helm: values: | fullnameOverride: "grafana-operator"

            namespaceScope: true
  
            resources:
              limits:
                cpu: 100m
                memory: 200Mi
              requests:
                cpu: 50m
                memory: 100Mi
  
            serviceMonitor:
              enabled: false
  
      - repoURL: https://kiwigrid.github.io
        chart: any-resource
        targetRevision: "0.1.0"
        helm:
          values: |
            anyResources:
              grafana-one: |-
                apiVersion: grafana.integreatly.org/v1beta1
                kind: Grafana
                metadata:
                  name: grafana-one
                  labels:
                    dashboards: "grafana-one"
                spec:
                  config:
                    log:
                      mode: "console"
                    auth:
                      disable_login_form: "false"
                    security:
                      admin_user: root
                      admin_password: secret
  
              grafana-two: |-
                apiVersion: grafana.integreatly.org/v1beta1
                kind: Grafana
                metadata:
                  name: grafana-two
                  labels:
                    dashboards: "grafana-two"
                spec:
                  config:
                    log:
                      mode: "console"
                    auth:
                      disable_login_form: "false"
                    security:
                      admin_user: root
                      admin_password: secret
  
              grafana-three: |-
                apiVersion: grafana.integreatly.org/v1beta1
                kind: Grafana
                metadata:
                  name: grafana-three
                  labels:
                    dashboards: "grafana-three"
                spec:
                  config:
                    log:
                      mode: "console"
                    auth:
                      disable_login_form: "false"
                    security:
                      admin_user: root
                      admin_password: secret
  
              datasource: |-
                apiVersion: grafana.integreatly.org/v1beta1
                kind: GrafanaDatasource
                metadata:
                  name: thanos-gc
                spec:
                  instanceSelector:
                    matchExpressions:
                      - key: dashboards
                        operator: In
                        values:
                          - grafana-one
                          - grafana-two
                  datasource:
                    name: thanos-gc
                    orgId: 1
                    access: proxy
                    uid: "P5DCFC7561CCDE821"
                    type: prometheus
                    url: http://thanos-query-frontend:9090/
                    isDefault: true
  
    destination:
      name: in-cluster
      namespace: monitoring
  
    syncPolicy:
      automated:
        prune: true
      syncOptions:
        - CreateNamespace=true
  ```

Test case:

                instanceSelector:
                  matchExpressions:
                    - key: dashboards
                      operator: In
                      values:
                        - grafana-one
                        - grafana-two

Datasource thanos-gc should be present in grafana-one/two, and be missing in grafana-three.

Test one (grafana-operator v5.9.0):

2024-05-03T16:39:10Z INFO GrafanaDatasourceReconciler found matching Grafana instances for datasource {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"thanos-gc","namespace":"monitoring"}, "namespace": "monitoring", "name": "thanos-gc", "reconcileID": "fff0d631-f384-4149-a81b-695e54fdfc7c", "count": 3}

Test result: Failed 🟥

Test two (grafana-operator local build from Ronan-WeScale:fix-1401):

2024-05-03T16:45:29Z INFO GrafanaDatasourceReconciler found matching Grafana instances for datasource {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"thanos-gc","namespace":"monitoring"}, "namespace": "monitoring", "name": "thanos-gc", "reconcileID": "f772e67e-393d-460a-804c-3fd188c97a8c", "count": 2}

Test result: Passed 🟩

Important

It was noticed operator does not remove datasource after reconcilation from instance which does not match with matchExpressions. In my test redeployment helped. In real case the datasource should be deleted (it will be removed from all instances) and then created again.

[weisdd]

@Ronan-WeScale Finally, got some updates for you :)

As you previously requested some help around writing/simplifying the unit-tests, here you go:

Let's start with basic recommendations:

• if a function is stored in file x.go, then tests should be in x_test.go:
  • in your case, the function was in controller_shared.go whereas tests - in dashboard_controller_test.go;
• unit-tests should be small and relatively simple:
  • no need to create objects/fields that have nothing to do with the function:
    • no need to populate Status field unless the function acts on this field;
    • no need to add any metadata that is not used in a function;
  • it's generally not a good practice to to create a list of slices and then have complex logic for various comparisons:
    • there's a concept of table-driven tests that helps to structure tests in a way that it's easy to read, understand, and maintain. There, you can just have one for-loop where you go through a list of tests, prepare structs and assert results in a straightforward predictable way.

To make it easier to understand what exactly I'm referring to, I've rewritten some code and pushed to your branch (I hope you don't mind :)). I would suggest you to just go through commits to see how the code was changing along the way.

With those changes, I think the code should be more or less ready to be merged, but I would like you, fellow maintainers, and any community members to go through the code to see if something is off.

P.S. Normally, it's easier to have your code merged to our code base, you just happened to take relatively difficult task that affects base functionality, so it requires more effort / precision :)

[weisdd]

@andrejshapal could you run your tests again against the latest code?

It was noticed operator does not remove datasource after reconciliation from instance which does not match with matchExpressions. In my test redeployment helped. In real case the datasource should be deleted (it will be removed from all instances) and then created again.

That's interesting. I think it's actually a bug that is not related to this PR, but that is actually present in other builds as well. Thanks for reporting that! It's now tracked under #1519

[andrejshapal]

All good

Test three (grafana-operator fix-1401 commit 3827260):

2024-05-05T18:04:35Z INFO GrafanaDatasourceReconciler found matching Grafana instances for datasource {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"thanos-gc","namespace":"monitoring"}, "namespace": "monitoring", "name": "thanos-gc", "reconcileID": "11b30888-4bc5-4308-81be-a7572254ad5c", "count": 2}

Test result: Passed 🟩

[Ronan-WeScale]

Hi @weisdd,

thank for all this informations and refactoring, like I said previously, I'm not developper and try to do my best in golang ( working and dirty :p ) but I prefer work or propose a solution, instead pointing a problem and go away.
I know for the x_test.go file, but I work on the wrong file because of example function I adapt and the other test file are absent so the base element are not super good :)
You deleted e2e test but that's worked fine for testing purpose and configuration use case for chainsaw and that's can simplify testing instead ask to other like @andrejshapal (thank for testing too). Also we can move this files to another folder (to skip it in make e2e) instead lost this files.

I will try to understand you're simplification of the unit tests, and maybe propose some unit-tests that's not already righted :)

PS: sorry for my poor english

[weisdd]

Well, proposing a solution is often the most difficult part, so it doesn't matter that much if something gets merged as is or not.
When I write something myself, most often, the first step is just to prepare something that works, and only then to think of making the code look good. It takes multiple iterations until I'm happy with the end result.

And I'm not a professional developer either. In fact, my first contribution here was actually my very first code written Go :)

As for removal of the e2e-test, it's simply not needed here. - Everything is already covered with unit-tests including the test case from @andrejshapal (I asked him to double-check just to make sure I haven't overlooked anything).