Consider Kubernetes namespace when matching EventType consumer Backstage entities

[aliok]

Triggers don't support cross namespace references in Knative+Kubernetes side.

However, Backstage side is a different story. It is possible there could be 2 Services in Kubernetes is in namespaces payment and fraud but they are registered in the Backstage namespace operations, which is a logical namespace.

To support this scenario, Backstage allows defining the Kubernetes namespace for a component:

  'backstage.io/kubernetes-namespace': dice-space

https://backstage.io/docs/features/kubernetes/configuration#adding-the-namespace-annotation

By default, Backstage will not check any namespaces on Kubernetes side. In the example above, it will fetch all the resources for the label backstage.io/kubernetes-id=payment and associate that with the payment Backstage component. See https://backstage.io/docs/features/kubernetes/configuration#surfacing-your-kubernetes-components-as-part-of-an-entity for more details.

So, on Kubernetes side, it doesn't matter in which namespace the payment Service is.

Back to our case...
When we're checking the consumers of an EventType through a Trigger, we only check the subscribers of the EventType within the same namespace of the EventType. The consumer id returned on the other hand, can be in another Backstage namespace.
To make things worse, it can be in another Backstage namespace, but it might not have the 'backstage.io/kubernetes-namespace' annotation. And, in the backend side, we won't have access to what annotations the Backstage resource has.

We can use the 'backstage.io/kubernetes-namespace' annotation for an additional mechanism for finding out the Backstage consumer component.

Backstage will return the consumer namespace and consumer Backstage ID. Currently, we only search for components with that ID in the current Backstage namespace (Backstage namespace that's identical to the Kubernetes namespace):

backstage-plugins/backstage/plugins/knative-event-mesh-backend/src/providers/knativeEventMeshProcessor.ts

Lines 104 to 107 in 408beed

	filter: {
	kind: 'component',
	'metadata.namespace': namespace,
	'metadata.annotations.backstage.io/kubernetes-id': componentId,

If the component for that Backstage ID is not found in the current Backstage namespace, we can make a search across all Backstage namespaces. For all components found, we can check their backstage.io/kubernetes-namespace annotation to see if it matches the Kube namespace. If it does, it is our consumer.

I know this is all very confusing. But here are the reasons:

• Backstage has a namespace concept, but a Backstage namespace doesn't have to match a Kube namespace
• Backstage resource and Kube resource matching is done with labels+annotations in an indirect way

[aliok]

flowchart TD
    GetEventType[Get EventType]
    GetConsumer[Get EventType consumer Backstage id]
    QueryConsumerInCurrentNamespace(Query consumer in current <br> Backstage namespace)
    CheckComponentInCurrentNamespaceExists{Component in current <br> Backstage namespace exists}
    RegisterConsumer1[Register component as the consumer]
    QueryConsumerInAllNamespaces[Query consumer in all <br> Backstage namespaces]
    CheckNamespaceAnnotation{Kubernetes namespace annotation <br> matches the Kubernetes namespace?}
    RegisterConsumer2[Register component as the consumer]

    GetEventType --> GetConsumer
    GetConsumer --> QueryConsumerInCurrentNamespace
    QueryConsumerInCurrentNamespace --> CheckComponentInCurrentNamespaceExists
    CheckComponentInCurrentNamespaceExists --> |Yes| RegisterConsumer1
    CheckComponentInCurrentNamespaceExists --> |No| QueryConsumerInAllNamespaces
    QueryConsumerInAllNamespaces --> CheckNamespaceAnnotation
    CheckNamespaceAnnotation --> |Yes| RegisterConsumer2
    CheckNamespaceAnnotation --> |No| Ignore

    GetConsumer -.- Note1["At this stage, we have the Kubernetes namespace and the <br> Backstage component ID of the consumer. <br> We don't have the Backstage namespace though."]
    Note1:::note
    classDef note fill:yellow

Loading

[aliok]

More concrete explanation for anybody interested.

Prerequisite

Understand if cross namespace relations are possible in Backstage. Backstage documentation doesn't say anything about any limitation, so I assume it is supported. However, we should better test this first.

1. Create a YAML file with component definitions of 2 things: an API and a Component.
2. Create the API in namespace A
3. Create the Component in namespace B, make it consumesApi the API.
4. Install the YAML file (register on Backstage UI)
5. Check if things work!

A YAML file like this will look like this: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/artist-lookup-component.yaml

Have a look at the YAML file format in Backstage docs: https://backstage.io/docs/features/software-catalog/descriptor-format

And you will register new components in that YAML file like this: https://backstage.io/docs/features/software-catalog/#manually-register-components

Implementaion

1. Querying consumers in current namespace is done here:

   backstage-plugins/backstage/plugins/knative-event-mesh-backend/src/providers/knativeEventMeshProcessor.ts

   Line 48 in 408beed

   const consumerComponents = await this.findComponentsByBackstageId(entity.metadata.namespace as string, consumedBy);

2. If nothing matched, make a new query, but this time across all namespaces. Like this:

let response = await this.catalogApi.queryEntities({
    filter: {
        kind: 'component',
        'metadata.annotations.backstage.io/kubernetes-id': componentId,
    },
});

Similar to

backstage-plugins/backstage/plugins/knative-event-mesh-backend/src/providers/knativeEventMeshProcessor.ts

Lines 103 to 109 in 408beed

	let response = await this.catalogApi.queryEntities({
	filter: {
	kind: 'component',
	'metadata.namespace': namespace,
	'metadata.annotations.backstage.io/kubernetes-id': componentId,
	},
	});

3. Check if the Kubernetes namespace of the potential consumers match the information we receive part of the consumer information. Kubernetes namespace is basically the EventType namespace, as we always register an EventType to a Backstage namespace that's identical to a Kubernetes namespace. Consumer's Kubernetes namespace is stored in the backstage.io/kubernetes-namespace annotation.

4. Adapt tests. Write docs. Add diagram to the readme (you can reuse my diagram above).

5. Update the demo (I will publish soon)

[parth721]

Hi @aliok, I would like to work on this issue :)

[parth721]

/assign

[aliok]

/assign parth721

hey @parth721

sure! I tried to add extensive instructions and explanations above. I would be happy to clarify any questions you might have.

[parth721]

hey @parth721
sure! I tried to add extensive instructions and explanations above. I would be happy to clarify any questions you might have.

Hi @aliok, I'm going through the issue, and currently my university exam also going to start form coming monday. That's why I'm taking some time here.
I planned to fix it before my exam, but due to some reason it gets delayed by few days. I hope you can understand.

[aliok]

@parth721 no worries. Take your time.

[parth721]

Hi @aliok, I am gone through the issue and understand the suggested appoach to fix it.
Can you help , As much I understand about it :

1. I create yaml files for the component and api, then add backstage npx @backstage....
2. Later start cluster and register the repo to backstage from github.

Q. don't we need images, I mean I know that yaml files are to defining the ideal configuration state to k8s ?

please tell me, if something I misinterpret.

Having so much question, because everytime I tried it find new error, once PAT of Github not linking. Tried out you demo but my PC started lagging & found the component are not mapped. Now npx giving error while create backstage app. Till now I can't draw a conclusion

[parth721]

Hi @aliok, I followed your suggestion to separate components into one namespace and APIs into another, and registered them accordingly. I have a question: how can I verify if everything is working correctly? Could you provide some guidance on this matter, or suggest where I should look for?
Thank you

[parth721]

Hi @aliok I had done the task as suggested. Please share something need to be updated.repository link

[aliok]

@parth721 Can you send a PR please

[parth721]

Sure, need some time, exams are going on.

[parth721]

due to some personal issue i got engaged in other task right now.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[aliok]

/remove-lifecycle stale