[Bug]: Delay Model Serving Annotation until First Server

[andrewballantyne]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Today, we create every namespace with the expectation of Model Serving.

We should delay the adding of the modelmesh-enabled label until we reach the point of Model Server creation.

This can create scale issues, as this NetworkPolicy relies on this label to define network flow rules:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  labels: ...
  name: etcd
  namespace: redhat-ods-applications
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          modelmesh-enabled: "true"
    ports:
    - port: 2379
      protocol: TCP
  podSelector:
    matchLabels:
      app: model-mesh
      app.kubernetes.io/part-of: model-mesh
      component: model-mesh-etcd
  policyTypes:
  - Ingress

Expected Behavior

Delay labelling of project until we need it.

Steps To Reproduce

1. Create Project
2. View Labels on Project

Workaround (if any)

No response

What browsers are you seeing the problem on?

No response

Open Data Hub Version

Dashboard: v2.7.0

Anything else

No response

[lucferbux]

To add more context, we are adding that label in case it's missing when creating a new model server, the outcome here would be removing the label creation when creating a project.