Unable to deploy app to OCP with Knative

[tam512]

On Power 9 OCP, installed Open Liberty Operator 1.4.0, install RH Serverless operator and Knative serving using script https://github.ibm.com/websphere/operators/tree/main/scripts/configure-cluster/setup-ocp-cluster.sh -S -K --arch P

• In the namespace to deploy app, create secret to pull app image from artifactory

oc get secret hyc-secret
NAME         TYPE                             DATA   AGE
hyc-secret   kubernetes.io/dockerconfigjson   1      5h37m

• Here is my OpenLibetyApplication yaml

apiVersion: apps.openliberty.io/v1
kind: OpenLibertyApplication
metadata:
  name: dt10-olk-java21-kn
  labels:
    app: daytrader10
spec:
  serviceAccount:
    mountToken: true
    name: instanton-sa
  applicationImage: 'my-app-image'
  createKnativeService: true
  expose: true 
  autoscaling:
    maxReplicas: 5
    minReplicas: 1
    targetCPUUtilizationPercentage: 80
  pullPolicy: Always
  pullSecret: hyc-secret
  statefulSet: {}
  service:
    type: ClusterIP
    port: 9080
  route:
    termination: passthrough
  resources:
    limits:
      cpu: 1024m
      memory: 1024Mi
    requests:
      cpu: 512m
      memory: 512Mi
  probes:
    liveness:
      httpGet:
        path: /health
        port: 9080
      failureThreshold: 5
      periodSeconds: 10
    readiness:
      httpGet:
        path: /daytrader
        port: 9080
      failureThreshold: 5
      periodSeconds: 10
  env:
    - name: WLP_LOGGING_CONSOLE_FORMAT
      value: 'simple'
    - name: dbHost
      value: 'app-db2' # DB2 container service.namespace
    - name: dbName
      value: TRADEDB
    - name: dbPort
      value: "50000"
    - name: dbUser
      value: db2inst1
    - name: dbPasswd
      valueFrom:
        secretKeyRef:
          name: db-credential
          key: dbpw
    - name: QueueManager
      value: QM1
    - name: mqHost
      value: app-mq  #use MQ container service.namespace
    - name: mqPasswd
      valueFrom:
        secretKeyRef:
          key: mqpw
          name: mqapp-credential
    - name: httpPort
      value: "9080"
    - name: httpsPort
      value: "9443"
  securityContext:
    allowPrivilegeEscalation: true
    privileged: false
    runAsNonRoot: true
    capabilities:
      add:
      - CHECKPOINT_RESTORE
      - SETPCAP
      drop:
      -  ALL

My app image failed to deployed. I see Authentication is required when describe revision

Events:
  Type     Reason         Age               From                 Message
  ----     ------         ----              ----                 -------
  Warning  InternalError  6s (x4 over 13s)  revision-controller  Unable to fetch image "docker-na-public.artifactory.swg-devops.com/hyc-wassvt-team-image-registry-docker-local/instanton/24.0.0.11/daytrader10-mq:ol-kernel-java21-ppc64le": failed to resolve image to digest: GET https://docker-na-public.artifactory.swg-devops.com/artifactory/api/docker/null/v2/token?scope=repository%3Ahyc-wassvt-team-image-registry-docker-local%2Finstanton%2F24.0.0.11%2Fdaytrader10-mq%3Apull&service=docker-na-public.artifactory.swg-devops.com: : Authentication is required

Check revision and kservice resource and I do not pullSecret in them

[tam512]

This is a regression since this worked in Liberty 24.0.0.9

[tam512]

This problem is also seen on x86 OCP

[leochr]

At a glance in the Operator code, I don't see the pull secret from CR being set on Knative workloads (now or before). So I don't believe this is a regression. A global pull secret was likely in place in the clusters where the image pull failure didn't occur in the past. We'll look into setting the pull secret on Knative workloads.

[tam512]

Our experience in the past was that we always need to have a pull secret in the namespace of the app even when we have a global pull secret. Knative does not work with global pull secret.

We also have global pull secret in OCP when we saw this problem

[idlewis]

@tam512 Your original report says that you saw this issue on Power. Have you also seen the issue on X or is it Power only?

[idlewis]

@tam512 Your original report says that you saw this issue on Power. Have you also seen the issue on X or is it Power only?
@tam512 Ah, sorry, just seen your later update. Pls ignore!

[idlewis]

I think this is working as designed.
In CR yaml above, both the pull secret but also the service account name are specified.
For the container image to be pulled, the pull secret has to be added to the service account.
However, the operator doesn't add the pull secret for custom service accounts, it only does this when the operator has created the service account itself. This is not a change in behaviour.
We will look into whether we can support the scenario of 'custom service account and pull secret specifed' in a future release.

For the moment, you can get this scenario to work if you specify the pull secret when you create the custom service account

[leochr]

@tam512 Please see Iain's response above. The pull secret is specified via the Service Account and Liberty Operator adds the pull secret only if the default service account it manages is used. For custom service accounts, the pull secret should be added by the user. Please try that.

Closing. Please reopen if the suggestion doesn't work.