This step is in Error state with this message: unexpected end of JSON input

[AaronUST]

Environment

• KFP has been installed on a Google Cloud kubernetes cluster
• KFP version: 1.8.5

Steps to reproduce

The error appears in the runs console when trying to perform a run. The first time I ran the same pipeline it worked fine, but the second and subsequent times I get this error.

[gkcalat]

Hi @AaronUST!
Could you please check if you have specified a correct input for the component?

[thesuperzapper]

@AaronUST @gkcalat I found the cause of this issue, please see #8942 for more information.

I imagine you will be able to fix this by removing WorkflowTaskResults RBAC access from your task executor ServiceAccount (which will often be default-editor in the profile namespaces).

However, I am not sure if argo-workflows 3.4 supports the old Pod-annotation arpproach, so you may also have to downgrade to argo-workflows 3.3.

Also, you might need to try and clear out the broken cache content if you want to try and run these tasks again.

[gkcalat]

Thank you, @thesuperzapper!

Closing this to dedupe the issues.

[LorenzoColombi]

@AaronUST @gkcalat I found the cause of this issue, please see #8942 for more information.

I imagine you will be able to fix this by removing WorkflowTaskResults RBAC access from your task executor ServiceAccount (which will often be default-editor in the profile namespaces).

However, I am not sure if argo-workflows 3.4 supports the old Pod-annotation arpproach, so you may also have to downgrade to argo-workflows 3.3.

Also, you might need to try and clear out the broken cache content if you want to try and run these tasks again.

Which is the correct way for doing this? If try getting in yaml /serviceaccount/default-editor I find this configuration:

apiVersion: v1
items:
- apiVersion: v1
  kind: ServiceAccount
  metadata:
    creationTimestamp: "2023-09-28T08:50:46Z"
    name: default
    namespace: kubeflow-user-example-com
    resourceVersion: "370279"
    uid: 829ab899-12da-4f49-a24d-680546a2dc10
- apiVersion: v1
  kind: ServiceAccount
  metadata:
    creationTimestamp: "2023-09-28T08:50:50Z"
    name: default-editor
    namespace: kubeflow-user-example-com
    ownerReferences:
    - apiVersion: kubeflow.org/v1
      blockOwnerDeletion: true
      controller: true
      kind: Profile
      name: kubeflow-user-example-com
      uid: dbe9a527-f76e-435c-b281-6c4f056fdc22
    resourceVersion: "370351"
    uid: 93109857-1c8a-4188-9de8-467d12353cbd
- apiVersion: v1
  kind: ServiceAccount
  metadata:
    creationTimestamp: "2023-09-28T08:50:50Z"
    name: default-viewer
    namespace: kubeflow-user-example-com
    ownerReferences:
    - apiVersion: kubeflow.org/v1
      blockOwnerDeletion: true
      controller: true
      kind: Profile
      name: kubeflow-user-example-com
      uid: dbe9a527-f76e-435c-b281-6c4f056fdc22
    resourceVersion: "370359"
    uid: 19c5b5f3-15d9-464d-bf4f-02d07d47833e
kind: List
metadata:
  resourceVersion: ""