allow output artifact store configuration (vs hard coded)

[yaronha]

it seems like the output artifacts are always stored in a specific minio service, port, namespace, bucket, secrets, etc (minio-service.kubeflow:9000).

see:

pipelines/sdk/python/kfp/compiler/_op_to_template.py

Line 148 in f40a22a

def _build_conventional_artifact(name, path):

it would be great to make it flexible, e.g. allow using S3, or change namespace or bucket names.
i suggest making it configurable, i can do such PR if we agree its needed.

[Ark-kun]

There is already a PR for this. But we'd like to solve this in a different way - by having an in-cluster configMap with the storage configuration. Argo already supports that, but we're hesitating to use that mechanism since KF Pipelines might not be the exclusive user of the Argo controller.

Fortunately, the problem will be solved in Argo 2.4 (per-pipeline configMap reference)

[Ark-kun]

Also #363 #1195 #362

[yaronha]

@Ark-kun a mechanism like config map sounds better, it will also make the YAMLs cleaner, currently u get the same 27 lines added to every container, see below, would be nice to hide it:

    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact

[Ark-kun]

Fixed by #1297