Does Seldon Batch Processing Work with Azure Blob Storage?

[snowneji]

According to the tutorial here (https://docs.seldon.io/projects/seldon-core/en/v1.5.1/examples/argo_workflows_batch.html), looks like Seldon Batch Processing uses Minio as the storage, but its repo used by Helm is blocked in our corporate environment. So I'm wondering can we use Azure blog storage or GCP Storage instead?

[RafalSkolasinski]

Hi @snowneji,

We're exploring rclone-based storage initialisers / artefact managements.

There is example on custom init container that uses rclone.

I was also exploring rclone-based workflow in context of hdfs. Though main example uses hdfscli python client but there is rclone based workflows in extra folder here: https://github.com/SeldonIO/seldon-core/tree/master/examples/batch/hdfs-argo-workflows/extra-rclone

Here are some docs on rclone azure support: https://rclone.org/azureblob/

[RafalSkolasinski]

In principle the seldon-batch-processor CLI takes input/output files that are locally accessible to it.
It is the other two steps in Argoworkflow example that are responsible for pulling the input data file and pushing the output data file. These steps can be customised to your needs as you please.

[snowneji]

Thanks, I will take a look at the docs you shared