Carbonplan: Provide service account for dask workers and notebook pods

[jhamman]

Summary

We would like to be able to write to S3 using Dask Workers. Ideally we do this with some sort of service account that is shared between notebook and worker pods.

User Stories

• As a Dask Gateway user, I want to read/write to s3 without having to manually pass credentials around
• As a hub administrator, I want to make sure people don't have to manually pass security credentials to workers

Acceptance criteria

• The following pseudo code should work:

cluster = GatewayCluster()
cluster.scale(4)
client = cluster.get_client()

ds = xr.open_dataset('s3://path-to-zarr-store', engine='zarr')  # currently works only for public data
ds.to_zarr('s3://path-to-another-store')  # currently fails because workers don't have write credentials

Important information

• This has been done quite successfully on the Pangeo hubs
• See the s3fs section on credentials: https://s3fs.readthedocs.io/en/latest/index.html#credentials
• See this TF script: https://github.com/pangeo-data/terraform-deploy/blob/73b7de909413a8189e2391bec7ab444ad740a0ae/aws-examples/hackweek-infrastructure/infrastructure/s3-data-bucket.tf

Tasks to complete

I think this is the basic workflow but I've only done this on GCP.

• Discuss and refine the dask workers / notebook pods issue #614
• Create a service account
• Attach service account to kubernetes pods
• Give service account write permissions on s3
• Decide how we'd like to generalize this for our other infrastructure

[damianavila]

I think these links might be relevant to the discussion:

• Feature - Support AWS IAM Roles for Service Accounts kubernetes/kops#8264
• https://dev.to/olemarkus/irsa-support-for-kops-1doe

[jhamman]

Hi folks! Wondering if there is anything specific we can do to help move this one forward? For most of our use cases, we've been able to work around this but it is starting to hold up a few workflows.

[yuvipanda]

@jhamman try this out now?