Skbn is a tool for copying files and directories between Kubernetes and cloud storage providers. It is named after the 1981 video game Sokoban. Skbn uses an in-memory buffer for the copy process, to avoid excessive memory consumption. Skbn currently supports the following providers:
- AWS S3
- Minio S3
- Azure Blob Storage
- Google Cloud Storage
- git
mkdir -p $GOPATH/src/github.com/rubroboletus && cd $_
git clone https://github.com/rubroboletus/skbn.git && cd skbn
make
skbn cp \
--src k8s://<namespace>/<podName>/<containerName>/<path> \
--dst s3://<bucket>/<path>
skbn cp \
--src s3://<bucket>/<path> \
--dst k8s://<namespace>/<podName>/<containerName>/<path>
skbn cp \
--src k8s://<namespace>/<podName>/<containerName>/<path> \
--dst abs://<account>/<container>/<path>
skbn cp \
--src abs://<account>/<container>/<path> \
--dst k8s://<namespace>/<podName>/<containerName>/<path>
skbn cp \
--src k8s://<namespace>/<podName>/<containerName>/<path> \
--dst gcs://<bucket>/<path>
skbn cp \
--src ... \
--dst ... \
--parallel <n>
n
is the number of files to be copied in parallel (for full parallelism use 0)
Skbn copies files using an in-memory buffer. To control the buffer size:
skbn cp \
--src ... \
--dst ... \
--buffer-size <f>
f
is the in-memory buffer size (in MB) to use for files copy. This flag should be used with caution when used in conjunction with--parallel
- The default value for
buffer-size
is 6.75 MB, and was decided based on benchmark
Skbn supports file copy from and to a Minio S3 endpoint. To let skbn know how your minio is configured, you can set the following environment variables:
AWS_ACCESS_KEY_ID=<your username>
AWS_SECRET_ACCESS_KEY=<your password>
AWS_S3_ENDPOINT=http(s)://<host>:<port>
AWS_S3_NO_SSL=true # disables SSL
AWS_S3_FORCE_PATH_STYLE=true # enforce path style bucket access
skbn cp \
--src s3://<bucket>/<path> \
--dst abs://<account>/<container>/<path>
skbn cp \
--src abs://<account>/<container>/<path> \
--dst s3://<bucket>/<path>
skbn cp \
--src k8s://<namespace>/<podName>/<containerName>/<path> \
--dst k8s://<namespace>/<podName>/<containerName>/<path>
skbn cp \
--src s3://<bucket>/<path> \
--dst s3://<bucket>/<path>
skbn cp \
--src abs://<account>/<container>/<path> \
--dst abs://<account>/<container>/<path>
Skbn tries to get credentials in the following order:
- if
KUBECONFIG
environment variable is set - skbn will use the current context from that config file - if
~/.kube/config
exists - skbn will use the current context from that config file with an out-of-cluster client configuration - if
~/.kube/config
does not exist - skbn will assume it is working from inside a pod and will use an in-cluster client configuration
Skbn uses the default AWS credentials chain.
In addition, the AWS_REGION
environment variable should be set (default is eu-central-1
).
Skbn uses AZURE_STORAGE_ACCOUNT
and AZURE_STORAGE_ACCESS_KEY
environment variables for authentication.
Skbn uses Google Application Default Credentials.
Basically, it will first look for the GOOGLE_APPLICATION_CREDENTIALS
environment variable. If it is not defined, it will look for the default service account, or throw an error if none is configured.