Add S3 sync jobs

[upodroid]

Fixes: kubernetes/k8s.io#3807

Adding a Prow Job to write blobs to S3.

The script isn't ready yet so for now it is just going to authenticate to AWS to test that it works. @BobyMCbobs will raise a PR later with the scripts.

AWS Infra Change: cncf-infra/aws-infra#15
Prow Infra Change: kubernetes/k8s.io#4170
We will be using an AWS service to replicate the blobs from the primary bucket to all the other S3 buckets. kubernetes/k8s.io#4118

[upodroid]

/cc @ameukam

[BobyMCbobs]

I will convert this one into a sync between the US GCS bucket->s3 us-east-2 for registry.k8s.io.
Might create a periodic per region for syncing s3->s3.

[BobyMCbobs]

Looks like we'll need a custom container image for use in the job.
The other dependency we need, besides awscli is rclone.

Unsure what the policy on building such images is, whether to use existing external images as base (such as amazon/aws-cli) and pin it and use a differently tagged version, or to start from scratch and to bring in awscli via python-pip

[dims]

/approve
/lgtm

Looks like we will be iterating. The second variation (awscli using python-pip) will probably give us better control over what we need.

/hold feel free to remove hold when ready

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes/sig-k8s-infra/trusted/releng/OWNERS [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[upodroid]

This can be merged once kubernetes/k8s.io#4185 has been merged and the IAM roles on the prow bucket has been added.

[upodroid]

/hold cancel

[upodroid]

/retest-required

[k8s-ci-robot]

@upodroid: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key releng-gcs-to-aws-sync.yaml using file config/jobs/kubernetes/sig-k8s-infra/trusted/releng/releng-gcs-to-aws-sync.yaml

In response to this:

Fixes: kubernetes/k8s.io#3807

Adding a Prow Job to write blobs to S3.

The script isn't ready yet so for now it is just going to authenticate to AWS to test that it works. @BobyMCbobs will raise a PR later with the scripts.

AWS Infra Change: cncf-infra/aws-infra#15
Prow Infra Change: kubernetes/k8s.io#4170
We will be using an AWS service to replicate the blobs from the primary bucket to all the other S3 buckets. kubernetes/k8s.io#4118

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.