add mirror service

[mikehelmick]

Towards #928

Proposed Changes

• Add mirror service that can mirror an index.txt / export archive structure from a CDN to the configured cloud storage. For use in key server migrations.

Release Note

New: Mirror service `cmd/mirror` for mirroring an export directory from one key server to another.

[sethvargo]

I think the mirror algorithm is fine, but the implementation has a few flaws. Since everything is a giant loop, none of the defers are running until the very end. Furthermore, for a very large N, we'll hit unnecessary timeouts due to lack of parallelization.

I'd like to suggest we:

1. Break apart the main HTTP handler into (at least) one testable function.

2. Leverage some parallelization (either primitively with goroutines or using a library like https://github.com/gammazero/workerpool) with a configurable max.

3. Add tests for internal/mirror/mirror.go. The blobstore should be stubbable to the in-memory implementation.

[sethvargo]

I'd really like to see this broken down into smaller functions that are easily tested. The current HTTP handler is very long and very nested.

[mikehelmick]

can do as a follow up if that's OK

[sethvargo]

yup

[mikehelmick]

I think the mirror algorithm is fine, but the implementation has a few flaws. Since everything is a giant loop, none of the defers are running until the very end. Furthermore, for a very large N, we'll hit unnecessary timeouts due to lack of parallelization.

Parallelization is achieved at the HTTP layer - will have a cron to active this service every 60 seconds. if one mirror is locked and taking a while, the next invocation will move onto the next one.

State is saved on an http request timeout and processing will continue on next invoke.

I'd like to suggest we:

1. Break apart the main HTTP handler into (at least) one testable function.

will do in follow up.

2. Leverage some parallelization (either primitively with goroutines or using a library like https://github.com/gammazero/workerpool) with a configurable max.

we can add additional parallelism, but doesn't seem to be necessary in the testing i've done.

3. Add tests for internal/mirror/mirror.go. The blobstore should be stubbable to the in-memory implementation.

would like to do in follow up

[sethvargo]

/lgtm
/hold

in case someone else wants to give it a once-over. Feel free to unhold.

[sethvargo]

yup

[google-oss-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikehelmick, sethvargo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mikehelmick,sethvargo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mikehelmick]

/cancel hold

[mikehelmick]

/unhold