Proposal: Key Sever Migration Support

[mikehelmick]

TL;DR

For various reasons, some jurisdictions that have launched exposure notifications systems would like to change servers (usually to merge with neighboring jurisdictions).

We make a couple of assumptions / state some facts

1. The severs are not able to share export signing keys

2. It is not possible for there to be 2 active export signing keys for an application and that active signing key version only rolls forward from a -> b

3. There is some transition period while the end user applications switch over where they read and write from. Because of the way that signing keys are published, an application can only read from one server.

4. Bidirectional federation is not an option either due to size of keys that would have to be replicated to the smaller jurisdiction or legal agreements that might need to be put into place to allow for this.

In order to move a jurisdiction from one key sever to another we need to transition where their client applications read exports from and publish TEKs to and we need to ensure that clients that haven't yet migrated are able to get the unified key set.

Overview of proposed process

Quick path ⏱️

Ideally, applications will have a built in remote configuration capability that allows the jurisdiction to dynamically switch from one server to another. This allows for a rapid cutover and a one time ETL to move recently upload keys from server A to server B.

This is dependent on public key rotation speed as well. Even in the quick path, we may have to execute the not so quick path for a period.

Not so quick path ⌛

If, the only way to make this switch is through pushing an application update, we have to plan for a prolonged period where older versions only know how to talk to the old server.

We use this high level process, assuming a move from server A to server B

1. Set up outbound federation from server A to server B

2. Start the proxy / resigner server at server A (see design below). At this point in time, operators of server A stop generating their own export files and all of their keys transit through server B and the proxied exports.

3. Change the public key for the application to be the public key for server B

4. Push the application update. New versions of the app will only communicate with server B. Clients that haven't updated will only communicate with server A. Once download traffic to server A subsides, that sever can be decommissioned.

Design

Proposal

Create an export proxy job that knows how to mirror exports from one server into the CDN storage of the other.

During the proxy, the zip file is unpacked, a second signature is added, and the zip file is repackaged.

Once the key has been rotated for the app to just the sever the exports originate from, the resigner functionality can be turned off. At that point, it's just mirroring the export files from one CDN to another.

Alternatives considered

Dual read based solutions are not viable because of the key rotation limitations.

/kind enhancement

[gurayAlsac]

If constraint 1 can be relaxed somewhat and both export servers can use a temporary manually managed (http://cloud/kms/docs/importing-a-key) transition key, that might trade the complexity of the proxy server for one additional key rotation and two-way federation. Perhaps not a huge win in terms of complexity, but might be less code needing to get written.

[mikehelmick]

This plan needs to be revisited. The proposal with adding signature blocks will not work since the listed signatories is part of what is signed. We did a good job at preventing tampering.

It's on me to think of another plan

/assign

[mikehelmick]

The current best path forward is

1. Enable bidirectional federation
2. Gradually shift users over to target server.

We could implement a second form of federation that works by importing export files. This would be easier to configure and maintain than the existing gRPC protocol.

[gurayAlsac]

I suppose if its a single server A migrating to server B, server B can start including the SignatureInfo from server A but a dummy signature. Then the listed signatories could match during the transition. But if there are lots of A's migrating to a B that might be more challenging.

[mikehelmick]

I suppose if its a single server A migrating to server B, server B can start including the SignatureInfo from server A but a dummy signature. Then the listed signatories could match during the transition. But if there are lots of A's migrating to a B that might be more challenging.

That's an interesting take. I think we need a code change at the server to support that, but it's possible. Thanks for the idea!

[mikehelmick]

Download resigning will not work because the list of signatories is part of what is signed. Adding another one will invalidate the signature.

Updated proposal

First - enhance federation to support v1.5 fields + revised keys. This is in progress.

Migration process will look like - moving from local server A to larger region B.

1. Enable federation from server A to server B. - TEKs from server A will start appearing in server B's exports

2. Make region B's signing key available to apps, but leave at 0% / on-demand only

3. At the same time:

   • Push app update that causes clients to write to / read from server B only. People who take the app update are done.

   • Mirror server B's exports on server A's download path. People who haven't updated their app will publish TEKs to sever A, and those TEKs will be federated up to server B. Export files from server B will contain all keys and will be mirrored on server A so people who haven't update the app will see the full picture.

[georules]

It sounds like this solution requires a future release level of the exposure-notifications-server (referencing future v1.5 spec support). Am I correct in this interpretation?

[mikehelmick]

The server as a whole already supports v1.5+. The federationin and federationout servers need to be updated w/ such support. This is ~50% done and I can finish that work once all agree on this plan.

[day-jeff]

Mirror server B's exports on server A's download path.

How does this work?

[mikehelmick]

Mirror server B's exports on server A's download path.

How does this work?

Most likely full copies would need to be downloaded and stored, correcting for the expected base path.

[mikehelmick]

Still to do

• Terraform for export-importer job

• Cloud rsync - mirror export files from one CDN to another

  • meant to be intelligent, looking at index.txt, adding new, deleting old