Document Corruption/Recovery Plan

[dlorenc]

Placeholder to figure out, design,and document our corruption/recovery plan.

Ref https://groups.google.com/a/chromium.org/g/ct-policy/c/PCkKU357M2Q/

At a high level, I think we would freeze the log and create a new one at the point of corruption. We'd have to replay any entries that came in after the corruption into the new log, very very carefully and transparently.

We also need a plan in general for temporal sharding of logs. I'll open another issue for that.

[lukehinds]

I guess a key part of this is also how we would know a failure has occurred, which is where our monitors would come in.

[haydentherapper]

With sharding, this should be resolved. We also have playbooks now