Temporal Log Sharding

[dlorenc]

We need to plan a way to shard out the log by time. CT logs typically start fresh every year, and require users to fetch entries from the correct log by knowing when the certificate was issued.

We could do this easily - but it might be nice to think through an API server that aggregates multiple logs, to allow for querying across a range of them. A frozen log is still useful, artifacts run for many years. It becomes easier to publish the entire year as a frozen merkle tree, and querying can even become easier with multiple verifiable maps built up on different indices.

[dlorenc]

Ref #352

[dlorenc]

High level ideas:

We could maintain a (signed) shard map, which specifies:

• The root hash for each log
• Which index ranges belong to each log

When you query by index, we can easily look up the correct log using these ranges
When you query by UUID, there should be at most one log with the right entry. How can we make sure of this? Should we iterate through all the logs?

[lkatalin]

I'm adding @dlorenc 's comments from #433 here to keep it all in one place:

Up next:

• support starting the server with multiple log IDs, instead of just one.
  • this will probably look like a config file containing a map of index ranges to log ids.
  • only one can be "active"
• support parsing the UUID during GET calls to determine which log to query (with defaulting for the 0 case here)

Also related: #352
Also related: #255

[dlorenc]

Also see: #487

[lkatalin]

  * this will probably look like a config file containing a map of index ranges to log ids.

Do we still want a config file, since we have code in place that does this mapping?

[lkatalin]

Related:
#629 #630 #631 #632

[haydentherapper]

Logs are sharded!