Add Ceph cluster

[celskeggs]

THIS PR IS INCOMPLETE

This PR contains code for deploying and running Ceph in a homeworld cluster.

Elements:

• Upstream source code fetching for Ceph
• Patches to the Ceph build process
• Containers for Ceph
• Kubernetes spec for Ceph deployment
• Partition creation during preseed
• New configuration fields for Ceph UUID and node subnet

Feel free to review, but this code is not yet finalized, so more changes will be coming before merge.

[celskeggs]

Current set of extra steps, after deployment:

• Work around resource overuse by aci-pull-monitor:

  $ spire foreach node systemctl stop aci-pull-monitor
  

• Rerun the key generator, because it launched too soon and didn't succeed:

  $ spire kubectl delete job ceph-gen-keyrings && spire deploy ceph
  

• Add node labels to enable ceph daemons:

  $ spire kubectl label nodes eggs-benedict huevos-rancheros ole-miss ceph-monitor=enable
  $ spire kubectl label nodes grilled-cheese avocado-burger french-toast ceph-osd=enable
  

• Enter a ceph-ready container:

  $ spire kubectl get secret ceph-keyrings -o yaml
      [copy the base64 text after client.admin.keyring:]
  $ spire kubectl get configmap ceph-config
      [copy the text starting at '[global]' and ending after 'osd crush chooseleaf type = 1']
  $ ssh -t [email protected] -- rkt run --stage1-path /usr/lib/rkt/stage1-images/stage1-coreos.aci --interactive homeworld.mit.edu/ceph-run:12.2.3-1-11 --exec /bin/bash --net=host
    # base64 -d >keyring
        [paste the earlier client.admin.keyring text]
    # cat >ceph.conf
        [paste the **DEINDENTED** version of the ceph-config text]
    # ceph -k keyring mgr module enable dashboard
    # ceph -k keyring mgr services
        [you can follow this dashboard link]
  

• Follow the dashboard link (http://HUEVOS-RANCHEROS.MIT.EDU:7000/ probably) and check that there are three monitors, three OSDs, and three managers registered.

There's something broken right now that prevents OSDs from being created properly, even though they worked a bit ago.

[celskeggs]

Since this is far from mergeable, I'm closing it.