From 31dbf96b8f5e6b138becdac187ba4f535c65cee4 Mon Sep 17 00:00:00 2001 From: Michael Snoyman Date: Tue, 14 Aug 2018 12:25:20 +0300 Subject: [PATCH] Write-up on curator tool --- subs/curator/README.md | 74 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/subs/curator/README.md b/subs/curator/README.md index 1971084af1..4372697aa1 100644 --- a/subs/curator/README.md +++ b/subs/curator/README.md @@ -1,3 +1,77 @@ # curator Snapshot curator tool for, e.g., creating Stackage snapshots. + +This is the "curator 2.0", replacing +https://github.com/fpco/stackage-curator. It relies on pantry for +finding appropriate packages, and Stack for performing the builds. It +is intended to be much simpler to maintain than the old +stackage-curator tool. + +## Incomplete! + +This tool is not yet complete. Here's a (likely incomplete) list of +things that still need to be handled to replace `stackage-curator`: + +* Collect the Haddocks in a way that stackage-server can handle them +* Proper CLI, right now the `app/Main.hs` just runs through a bunch of + steps. We need to have individual commands like the current tool, so + each command can be called in an appropriately locked-down Docker + container. +* Logic for uploading generated snapshots and other info to Github, + S3, etc. +* Ability to roll an LTS minor version bump. +* Ability to specify package locations from Git. +* External, but: stackage-server needs to be updated to support the + new snapshot format/location + +## Basic workflow + +Here's a rundown of how this tool is intended to be used. + +We update the Hackage index to get a list of all of the most recent +package versions. This is pantry's `updateHackageIndex` command. + +We start with `build-constraints.yaml`, the configuration file in +commercialhaskell/stackage. This specifies all of the packages we want +to include in a snapshot, along with a bunch of configuration. + +We parse `build-constraints.yaml` and convert it into the +`constraints.yaml` file, which contains a more properly structures set +of constraints. We'll continue to let users edit the +`build-constraints.yaml` file, since it's more user-friendly. But +`constraints.yaml` gives us more flexibility. + +* For LTS minor bumps, instead of generating `constraints.yaml` from + `build-constraints.yaml`, we'll take the `constraints.yaml` used for + the last LTS release in the series. Details still need to be worked + out on how upper bounds are added and where this file is stored. + +Curator team: at this point, you can edit `constraints.yaml` to make +tweaks to the build plan. This replaces the old `CONSTRAINTS` +environment variable. + +We combine the `constraints.yaml` file and the information from +Hackage to produce `snapshot-incomplete.yaml`. This has a concrete +list of all of the packages we intend to include in the +snapshot. Again, this file can be manually modified if desired. + +* When we support Git repos, we'll also be checking those repos to + find the latest appropriate release. We'll need to figure out + exactly how that plays in with LTS upper bounds; I'm thinking we'll + have logic like "use commit X, or the latest if it meets version + range Y." + +The `snapshot-incomplete.yaml` file does not have all of the +cryptographic hashes necessary for fully reproducible builds. We next +generate `snapshot.yaml` with all of this information. This file +should _never be manually edited_, instead edits should occur at the +`snapshot-incomplete.yaml` and `constraints.yaml` phases. + +We unpack all of the package specified by `snapshot.yaml` into a local +directory, and generate a `stack.yaml` that gives instructions to +build all of those packages. + +We build the packages, run test suites, and generate Haddocks. + +__TODO__ Grab artifacts and upload them to the right place.