diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 0e7aeb8..1e98e7c 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -11,5 +11,11 @@ jobs: - uses: actions/checkout@v4 - uses: DeterminateSystems/nix-installer-action@main - uses: DeterminateSystems/magic-nix-cache-action@main - - run: nix build -L + - run: | + hostnames=$(nix eval --raw ".#nixosConfigurations" --apply 'x: builtins.concatStringsSep " " (builtins.attrNames x)') + for h in $hostnames; do + attribute="nixosConfigurations.$h.config.system.build.toplevel" + nix build -L .#$attribute --no-link + done + - run: nix flake check -L diff --git a/README.md b/README.md index 005d11b..a239787 100644 --- a/README.md +++ b/README.md @@ -9,17 +9,19 @@ You need Nix installed on your local machine. You can use the curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install ``` -## How do I deploy to the server? +## Standard operating procedures +- [Provisioning a new machine](playbooks/new-machine-provisioning.md) +- [Deploying a new instance](playbooks/new-packit-instance.md) +- [Wiping a Packit instance](playbooks/wipe-packit-instance.md) + +## How do I? +### How do I deploy to the server? ```sh -nix run .#deploy +nix run .#deploy ``` -TODO: doing builds locally and then uploading them is pretty inefficient. -There's almost certainly a way to do the build remotely and not have to transfer -anything. - -## How do I update outpack_server or Packit? +### How do I update outpack_server or Packit? ``` nix run .#update packit @@ -34,40 +36,40 @@ It also accepts a `--branch` argument which may be used to specify a branch name. Otherwise the default branch of the repository will be used (usually main or master). -## How do I build the deployment? +### How do I build the deployment? ```sh -nix build +nix build .#nixosConfigurations..config.system.build.toplevel ``` This is useful to check syntax and that everything builds, but you don't typically need to do it. Deploying will do it implicitely. -## How do I visualize the effect of my changes? +### How do I visualize the effect of my changes? ``` -nix run .#diff +nix run .#diff ``` This will download the system that is currently running on the server and compare it with what would be deployed using [nix-diff](https://github.com/Gabriella439/nix-diff). -`nix-diff` will omit derivations' environment comparison if some dependencies -already differ. `nix run .#diff -- --environment` can help print all of these, -but is likely to be very verbose. +Alternatively, when opening a pull request on GitHub an action will run and +compute the difference against the main branch for all machines, and post the +result as a comment. -## How do I start a local VM? +### How do I start a local VM? ```sh -nix run .#start-vm +nix run .#start-vm ``` This starts a local VM running in QEMU. Handy to check everything works as expected before deploying. -When starting, this command will obtain a Vault token and inject it into the VM -to be used for fetch GitHub client ID and secrets. If needed, you may be -prompted for your GitHub personal access token. +Before starting the local VM, this command will obtain a Vault token and inject +it into the VM to be used for fetching GitHub client ID and secrets. If needed, +you may be prompted for your GitHub personal access token. Port 443 of the VM is exposed as port 8443, meaning you may visit https://localhost:8443/ once the VM has started. nginx is configured using a @@ -81,11 +83,11 @@ the VM console. grant-role ADMIN ``` -## How do I run the integration tests? +### How do I run the integration tests? ```sh -nix run .#vm-test -nix run .#vm-test -- --interactive +nix run .#integration-test +nix run .#integration-test -- --interactive ``` The second command starts a Python session which may be used to interact with the test machine. @@ -116,7 +118,7 @@ EOF sudo udevadm control --reload-rules && sudo udevadm trigger ``` -## How do I add new SSH keys? +### How do I add new SSH keys? The keys are fetched from GitHub and committed into this repository as the `authorized_keys` files. You can edit the `scripts/update-ssh-keys.sh` file to @@ -130,36 +132,7 @@ nix run .#update-ssh-keys Carefully review the changes to avoid locking yourself out and re-deploy to the server. -## How do I add a new Packit instance? - -Edit `services.nix` by adding a new entry to the `services.multi-packit.instances` -list. - -You will need to customize the instance by configuring some of the -`services.packit-api.` options. A GitHub organisation and team needs to -be specified. All members of this organisation/team will be allowed to access -the instance. The team can be omitted or left blank, in which case any member -of the organisation will have access. - -The Packit application on Github manually needs to be granted permission, by -one of the organisation's admins, to access the organisation. See [the GitHub -documentation][github-oauth-org]. Because we use a single OAuth app for all -instances, this only needs to be done once per org, even if uses by multiple -instances. - -[github-oauth-org]: https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-your-membership-in-organizations/requesting-organization-approval-for-oauth-apps - -The initial user needs to be granted the ADMIN role manually. - -1. Log in to the instance with your GitHub account. -1. SSH onto the server. -1. Run `grant-role ADMIN` where `` is the name - of the instance and `` is your GitHub username. -1. Log out and back in for the changes to take effect. - -Afterwards permissions may be managed through the web UI. - -## How do I add secrets to a machine? +### How do I add secrets to a machine? All secrets are fetched from Vault. To add a new secret to a machine, add it to the Vault and add an entry to the `vault.secrets` option. @@ -187,7 +160,7 @@ nix run .#nixosConfigurations..config.vault.tool -- --root here This will fetch all the secrets as the command on the server would have done, but store them relative to the `here` folder. -## How do I inspect / edit the database? +### How do I inspect / edit the database? If doing this on the production instance, start by thinking very carefully about what you are about to do. @@ -205,14 +178,10 @@ The database is initially populated by the packit-api service. During the first start of a new instance, it can take a minute for the service to start and for the database's tables to exist. -## How do I update NixOS? +### How do I update NixOS? ```sh nix flake update ``` To switch to a new major version, you should edit the URL at the top of `flake.nix`. - -## How do I provision a new machine? - -See [`PROVISIONING.md`](PROVISIONING.md). diff --git a/flake.lock b/flake.lock index e1afb25..e0227db 100644 --- a/flake.lock +++ b/flake.lock @@ -20,6 +20,24 @@ "type": "github" } }, + "flake-parts": { + "inputs": { + "nixpkgs-lib": "nixpkgs-lib" + }, + "locked": { + "lastModified": 1727826117, + "narHash": "sha256-K5ZLCyfO/Zj9mPFldf3iwS6oZStJcU4tSpiXTMYaaL0=", + "owner": "hercules-ci", + "repo": "flake-parts", + "rev": "3d04084d54bedc3d6b8b736c70ef449225c361b1", + "type": "github" + }, + "original": { + "owner": "hercules-ci", + "repo": "flake-parts", + "type": "github" + } + }, "nixpkgs": { "locked": { "lastModified": 1726447378, @@ -36,9 +54,22 @@ "type": "github" } }, + "nixpkgs-lib": { + "locked": { + "lastModified": 1727825735, + "narHash": "sha256-0xHYkMkeLVQAMa7gvkddbPqpxph+hDzdu1XdGPJR+Os=", + "type": "tarball", + "url": "https://github.com/NixOS/nixpkgs/archive/fb192fec7cc7a4c26d51779e9bab07ce6fa5597a.tar.gz" + }, + "original": { + "type": "tarball", + "url": "https://github.com/NixOS/nixpkgs/archive/fb192fec7cc7a4c26d51779e9bab07ce6fa5597a.tar.gz" + } + }, "root": { "inputs": { "disko": "disko", + "flake-parts": "flake-parts", "nixpkgs": "nixpkgs" } } diff --git a/flake.nix b/flake.nix index d77bac4..be7c00b 100644 --- a/flake.nix +++ b/flake.nix @@ -2,149 +2,33 @@ inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05"; inputs.disko.url = "github:nix-community/disko"; inputs.disko.inputs.nixpkgs.follows = "nixpkgs"; - - outputs = { nixpkgs, self, ... } @inputs: - let - pkgsArgs = { - overlays = [ self.overlays.default ]; - config.allowUnfreePredicate = pkg: builtins.elem (nixpkgs.lib.getName pkg) [ + inputs.flake-parts.url = "github:hercules-ci/flake-parts"; + + outputs = inputs: inputs.flake-parts.lib.mkFlake { inherit inputs; } ({ withSystem, lib, self, nixpkgs, ... }: { + imports = [ + ./packages + ./machines + ./scripts + ./tests + ]; + perSystem = { system, pkgs, ... }: { + _module.args.pkgs = import inputs.nixpkgs { + inherit system; + config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [ "vault" "vault-bin" ]; }; - in - { - overlays.default = (final: prev: { - packit = final.callPackage ./packages/packit { }; - outpack_server = final.callPackage ./packages/outpack_server { }; - packit-app = final.callPackage ./packages/packit/packit-app.nix { }; - packit-api = final.callPackage ./packages/packit/packit-api.nix { }; - fetch-secrets = final.writers.writePython3Bin "fetch-secrets" - { - libraries = [ final.python3.pkgs.hvac ]; - } ./scripts/fetch-secrets.py; - }); - nixosConfigurations.wpia-packit = nixpkgs.lib.nixosSystem { - system = "x86_64-linux"; - specialArgs = { inherit inputs; }; - modules = [ - ./machines/wpia-packit.nix - ./hardware-configuration.nix - { nixpkgs = pkgsArgs; } + devShells.default = pkgs.mkShell { + buildInputs = [ + pkgs.nix-prefetch-github + pkgs.nixos-rebuild + pkgs.nix-diff ]; }; - - nixosConfigurations.wpia-packit-private = nixpkgs.lib.nixosSystem { - system = "x86_64-linux"; - specialArgs = { inherit inputs; }; - modules = [ - ./machines/wpia-packit-private.nix - ./hardware-configuration.nix - { nixpkgs = pkgsArgs; } - ]; - }; - - packages.x86_64-linux = - let pkgs = import nixpkgs ({ system = "x86_64-linux"; } // pkgsArgs); - in { - inherit (pkgs) outpack_server packit-app packit-api packit fetch-secrets; - - default = self.nixosConfigurations.wpia-packit.config.system.build.toplevel; - - deploy = pkgs.writeShellApplication { - name = "deploy-wpia-packit"; - runtimeInputs = [ pkgs.nixos-rebuild ]; - text = '' - nixos-rebuild switch \ - --flake .#wpia-packit \ - --target-host root@packit.dide.ic.ac.uk \ - --use-substitutes - ''; - }; - - deploy-private = pkgs.writeShellApplication { - name = "deploy-wpia-packit-private"; - runtimeInputs = [ pkgs.nixos-rebuild ]; - text = '' - nixos-rebuild switch \ - --flake .#wpia-packit-private \ - --target-host root@packit-private.dide.ic.ac.uk \ - --use-substitutes - ''; - }; - - update-ssh-keys = pkgs.writeShellApplication { - name = "update-ssh-keys"; - runtimeInputs = [ pkgs.curl ]; - text = builtins.readFile ./scripts/update-ssh-keys.sh; - }; - - diff = pkgs.writeShellApplication { - name = "diff"; - runtimeInputs = [ pkgs.nix pkgs.nix-diff ]; - text = '' - current=$(ssh root@packit.dide.ic.ac.uk readlink /run/current-system) - nix-copy-closure --from root@packit.dide.ic.ac.uk "$current" - - # In theory .drvPath should allow us to use string interpolation - # instead of re-evaluating the flake, but for some reason it - # pulls all the build-time dependencies. - target=$(nix path-info --derivation .#nixosConfigurations.wpia-packit.config.system.build.toplevel) - nix-diff "$@" "$current" "$target" - ''; - }; - - update = pkgs.writers.writePython3Bin "update" { } ./scripts/update.py; - - start-vm = pkgs.writeShellApplication { - name = "start-vm"; - runtimeInputs = [ pkgs.vault-bin ]; - text = - let vaultUrl = self.nixosConfigurations.wpia-packit.config.vault.url; - in '' - token=$(vault print token) - if [[ -z $token ]]; then - echo "Logging in to ${vaultUrl}" - token=$(env VAULT_ADDR="${vaultUrl}" vault login -method=github -field=token) - fi - - exec ${nixpkgs.lib.getExe self.nixosConfigurations.wpia-packit.config.system.build.vm} \ - -fw_cfg name=opt/vault-token,string="$token" "$@" - ''; - }; - - start-vm-private = pkgs.writeShellApplication { - name = "start-vm-private"; - runtimeInputs = [ pkgs.vault-bin ]; - text = - let vaultUrl = self.nixosConfigurations.wpia-packit-private.config.vault.url; - in '' - token=$(vault print token) - if [[ -z $token ]]; then - echo "Logging in to ${vaultUrl}" - token=$(env VAULT_ADDR="${vaultUrl}" vault login -method=github -field=token) - fi - - exec ${nixpkgs.lib.getExe self.nixosConfigurations.wpia-packit-private.config.system.build.vm} \ - -fw_cfg name=opt/vault-token,string="$token" "$@" - ''; - }; - - vm-test = self.checks.x86_64-linux.default.driver; - }; - - checks.x86_64-linux.default = - let pkgs = import nixpkgs ({ system = "x86_64-linux"; } // pkgsArgs); - in pkgs.callPackage ./tests { inherit inputs; }; - - devShells.x86_64-linux.default = - let pkgs = import nixpkgs ({ system = "x86_64-linux"; } // pkgsArgs); - in pkgs.mkShell { - buildInputs = [ - pkgs.nix-prefetch-github - pkgs.nixos-rebuild - ]; - }; }; + + systems = [ "x86_64-linux" ]; + }); } diff --git a/machines/common/configuration.nix b/machines/common/configuration.nix index bd8c1d5..373d713 100644 --- a/machines/common/configuration.nix +++ b/machines/common/configuration.nix @@ -1,14 +1,12 @@ -{ lib, pkgs, inputs, ... }: { +{ self, config, lib, pkgs, self', inputs, ... }: { imports = [ - ../../disk-config.nix - ../../tools.nix + ./tools.nix ../../modules/multi-packit.nix ../../modules/vault.nix ../../modules/outpack.nix ../../modules/packit-api.nix ../../modules/metrics-proxy.nix ./services.nix - inputs.disko.nixosModules.disko ]; @@ -25,12 +23,13 @@ networking.firewall.enable = true; networking.firewall.allowedTCPPorts = [ 22 80 443 9000 ]; + networking.domain = "dide.ic.ac.uk"; environment.systemPackages = [ pkgs.curl pkgs.htop pkgs.vim - pkgs.outpack_server + self'.packages.outpack_server pkgs.gitMinimal ]; @@ -45,5 +44,9 @@ nix.settings.experimental-features = [ "nix-command" "flakes" ]; + # Keep derivations when garbage collecting the Nix store. We use these + # derivations to run `nix-diff` against a running system. + nix.settings.keep-derivations = true; + system.stateVersion = "24.05"; } diff --git a/disk-config.nix b/machines/common/disk-config.nix similarity index 100% rename from disk-config.nix rename to machines/common/disk-config.nix diff --git a/hardware-configuration.nix b/machines/common/hardware-configuration.nix similarity index 100% rename from hardware-configuration.nix rename to machines/common/hardware-configuration.nix diff --git a/tools.nix b/machines/common/tools.nix similarity index 69% rename from tools.nix rename to machines/common/tools.nix index e85d063..45f207a 100644 --- a/tools.nix +++ b/machines/common/tools.nix @@ -1,9 +1,9 @@ -{ pkgs, ... }: +{ pkgs, self', ... }: let grant-role = pkgs.writeShellApplication { name = "grant-role"; runtimeInputs = [ pkgs.postgresql ]; - text = builtins.readFile ./scripts/grant-role.sh; + text = builtins.readFile ../../scripts/grant-role.sh; }; create-basic-user = pkgs.writeShellApplication { @@ -12,13 +12,13 @@ let pkgs.postgresql pkgs.apacheHttpd ]; - text = builtins.readFile ./scripts/create-basic-user.sh; + text = builtins.readFile ../../scripts/create-basic-user.sh; }; flyway-packit = pkgs.runCommand "flyway-packit" { nativeBuildInputs = [ pkgs.makeWrapper ]; } '' mkdir -p $out/bin makeWrapper ${pkgs.flyway}/bin/flyway $out/bin/flyway-packit \ - --add-flags -locations=filesystem:${pkgs.packit-api.src}/api/app/src/main/resources/db/migration + --add-flags -locations=filesystem:${self'.packages.packit-api.src}/api/app/src/main/resources/db/migration ''; in { diff --git a/machines/default.nix b/machines/default.nix new file mode 100644 index 0000000..d359a72 --- /dev/null +++ b/machines/default.nix @@ -0,0 +1,25 @@ +{ withSystem, inputs, self, ... }: +let inherit (inputs.nixpkgs.lib) nixosSystem; in +{ + flake.nixosConfigurations = { + wpia-packit = withSystem "x86_64-linux" ({ pkgs, system, self', ... }: nixosSystem { + specialArgs = { inherit inputs self'; }; + modules = [ + ./wpia-packit.nix + ./common/hardware-configuration.nix + ./common/disk-config.nix + ]; + inherit system pkgs; + }); + + wpia-packit-private = withSystem "x86_64-linux" ({ pkgs, system, self', ... }: nixosSystem { + specialArgs = { inherit inputs self'; }; + modules = [ + ./wpia-packit-private.nix + ./common/hardware-configuration.nix + ./common/disk-config.nix + ]; + inherit system pkgs; + }); + }; +} diff --git a/machines/wpia-packit.nix b/machines/wpia-packit.nix index a5265b2..946390d 100644 --- a/machines/wpia-packit.nix +++ b/machines/wpia-packit.nix @@ -1,4 +1,5 @@ -{ pkgs, config, lib, ... }: { +{ pkgs, config, lib, self, ... }: +{ imports = [ ./common/configuration.nix ]; diff --git a/modules/multi-packit.nix b/modules/multi-packit.nix index 965be12..dfdf86b 100644 --- a/modules/multi-packit.nix +++ b/modules/multi-packit.nix @@ -1,4 +1,4 @@ -{ pkgs, config, lib, ... }: +{ self', pkgs, config, lib, ... }: let inherit (lib) types; @@ -125,7 +125,7 @@ in }; "~ ^/${name}(?/.*)$" = { - root = pkgs.packit-app.override { + root = self'.packages.packit-app.override { PUBLIC_URL = "/${name}"; PACKIT_NAMESPACE = name; }; diff --git a/modules/outpack.nix b/modules/outpack.nix index 115d6dd..b512d1e 100644 --- a/modules/outpack.nix +++ b/modules/outpack.nix @@ -1,4 +1,4 @@ -{ pkgs, lib, config, ... }: +{ pkgs, self', lib, config, ... }: let inherit (lib) types; instanceModule = { name, ... }: { @@ -33,11 +33,12 @@ in wantedBy = [ "multi-user.target" ]; preStart = '' if [[ ! -d ${instanceCfg.path} ]]; then - ${pkgs.outpack_server}/bin/outpack init --require-complete-tree --use-file-store ${instanceCfg.path} + printf >&2 "Initializing outpack root at %s" "${instanceCfg.path}" + ${self'.packages.outpack_server}/bin/outpack init --require-complete-tree --use-file-store ${instanceCfg.path} fi ''; serviceConfig = { - ExecStart = "${pkgs.outpack_server}/bin/outpack start-server --root ${instanceCfg.path} --listen 127.0.0.1:${toString instanceCfg.port}"; + ExecStart = "${self'.packages.outpack_server}/bin/outpack start-server --root ${instanceCfg.path} --listen 127.0.0.1:${toString instanceCfg.port}"; }; }; }); diff --git a/modules/packit-api.nix b/modules/packit-api.nix index efa0cbe..bebf367 100644 --- a/modules/packit-api.nix +++ b/modules/packit-api.nix @@ -1,4 +1,4 @@ -{ pkgs, config, lib, ... }: +{ pkgs, self', config, lib, ... }: let inherit (lib) types; @@ -165,13 +165,19 @@ in wantedBy = [ "multi-user.target" ]; wants = [ "postgresql.service" ]; after = [ "postgresql.service" ]; + serviceConfig = { Type = "simple"; DynamicUser = true; ProtectSystem = true; + + # 143 exit code is standard Spring boot behaviour it seems. + # https://docs.spring.io/spring-boot/3.3.0/how-to/deployment/installing.html#howto.deployment.installing.system-d + SuccessExitStatus = "143"; + ExecStart = let arguments = lib.mapAttrsToList (k: v: "--${k}=${v}") (flattenProperties instanceCfg.properties); in - lib.escapeShellArgs ([ "${pkgs.packit-api}/bin/packit-api" ] ++ arguments); + lib.escapeShellArgs ([ "${self'.packages.packit-api}/bin/packit-api" ] ++ arguments); EnvironmentFile = instanceCfg.environmentFiles; }; diff --git a/modules/vault.nix b/modules/vault.nix index 06f4a6f..c09e86c 100644 --- a/modules/vault.nix +++ b/modules/vault.nix @@ -1,4 +1,4 @@ -{ pkgs, lib, config, ... }: +{ pkgs, self', lib, config, ... }: let secretModule = { options = { @@ -73,7 +73,7 @@ in meta.mainProgram = "fetch-secrets"; } '' mkdir -p $out/bin - makeWrapper ${lib.getExe pkgs.fetch-secrets} $out/bin/fetch-secrets \ + makeWrapper ${lib.getExe self'.packages.fetch-secrets} $out/bin/fetch-secrets \ --add-flags --url --add-flags "${config.vault.url}" \ --add-flags --spec --add-flags "${config.vault.spec}" ''; diff --git a/packages/default.nix b/packages/default.nix new file mode 100644 index 0000000..fa4343c --- /dev/null +++ b/packages/default.nix @@ -0,0 +1,12 @@ +{ + perSystem = { pkgs, system, self', ... }: { + packages = { + outpack_server = pkgs.callPackage ./outpack_server { }; + packit-app = pkgs.callPackage ./packit/packit-app.nix { }; + packit-api = pkgs.callPackage ./packit/packit-api.nix { }; + packit = pkgs.callPackage ./packit { + inherit (self'.packages) packit-app packit-api; + }; + }; + }; +} diff --git a/PROVISIONING.md b/playbooks/new-machine-provisioning.md similarity index 80% rename from PROVISIONING.md rename to playbooks/new-machine-provisioning.md index b7725fc..907e789 100644 --- a/PROVISIONING.md +++ b/playbooks/new-machine-provisioning.md @@ -3,8 +3,10 @@ This is only needed for the initial deployment. Subsequent modifications to the configuration can be deployed more easily. -This assumes that a Linux (eg. Ubuntu) machine is has been provisioned running -already, and that you have SSH access to that machine with sudo priviledges. +This assumes that a Linux (eg. Ubuntu) machine is running already, and that you +have SSH access to that machine with sudo priviledges. These steps will destroy +everything that existed on the machine, so make sure it wasn't being used for +anything important. The deployment works by downloading a minimal NixOS installer image onto the machine, using kexec to switch control from the currently running system into @@ -17,7 +19,7 @@ process is automated thanks to the [nixos-anywhere][nixos-anywhere] tool. The instructions below use `` and `` as placeholders. `` should be the target machine's short hostname, eg. `wpia-packit`, whereas `` is the full DNS name for the machine, eg. -`packit.dide.ic.ac.uk. +`packit.dide.ic.ac.uk`. ## Hardware configuration @@ -102,17 +104,24 @@ ssh-keygen -R ssh root@ reboot ``` -# Creating a new OAuth token +# Creating a new OAuth Application -1. Go to https://github.com/settings/developers -2. homepage: `https://packit.dide.ic.ac.uk` - callback URL: `https://packit.dide.ic.ac.uk` +Packit needs a GitHub OAuth application to function. Each application can only +be used with a single domain name. If deploying Packit to a new domain, a new +application will need to be created for it. + +1. Go to https://github.com/organizations/reside-ic/settings/applications and click "New OAuth app". +2. Fill in the form, using `https://` (eg. + `https://packit.dide.ic.ac.uk`) as both the "Homepage URL" and the + "Authentication callback URL". Tick the "Enable Device Flow" box. 3. Click "Register Application" 4. Copy the Client ID 5. Generate and copy a client secret 6. Click "Update Application" -7. Store the secrets in Vault at `secret/packit/oauth/production`, with fields +7. Store the two values in the Vault at a suitable location, with fields `clientId` and `clientSecret`. +8. Make sure the application is owned by `reside-ic`, not your own personal + account. If needed, transfer ownership of it to `reside-ic`. # Pulling secrets onto the server diff --git a/playbooks/new-packit-instance.md b/playbooks/new-packit-instance.md new file mode 100644 index 0000000..ea2085d --- /dev/null +++ b/playbooks/new-packit-instance.md @@ -0,0 +1,53 @@ +# Adding a new Packit instance + +These instructions will deploy a new Packit instance to an existing machine. +The instructions assume you already have a machine setup and an associated +GitHub application. Assuming the machine already has one instance deployed, its +GitHub application can be reused for the new instance. See +[the new machine playbook](new-machine-provisioning.md) for details on creating +a GitHub application. + +## Configuration and deployment + +These instructions use `` as a placeholder for the name of the +machine that will be hosting the instance and `` as a placeholder for +the name of the instance. The instance name should only use alphanumerical +characters and `-`. It should be unique within its machine, and preferably +globally unique among all machines to avoid confusion. + +Edit the relevant `machines/.nix` file and add a new entry to the +`services.multi-packit.instances` list. + +You will need to customize the instance by configuring some of the +`services.packit-api.` options. At the minimum, you will want to +set `authentication.method` to `"github"` and set `authentication.github.org`. +All members of this organisation will be allowed to access the instance using +their GitHub account. + +Additionally, you may set a `authentication.github.team` value to restrict +access to just one team within that organisation. + +Deploy the modified configuration to the server as usual. + +## Authorizing the Github application + +The OAuth application on Github manually needs to be granted permission, by one +of the organisation's admins, to access the organisation. See +[the GitHub documentation][github-oauth-org]. Because we use a single OAuth app +for all instances hosted on the same machine, this only needs to be done once +per org and domain, even if it is used by multiple instances. + +[github-oauth-org]: https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-your-membership-in-organizations/requesting-organization-approval-for-oauth-apps + +## Creating an initial admin + +When the server is first deployed, no user has administrator priviledges. You +must manually grant add you GitHub account to the ADMIN role: + +1. Log in to the instance with your GitHub account. +1. SSH onto the server. +1. Run `grant-role ADMIN` where `` is your + GitHub username. +1. Log out and back in for the changes to take effect. + +Afterwards permissions may be managed through the web UI. diff --git a/playbooks/wipe-packit-instance.md b/playbooks/wipe-packit-instance.md new file mode 100644 index 0000000..fe5ee49 --- /dev/null +++ b/playbooks/wipe-packit-instance.md @@ -0,0 +1,60 @@ +# Wiping a Packit instance + +These steps remove all packets and associated data from a Packit instance. +This is a destructive operation with no way of rolling it back. It removes all +data from the instance indiscriminately. There currently isn't any way to +remove a single packet. Make sure you understand the consequences. + +These steps will preserve auxiliary data held in Packit, including users, +roles and permissions. + +There are two places where data needs to be deleted: we need to remove the data +from outpack_server and then clear the cache from Packit's database. + +In the following instructions, `` refers to the hostname of the machine +hosting the instance (eg. `packit.dide.ic.ac.uk`) and `` refers to +the name of the instance (eg. `training`). + +Start by connecting to the machine hosting the server using SSH. All subsequent +commands are to be run on the remote host. +```sh +ssh root@ +``` + +The outpack and packit-api services need to be shutdown before proceeding: +```sh +systemctl stop outpack- packit-api- +``` + +Make sure the services are stopped as expected. The command should print +`Active: inactive (dead)` for both services. +```sh +systemctl status outpack- packit-api- +``` + +Delete the outpack root folder: +```sh +rm -r /var/lib/outpack/ +``` + +Open an SQL session and delete all entries from the `packet` and `packet_group` +tables. Make sure you exit the session afterwards to return to the bash shell. +```sh +psql +=# DELETE FROM "packet"; +=# DELETE FROM "packet_group"; +=# exit +``` + +Start outpack server and Packit again. The outpack service unit will re-create +its root folder on start, and the Packit API server will populate the database +as new packets get pushed to the instance. +```sh +systemctl start outpack- packit-api- +``` + +Check the service logs to make sure everything is up and running again. The +`-r` flag shows logs in reverse order, ie. most recent entries first. +```sh +journalctl -u outpack- -u packit-api- -r +``` diff --git a/scripts/default.nix b/scripts/default.nix new file mode 100644 index 0000000..6246f2e --- /dev/null +++ b/scripts/default.nix @@ -0,0 +1,39 @@ +{ self, ... }: +{ + perSystem = { pkgs, lib, ... }: { + packages = { + fetch-secrets = pkgs.writers.writePython3Bin "fetch-secrets" + { + libraries = [ pkgs.python3.pkgs.hvac ]; + } ./fetch-secrets.py; + }; + + apps = { + update.program = pkgs.writers.writePython3Bin "update" { } ./update.py; + + update-ssh-keys.program = pkgs.writeShellApplication { + name = "update-ssh-keys"; + runtimeInputs = [ pkgs.curl ]; + text = builtins.readFile ./update-ssh-keys.sh; + }; + + deploy.program = pkgs.writeShellApplication { + name = "deploy"; + runtimeInputs = [ pkgs.nix pkgs.openssh ]; + text = builtins.readFile ./deploy.sh; + }; + + start-vm.program = pkgs.writeShellApplication { + name = "start-vm"; + runtimeInputs = [ pkgs.vault-bin ]; + text = builtins.readFile ./start-vm.sh; + }; + + diff.program = pkgs.writeShellApplication { + name = "diff"; + runtimeInputs = [ pkgs.nix pkgs.openssh pkgs.nix-diff ]; + text = builtins.readFile ./diff.sh; + }; + }; + }; +} diff --git a/scripts/deploy.sh b/scripts/deploy.sh new file mode 100755 index 0000000..eaf59a4 --- /dev/null +++ b/scripts/deploy.sh @@ -0,0 +1,68 @@ +# This a rough equivalent of the standard nixos-rebuild command. +# +# The main difference is that in addition to pushing the built system, it also +# pushes its derivation. Having the derivation allows us to subsequently +# inspect the build and perform diffs against it. +set -eu + +if [[ $# != 1 && $# != 2 ]]; then + hostnames=$(nix eval --quiet --raw ".#nixosConfigurations" --apply 'x: builtins.concatStringsSep ", " (builtins.attrNames x)') + cat >&2 <&2 <&2 <