forked from NixOS/hydra
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge remote-tracking branch 'upstream/master' into relationalai-2023…
…1201
- Loading branch information
Showing
87 changed files
with
2,389 additions
and
1,032 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| This is a rough overview from informal discussions and explanations of inner workings of Hydra. | ||
| You can use it as a guide to navigate the codebase or ask questions. | ||
|
|
||
| ## Architecture | ||
|
|
||
| ### Components | ||
|
|
||
| - Postgres database | ||
| - configuration | ||
| - build queue | ||
| - what is already built | ||
| - what is going to build | ||
| - `hydra-server` | ||
| - Perl, Catalyst | ||
| - web frontend | ||
| - `hydra-evaluator` | ||
| - Perl, C++ | ||
| - fetches repositories | ||
| - evaluates job sets | ||
| - pointers to a repository | ||
| - adds builds to the queue | ||
| - `hydra-queue-runner` | ||
| - C++ | ||
| - monitors the queue | ||
| - executes build steps | ||
| - uploads build results | ||
| - copy to a Nix store | ||
| - Nix store | ||
| - contains `.drv`s | ||
| - populated by `hydra-evaluator` | ||
| - read by `hydra-queue-runner` | ||
| - destination Nix store | ||
| - can be a binary cache | ||
| - e.g. `[cache.nixos.org](http://cache.nixos.org)` or the same store again (for small Hydra instances) | ||
| - plugin architecture | ||
| - extend evaluator for new kinds of repositories | ||
| - e.g. fetch from `git` | ||
|
|
||
| ### Database Schema | ||
|
|
||
| [https://github.com/NixOS/hydra/blob/master/src/sql/hydra.sql](https://github.com/NixOS/hydra/blob/master/src/sql/hydra.sql) | ||
|
|
||
| - `Jobsets` | ||
| - populated by calling Nix evaluator | ||
| - every Nix derivation in `release.nix` is a Job | ||
| - `flake` | ||
| - URL to flake, if job is from a flake | ||
| - single-point of configuration for flake builds | ||
| - flake itself contains pointers to dependencies | ||
| - for other builds we need more configuration data | ||
| - `JobsetInputs` | ||
| - more configuration for a Job | ||
| - `JobsetInputAlts` | ||
| - historical, where you could have more than one alternative for each input | ||
| - it would have done the cross product of all possibilities | ||
| - not used any more, as now every input is unique | ||
| - originally that was to have alternative values for the system parameter | ||
| - `x86-linux`, `x86_64-darwin` | ||
| - turned out not to be a good idea, as job set names did not uniquely identify output | ||
| - `Builds` | ||
| - queue: scheduled and finished builds | ||
| - instance of a Job | ||
| - corresponds to a top-level derivation | ||
| - can have many dependencies that don’t have a corresponding build | ||
| - dependencies represented as `BuildSteps` | ||
| - a Job is all the builds with a particular name, e.g. | ||
| - `git.x86_64-linux` is a job | ||
| - there maybe be multiple builds for that job | ||
| - build ID: just an auto-increment number | ||
| - building one thing can actually cause many (hundreds of) derivations to be built | ||
| - for queued builds, the `drv` has to be present in the store | ||
| - otherwise build will fail, e.g. after garbage collection | ||
| - `BuildSteps` | ||
| - corresponds to a derivation or substitution | ||
| - are reused through the Nix store | ||
| - may be duplicated for unique derivations due to how they relate to `Jobs` | ||
| - `BuildStepOutputs` | ||
| - corresponds directly to derivation outputs | ||
| - `out`, `dev`, ... | ||
| - `BuildProducts` | ||
| - not a Nix concept | ||
| - populated from a special file `$out/nix-support/hydra-build-producs` | ||
| - used to scrape parts of build results out to the web frontend | ||
| - e.g. manuals, ISO images, etc. | ||
| - `BuildMetrics` | ||
| - scrapes data from magic location, similar to `BuildProducts` to show fancy graphs | ||
| - e.g. test coverage, build times, CPU utilization for build | ||
| - `$out/nix-support/hydra-metrics` | ||
| - `BuildInputs` | ||
| - probably obsolute | ||
| - `JobsetEvalMembers` | ||
| - joins evaluations with jobs | ||
| - huge table, 10k’s of entries for one `nixpkgs` evaluation | ||
| - can be imagined as a subset of the eval cache | ||
| - could in principle use the eval cache | ||
|
|
||
| ### `release.nix` | ||
|
|
||
| - hydra-specific convention to describe the build | ||
| - should evaluate to an attribute set that contains derivations | ||
| - hydra considers every attribute in that set a job | ||
| - every job needs a unique name | ||
| - if you want to build for multiple platforms, you need to reflect that in the name | ||
| - hydra does a deep traversal of the attribute set | ||
| - just evaluating the names may take half an hour | ||
|
|
||
| ## FAQ | ||
|
|
||
| Can we imagine Hydra to be a persistence layer for the build graph? | ||
|
|
||
| - partially, it lacks a lot of information | ||
| - does not keep edges of the build graph | ||
|
|
||
| How does Hydra relate to `nix build`? | ||
|
|
||
| - reimplements the top level Nix build loop, scheduling, etc. | ||
| - Hydra has to persist build results | ||
| - Hydra has more sophisticated remote build execution and scheduling than Nix | ||
|
|
||
| Is it conceptually possible to unify Hydra’s capabilities with regular Nix? | ||
|
|
||
| - Nix does not have any scheduling, it just traverses the build graph | ||
| - Hydra has scheduling in terms of job set priorities, tracks how much of a job set it has worked on | ||
| - makes sure jobs don’t starve each other | ||
| - Nix cannot dynamically add build jobs at runtime | ||
| - [RFC 92](https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md) should enable that | ||
| - internally it is already possible, but there is no interface to do that | ||
| - Hydra queue runner is a long running process | ||
| - Nix takes a static set of jobs, working it off at once |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| ## The RunCommand Plugin | ||
|
|
||
| Hydra supports executing a program after certain builds finish. | ||
| This behavior is disabled by default. | ||
|
|
||
| Hydra executes these commands under the `hydra-notify` service. | ||
|
|
||
| ### Static Commands | ||
|
|
||
| Configure specific commands to execute after the specified matching job finishes. | ||
|
|
||
| #### Configuration | ||
|
|
||
| - `runcommand.[].job` | ||
|
|
||
| A matcher for jobs to match in the format `project:jobset:job`. Defaults to `*:*:*`. | ||
|
|
||
| **Note:** This matcher format is not a regular expression. | ||
| The `*` is a wildcard for that entire section, partial matches are not supported. | ||
|
|
||
| - `runcommand.[].command` | ||
|
|
||
| Command to run. Can use the `$HYDRA_JSON` environment variable to access information about the build. | ||
|
|
||
| ### Example | ||
|
|
||
| ```xml | ||
| <runcommand> | ||
| job = myProject:*:* | ||
| command = cat $HYDRA_JSON > /tmp/hydra-output | ||
| </runcommand> | ||
| ``` | ||
|
|
||
| ### Dynamic Commands | ||
|
|
||
| Hydra can optionally run RunCommand hooks defined dynamically by the jobset. In | ||
| order to enable dynamic commands, you must enable this feature in your | ||
| `hydra.conf`, *as well as* in the parent project and jobset configuration. | ||
|
|
||
| #### Behavior | ||
|
|
||
| Hydra will execute any program defined under the `runCommandHook` attribute set. These jobs must have a single output named `out`, and that output must be an executable file located directly at `$out`. | ||
|
|
||
| #### Security Properties | ||
|
|
||
| Safely deploying dynamic commands requires careful design of your Hydra jobs. Allowing arbitrary users to define attributes in your top level attribute set will allow that user to execute code on your Hydra. | ||
|
|
||
| If a jobset has dynamic commands enabled, you must ensure only trusted users can define top level attributes. | ||
|
|
||
|
|
||
| #### Configuration | ||
|
|
||
| - `dynamicruncommand.enable` | ||
|
|
||
| Set to 1 to enable dynamic RunCommand program execution. | ||
|
|
||
| #### Example | ||
|
|
||
| In your Hydra configuration, specify: | ||
|
|
||
| ```xml | ||
| <dynamicruncommand> | ||
| enable = 1 | ||
| </dynamicruncommand> | ||
| ``` | ||
|
|
||
| Then create a job named `runCommandHook.example` in your jobset: | ||
|
|
||
| ``` | ||
| { pkgs, ... }: { | ||
| runCommandHook = { | ||
| recurseForDerivations = true; | ||
| example = pkgs.writeScript "run-me" '' | ||
| #!${pkgs.runtimeShell} | ||
| ${pkgs.jq}/bin/jq . "$HYDRA_JSON" | ||
| ''; | ||
| }; | ||
| } | ||
| ``` | ||
|
|
||
| After the `runcommandHook.example` build finishes that script will execute. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.