Crucible-CI is a Continuous Integration (CI) testing harness targeted at the Crucible performance automation and analysis framework. Crucible is comprised of many subprojects with each subproject having it's own repository. As such, the primary goal of Crucible-CI is to be able to perform integration testing of a given subproject when changes to it are proposed via a pull request.
What this means is that when a pull request is submitted for a project that is a member of the Crucible family a working Crucible installation is created using the primary branch of all the other repositories and the proposed code for the specific repository that the pull request is for. This installation is then used to perform integration testing of the pull request's changes combined with all of the other project's current upstream code.
The intent/goal here is that the upstream code in the primary branch for all repositories in the Crucible family of projects is ready for user delivery and consumption at all times. This is a very ambitious goal but given the distribution model that Crucible uses, which is distribution of code updates via Git clone/pull from the upstream repositories, we believe it is the most appropriate target.
Crucible-CI is primarily intended to be executed in the GitHub runner environment (both GitHub hosted and self-hosted runners). However, some of its components can be executed outside of a GitHub runner environment in order to provide developers with the ability to integration test their changes prior to pull request submission. More on this in the details below.
Crucible-CI is essentially a collection of GitHub Actions and reusable GitHub workflows.
Each action in Crucible-CI is unique and can be consumed in one or more GitHub Action workflows. Here is the current list of provided actions:
- Helper Actions
- Primary Actions
The actions are, for the most part, implemented as shell scripts that are provided to GitHub's workflow framework via action.yml. The way in which they are aggregated together is very GitHub specific, but since they are simple shell scripts at their core we can potentially re-use them outside of the GitHub environment. Where applicable/useful that is described below.
These actions are used to provide information that can be used in the workflows in a variety of different ways. These are not performing any testing of Crucible as it is meant to be used by an end user.
The build-controller action is used to build a Crucible controller image for use in one or more jobs.
The check-controller-build action is used to determine if changes to the main Crucible repository require the controller image to be rebuilt.
The clean-environment action is used to cleanup the runner environment after a run. This really isn't necessary for GitHub hosted runners (since they are ephemeral) but self-hosted runners are not and must have their environments cleaned to avoid influencing future jobs.
NOTE: This action is DEPRECATED in favor of get-scenarios
The get-benchmarks action is used to determine which benchmarks can be run in the specified runner type. This is necessary because some benchmarks have requirements that cannot be met by a certain type of runner. For example, GitHub hosted runners do not provide a CPU isolation environment which some benchmarks require. Additionally, not all Crucible supported benchmarks are currently supported by Crucible-CI (usually due to special hardware requirements that cannot be met in a runner environment due to cost/complexity).
NOTE: This action is DEPRECATED in favor of get-scenarios
The get-endpoints action is used to determine which endpoints can be run in the specified runner type. Some runner environments (such as self hosted) are specifically targeting at testing specific endpoints and this action provides that information.
The get-repo-name action is used to extract a repository name from the github.repository context variable (which includes the owner/project name).
The get-scenarios action is used to determine what "scenarios" should be executed as part of a specific job (where a scenario is a combination of an endpoint and a benchmark). The parameters are combined because they cannot be adequately evaluated in isolation with respect to the different runner environments (GitHub hosted vs. self hosted).
The get-userenvs action is used to look at a rickshaw repository to determine what the list of currently supported userenvs is.
The install-crucible action provides a reusable Crucible installation tool that can be configured to properly install upstream code for repositories not being tested while properly using the pull request code for the subproject being tested.
These actions are where actual Crucible testing is performed.
The integration-tests action is where, as the name implies, the integration testing is performed. This action has several different possible inputs, but all of them are optional. By default it will run a very simple integration test based on the default values. By specifying one or more of the available inputs the action can be "tuned" for the type of testing that is desired in the calling workflow.
The integration test is comprised of multiple shell scripts that perform different actions. Some of these scripts are usually specific to the GitHub runner environment, such as setup-ci-environment (since it performs a custon install of Crucible), while others can be used outside of the Github runner environment to perform testing in a development environment with an existing running Crucible installation. An example of this is the use of run-ci-stage1 (which is the actual "test" in this action, the other scripts are primarily for setup and reporting) by the Crucible run-ci command.
There are two different kinds of workflows present in Crucible-CI. They are:
- Reusable Workflows
- Runnable Workflows
These workflows can be called from any Crucible subproject to easily provide a detailed CI test based on provided input.
The benchmark-crucible-ci workflow is intended to be called from Crucible benchmark subprojects. It limits its testing focus to benchmark specific jobs.
The core-crucible-ci workflow is intended to be called by any of the core Crucible subprojects. Some of the core subprojects required more detailed testing (such as controller building under certain conditions) which is implemented in this workflow.
These workflows are what run as part of pull requests that are submitted against the Crucible-CI subproject itself.
The faux-crucible-ci workflow is a "fake" workflow that is run when the pull request in question does not actually affect runtime behavior (such as documentation only changes). It provides "null" jobs that satisfy Crucible-CI's GitHub branch protection rules.
The run-core-crucible-ci workflow is used to execute the core-crucible-ci reusable workflow with Crucible-CI being the target core subproject. This workflow is an excellent example to base a workflow off of for use in another core subproject.
The test-benchmark-crucible-ci workflow is used to test the benchmark-crucible-ci reusable workflow using different benchmarks.
The test-core-crucible-ci workflow is used to test the core-crucible-ci reusable workflow using different core subprojects.
Crucible-CI is a subproject in the Crucible family just like any other. This means that Crucible-CI can be used to test itself when changes to it are proposed. As mentioned above, it also means that Crucible-CI has an embedded workflow implemented that demonstrates how its reusable workflows can be called.