An agent used to connect CI providers to a Docksal Sandbox Server.
Supported CI providers:
- Bitbucket Pipelines (with build status integration)
- CircleCI (with build status integration)
- GitLab
- Github Actions
This image(s) is part of the Docksal image library.
Docksal Sandboxes are continuous integration environments powered by Docksal and deployed to a Docksal Sandbox Server. They can be provisioned for any git branch and are feature-wise identical to project's local Docksal environment.
Use cases:
- automated testing (full stack)
- manual testing
- enhanced pull request review experience
- demos
Build status updates (and sandbox URLs) can be posted to Github and Bitbucket via respective build status APIs.
URLs to sandbox environments can also be published to a Slack channel.
base- basic (bash, curl, git, etc.), latest versionphp,latest- basic + php stack tools (composer, drush, drupal console, wp-cli, etc.), latest version1.2-base- basic, a specific stable version1.2-php- php, a specific stable version
edge-base- base, latest development versionedge-php, php, latest development version
The following required variables are usually configured at the organization level. This way, all project repos will have access to them. They can as well be configured at the repo level.
DOCKSAL_HOST or DOCKSAL_HOST_IP
The address of the sandbox server. Configure one or the other.
If using DOCKSAL_HOST, make sure the domain is configured as a wildcard DNS entry.
If using DOCKSAL_HOST_IP, the agent will use nip.io for dynamic wildcard domain names for sandboxes.
DOCKSAL_HOST_SSH_KEY
A base64 encoded private SSH key, used to access the sandbox server.
Note: on macOS cat /path/to/<private_key_file> | base64 can be used to create a base64 encoded string from a private SSH key, while on Linux and in WSL on Windows 10 cat /path/to/<private_key_file> | base64 -w 0 should be used to avoid output wrapping of the base64 command).
BUILD_ENVIRONMENT
Used to set the environment built against. Defaults to local.
CI_SSH_KEY
A base64 encoded private SSH key, used by default for all hosts (set as Host * in ~/.ssh/config).
This key will be used to clone/push to git, run commands over SSH on a remote deployment environment, etc.
DOCKSAL_HOST_SSH_PORT
Can be used to override the SSH port of the sandbox server where the agent connects (defaults to 22).
DOCKSAL_DOMAIN
Can be used to set the base URL for sandbox builds (defaults to DOCKSAL_HOST if not set), individually from DOCKSAL_HOST.
This is useful when working with CDNs/ELBs/WAFs/etc (when DOCKSAL_DOMAIN is different from the DOCKSAL_HOST).
DOCKSAL_HOST_USER
The user name used as the build user on the sandbox server. Defaults to build-agent. Must have SSH access to the server.
REMOTE_BUILD_BASE
The default directory location on the sandbox server where the repositories should be cloned down to and built.
Defaults to /home/build-agent/builds.
REMOTE_CODEBASE_METHOD
Pick between rsync (default) and git for the codebase initialization method on the sandbox server.
The codebase is initialized on the sandbox server by the sandbox-init (or build-init) command.
git - code is checkout on the sandbox server via git. The server must have access to checkout from the repo.
Any build settings and necessary code manipulations must happen on the sandbox server using build-exec commands.
rsync - code is rsync-ed to the sandbox server from the build agent. You can perform necessary code adjustments in the
build agent after running build-env and before running sandbox-init (or build-init), which pushes the code to the
sandbox server.
REMOTE_BUILD_DIR_CLEANUP
Whether or not the remote build directory is reset during the build. Only supported with REMOTE_CODEBASE_METHOD=git.
Defaults to 1 which wipes the remote build directory and produces a "clean build".
Set to 0 to produce "dirty builds", when file changes in the remote codebase should be preserved.
Note: Switching REMOTE_CODEBASE_METHOD mode will result in a clean build.
SANDBOX_PERMANENT
Set SANDBOX_PERMANENT=true to have a permanent sandbox provisioned.
Permanent sandboxes are exempt from scheduled garbage collection on the sandbox server. They would still hibernate after the configured period of inactivity, but won't be removed from the server after becoming dangling. See https://github.com/docksal/service-vhost-proxy#advanced-proxy-configuration for more information.
This variable is usually set at the branch level in the build settings to designate a specific (one or multiple) branch environments as permanent.
SANDBOX_DOMAIN
Sets a custom domain for a sandbox. Takes precedence over the automatic (branch name based) domain generation.
This can be used for sandbox environments which need a custom (nice) domain name.
GITHUB_TOKEN and BITBUCKET_TOKEN
Used for access to post sandbox URLs via build status API as well as comments on pull requests.
For Github, the token can be generated from the user's account.
Set access to "repo" (http://take.ms/nMqcW).
For Bitbucket, the token can be generated from the user's settings. Instructions on creating an app password.
Set access to "Repositories: Write", "Pull requests: Write" (http://take.ms/98BG5).
When storing the app password it is in the format: USER:PASSWORD.
GIT_USER_EMAIL
The user's email to perform Git operations as. Defaults to [email protected]
GIT_USER_NAME
The user's name to perform Git operations as. Defaults to Docksal CI
DOCKSAL_HOST_TUNNEL
If not empty, localhost:2374 in the agent is mapped to docker.sock on the remote DOCKSAL_HOST via a secure SSH tunnel.
The agent can then run docker commands against the remote DOCKSAL_HOST.
Other features and integrations are usually configured at the repo level. See below.
Here's the most basic configuration for Bitbucket Pipelines. Save it into bitbucket-pipelines.yml in your project repo.
image: docksal/ci-agent:base
pipelines:
default:
- step:
script:
- source build-env && sandbox-initFor a more advanced example see bitbucket-pipelines.yml.
Here's the most basic configuration for CircleCI. Save it into .circleci/config.yml in your project repo.
version: 2
jobs:
build:
working_directory: /home/agent/build
docker:
- image: docksal/ci-agent:base
steps:
- run:
name: Configure agent environment
command: echo 'source build-env' >> $BASH_ENV
- checkout
- run:
name: Build sandbox
command: sandbox-initFor a more advanced example see config.yml.
Here's the most basic configuration for GitLab. Save it into .gitlab-ci.yml in your project repo.
stages:
- sandbox
sandbox-launch:
stage: sandbox
image: docksal/ci-agent:base
script:
- export SANDBOX_DOMAIN=$CI_ENVIRONMENT_SLUG--$CI_PROJECT_NAME.$DOCKSAL_HOST
- source build-env
- sandbox-init
environment:
name: $CI_COMMIT_REF_NAME
url: https://$CI_ENVIRONMENT_SLUG--$CI_PROJECT_NAME.$DOCKSAL_HOSTFor a more advanced example see .gitlab-ci.yml.
Here's the most basic configuration for Github Actions. Save it into .github/workflows/sandbox.yml in your project repo.
You might need to add your ssh key to EC2 /home/build-agent/.ssh and the deploy(public) key to Github Actions for cloning the codebase via git.
name: Docksal sandbox
on:
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
container:
image: docksal/ci-agent:base
env:
CI_SSH_KEY: ${{ secrets.CI_SSH_KEY }}
DOCKSAL_HOST_IP: ${{ secrets.DOCKSAL_HOST_IP }}
DOCKSAL_HOST_SSH_KEY: ${{ secrets.DOCKSAL_HOST_SSH_KEY }}
GITHUB_TOKEN: ${{ github.token }}
REMOTE_CODEBASE_METHOD: git
steps:
- name: Build sandbox
shell: bash
run: |
source build-env
sandbox-initFor a more advanced example see .sandbox.yml.
For a complete list of built-in commands see base/bin.
build-env- configures build settings on the agent. Usage:source build-env(orDEBUG=1 source build-env)build-init- initializes the sandbox codebase and settings on the sandbox server. Usage:build-initbuild-exec- executes a shell command within the build directory on the sandbox server. Usage:build-init pwdbuild-notify- see "Build status notifications" docs belowsandbox-init- a convenient shortcut to provision a basic sandbox. See sandbox-init
The following variables are derived from the respective Bitbucket Pipelines, Circle CI, GitLab CI, and Github Actions build variables.
GIT_REPO_OWNER- git repo machine owner/slug nameGIT_REPO_NAME- git repo machine nameGIT_REPO_URL- git repo URLGIT_BRANCH_NAME- git branch nameGIT_COMMIT_HASH- git commit hashGIT_PR_NUMBER- git pull request / merge request numberGIT_REPO_SERVICE-github,bitbucketorgitlab(makes sense mostly for CircleCI)BUILD_ID- The unique identifier for a buildBUILD_DIR- The full path where the repository is cloned and where the job is run in the agent container
REMOTE_BUILD_DIR
The directory location on the sandbox server where current build will happen. Defaults to:
${REMOTE_BUILD_BASE}/${REPO_NAME_SAFE}-${BRANCH_NAME_SAFE}
Protect sandboxes from public access using Basic HTTP authentication.
Set the following environment variables at the repo level:
HTTP_USERHTTP_PASS
This integration allows the agent to post build status updates and sandbox URL via Github/Bitbucket build status API.
For CircleCI, it is also possible to enable posting the sandbox URL as a comment in pull requests.
GITHUB_TOKEN or BITBUCKET_TOKEN must be configured respectively (either globally or at the repo level).
build-notify <pending|success|failure>
Place the triggers right before and right after fin init call in your build script, e.g.,
build-notify pending
ssh docker-host "cd $REMOTE_BUILD_DIR && fin init" || ( build-notify failure && exit 1 )
build-notify successTo enable posting sandbox URLs in comments on pull requests, do export PR_COMMENT=1 prior to calling build-notify
This integrations allows the agent to post messages to a given Slack channel.
It can be used for notification purposes when a build is started, completed, failed, etc.
SLACK_WEBHOOK_URL
The Incoming Webhook integration URL from Slack,
e.g., SLACK_WEBHOOK_URL https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/XXxxXXXXxxXXXXxxXXXXxxXX
SLACK_CHANNEL
A public or private channel in Slack, e.g., SLACK_CHANNEL #project-name-bots
SLACK_USER
The username the message should post to Slack as. Defaults to Docksal CI
SLACK_ICON
The icon the message should use to accompany the message: Defaults to :desktop_computer:
slack 'message' ['#channel'] ['webhook_url'] ['slack_user'] ['slack_icon']Channel and webhook url can be passed via environment variables. See above.
Incoming Webhook integration won't work for private channels, which the owner of the integration does not belong to.
Build artifacts can be stored in an AWS S3 bucket.
Set the following environment variables at the organization or repo level:
AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYARTIFACTS_BUCKET_NAMEARTIFACTS_BASE_URL(optional)GITHUB_TOKEN(optional)BITBUCKET_TOKEN(optional)
To upload artifacts to the configured S3 bucket use the build-acp command.
build-acp /source/path/There is no file browsing capability available for private S3 buckets.
An index.html file is used as the directory index, however it has to be created/generated manually.
When uploading a prepared artifacts folder with the index.html file in it, add a trailing slash to the source path to
upload the contents of the source folder vs the folder itself.
You can upload additional folders/files by running the command multiple times.
The optional destination argument can be used to define a subdirectory at the destination in the bucket.
build-acp /source/path/ destination/pathBase URL
The base URL is derived from ARTIFACTS_BUCKET_NAME as follows (assuming AWS S3 us-east-1 region by default):
https://${ARTIFACTS_BUCKET_NAME}.s3.amazonaws.com
It can be overridden via the optional ARTIFACTS_BASE_URL configuration variable at the organization/repo level:
ARTIFACTS_BASE_URL = https://artifacts.example.com
Upload path
The upload path is unique for each commit and is derived as follows:
${REPO_NAME_SAFE}/${BRANCH_NAME_SAFE}-${GIT_COMMIT_HASH}In certain cases you may want to store build artifacts per branch instead of per commit.
To do this, override the ARTIFACTS_BUCKET_PATH variable before calling the build-acp command:
export ARTIFACTS_BUCKET_PATH="${REPO_NAME_SAFE}/${BRANCH_NAME_SAFE}"
build-acp my-artifacts/Posting build artifact URLs to Bitbucket
If BITBUCKET_TOKEN is set, the URL to the artifacts will be posted back to Bitbucket via
Bitbucket Build Status API.
If a bucket does not exist, it will be created automatically (with no public access). Existing bucket access permissions are not automatically adjusted. It's up to you whether you want to keep them open or not.
When artifacts are uploaded, the destination artifact folder in the bucket is set to be publicly accessible. Anyone with the direct link will be able to access the artifacts, but will not be able to browse the list of all available artifact folders in the bucket (so long as the bucket itself is set to be private).
The URL by default includes a git commit hash, which serves as an authentication token (the URL is impossible to guess). This provides a simple yet efficient level of security for artifacts.
To add an additional level of security follow this guide to set up username/password access to S3 via CloudFront and Lambda@Edge.
By combining the following configuration options you can get low overhead non-volatile environments.
SANDBOX_DOMAIN=<nice-domain>
SANDBOX_PERMANENT=true
REMOTE_CODEBASE_METHOD=git
REMOTE_BUILD_DIR_CLEANUP=0
Such environments can be used for non-critical production-ish workloads, whenever an on-demand delayed start (5-10s delay) is not a concern.
It is best security practice not to store secrets such as API keys in a code repository. Many CI systems already have the ability to set such environment variables during the build process. Any environment variables set at build time whose name starts with SECRET_ will be forwarded as-is to the built environment.
