Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic gitlab runner registration #137

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Dynamic gitlab runner registration
enggnr committed Dec 29, 2023
commit ac3af65b4a8526d08a0817f988e530da6a6d3d3a
2 changes: 2 additions & 0 deletions home/.chezmoi.yaml.tmpl
Original file line number Diff line number Diff line change
@@ -135,10 +135,12 @@ data:
domain: "{{ $domain }}"
gitlabRunners:
- glurl: "https://gitlab.com/"
groupID: 00000000
runnerDescription: "Docker executor"
runnerImage: "alpine:latest"
runnerTags: "alpine,linux"
- glurl: "https://gitlab.com/"
groupID: 00000000
baseVM: "debian"
runnerDescription: "VirtualBox executor - Debian, OpenJDK 20"
runnerTags: "bash,openjdk20,linux"
126 changes: 64 additions & 62 deletions software.yml
Original file line number Diff line number Diff line change
@@ -3119,7 +3119,7 @@ softwarePackages:
# @brief Logs into DockerHub for Docker Desktop
# @description
# This script logs into DockerHub so that Docker Desktop is pre-authenticated. This
# functionality requires that the `DOCKERHUB_USER` be passed in as an environment variable (or
# functionality requires that the `DOCKERHUB_USER` be passed in as an environment variable (or
# directly editted in the `~/.config/chezmoi/chezmoi.yaml` file) and that the `DOCKERHUB_TOKEN`
# be passed in as a secret (either via the encrypted secret method or passed in as an environment
# variable).
@@ -5109,9 +5109,9 @@ softwarePackages:
# @file GitLab Runner Configuration
# @brief Registers GitLab Runner(s) with the given GitLab instance
# @description
# This script registers the runner(s) with the given GitLab instance. SaaS GitLab can also be provided as the GitLab instance to register
# the runners with. The script configures the runners to use either Docker or VirtualBox Executor. Refer to
# [this page](https://docs.gitlab.com/runner/executors/docker.html) and [this page](https://docs.gitlab.com/runner/executors/virtualbox.html)
# This script registers the runner(s) with the given GitLab instance scoped to the Group `.host.gitlabRunners.[].groupID` input in the
# `.chezmoi.yaml.tmpl` file. [SaaS GitLab](https://gitlab.com) can also be provided as the GitLab instance. The script configures the runners to use
# either Docker or VirtualBox Executor. Refer to [this page](https://docs.gitlab.com/runner/executors/docker.html) and [this page](https://docs.gitlab.com/runner/executors/virtualbox.html)
# for details about the available configuration settings.
#
# Runners are always tagged with these 2 values: `hostname` and `docker`/`virtualbox` depending on the type of executor. If a list of tags is provided,
@@ -5124,11 +5124,11 @@ softwarePackages:
#
# The following chart details the secret(s) that are needed to configure the runner:
#
# | Secret | Description |
# |------------------------|------------------------------------------------------------|
# | `GITLAB_RUNNER_TOKEN` | The token generated when the runner was created in GitLab |
# | Secret | Description |
# |------------------------|--------------------------------------------------------------------------------|
# | `GITLAB_ACCESS_TOKEN` | (Personal) Access Token with `create_runner` access to gitlab.com or the group |
#
# For more information about storing secrets like SSH keys and API keys, refer to our Secrets documentation provided below
# For more information about storing secrets like SSH keys and API keys, refer to our Secrets documentation provided below.
#
# ## Configuration Variables
#
@@ -5137,6 +5137,7 @@ softwarePackages:
# | Variable | Description |
# |---------------------|----------------------------------------------------------------------------------------------------------|
# | `glurl` | The URL of the Gitlab instance to associate the Runner with |
# | `groupID` | The ID of Group in the Gitlab instance to associate the Runner with |
# | `runnerImage` | Docker image to use to configure the runner. Needed only when configuring `Docker` executor |
# | `runnerDescription` | Description of this runner |
# | `runnerTags` | Comma separated list of tags for this runner. See details in the description for more info |
@@ -5146,6 +5147,9 @@ softwarePackages:
#
# * [Secrets / Environment variables documentation](https://install.doctor/docs/customization/secrets)

HAS_DOCKER=0
HAS_VIRTUALBOX=0

### Check if Docker is installed and operational so Docker executor(s) can be registered
if command -v docker > /dev/null && docker run --rm hello-world > /dev/null; then
HAS_DOCKER=1
@@ -5167,55 +5171,53 @@ softwarePackages:
### Run logic if gitlab-runner is installed
if command -v gitlab-runner > /dev/null; then
### Populate appropriate token
case "$OSTYPE" in
solaris*) echo "TODO" ;;
darwin*) GITLAB_RUNNER_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_RUNNER_TOKEN_DARWIN")) }}{{ includeTemplate "secrets/GITLAB_RUNNER_TOKEN_DARWIN" | decrypt | trim }}{{ else }}{{ env "GITLAB_RUNNER_TOKEN_DARWIN" }}{{ end }}" ;;
linux*) GITLAB_RUNNER_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_RUNNER_TOKEN_LINUX")) }}{{ includeTemplate "secrets/GITLAB_RUNNER_TOKEN_LINUX" | decrypt | trim }}{{ else }}{{ env "GITLAB_RUNNER_TOKEN_LINUX" }}{{ end }}" ;;
bsd*) echo "TODO" ;;
msys*) GITLAB_RUNNER_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_RUNNER_TOKEN_WINDOWS")) }}{{ includeTemplate "secrets/GITLAB_RUNNER_TOKEN_WINDOWS" | decrypt | trim }}{{ else }}{{ env "GITLAB_RUNNER_TOKEN_WINDOWS" }}{{ end }}" ;;
cygwin*) GITLAB_RUNNER_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_RUNNER_TOKEN_WINDOWS")) }}{{ includeTemplate "secrets/GITLAB_RUNNER_TOKEN_WINDOWS" | decrypt | trim }}{{ else }}{{ env "GITLAB_RUNNER_TOKEN_WINDOWS" }}{{ end }}" ;;
*) echo "unknown: $OSTYPE" ;;
esac

### Check if Runner Token value is present before attempting to register runner(s)
if [ $GITLAB_RUNNER_TOKEN != "" ]; then
### Registering runners
{{ $cmd := `gitlab-runner register \
--non-interactive \
--token $GITLAB_RUNNER_TOKEN \` }}
### Register Docker based runners if Docker is installed and operational
if [ $HAS_DOCKER -eq 1 ]; then
logg info 'Registering GitLab Runner(s) that use Docker executor'
GITLAB_TOKEN="{{ if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "GITLAB_TOKEN")) }}{{ includeTemplate "secrets/GITLAB_TOKEN" | decrypt | trim }}{{ else }}{{ env "GITLAB_TOKEN" }}{{ end }}"

### Check if Gitlab Access Token value is present before attempting to register runner(s)
if [ -n "$GITLAB_TOKEN" ]; then
if command -v jq > /dev/null; then
{{- range .host.gitlabRunners }}
{{- if .runnerImage }}
{{- $cmd }}
--url {{ .glurl }} \
--executor "docker" \
--description "{{ .runnerDescription }} - on {{ .chezmoi.hostname }}" \
--docker-image {{ .runnerImage }} \
{{ if and .runnerTags (gt (len .runnerTags) 0) }}--tag-list "{{ .runnerTags }},{{ .chezmoi.hostname }},docker"
{{ else }}--tag-list "{{ .chezmoi.hostname }},docker" --run-untagged{{ end }} || echo 'Runner registration failed"
{{ end -}}
{{- if or (index . "runnerImage") (index . "baseVM") }}
{{ $cmd := `sudo gitlab-runner register \
--non-interactive \` }}
{{- if index . "runnerImage" }}
### Acquire token
logg info 'Acquiring runner token'
GITLAB_RUNNER_TOKEN=$(curl -sSL -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" -d runner_type=group_type -d group_id={{ .groupID }} \
-d description="{{ .runnerDescription }} - on {{ $.chezmoi.hostname }}" \
{{ if and .runnerTags (gt (len .runnerTags) 0) }}-d tag_list="{{ .runnerTags }},{{ $.chezmoi.hostname }},docker"
{{- else }}-d tag_list="{{ $.chezmoi.hostname }},docker" -d run_untagged=true{{ end }} \
{{ .glurl }}api/v4/user/runners | cat | jq -r .token)
### Register Docker based runners if Docker is installed and operational
logg info 'Registering GitLab Runner(s) that use Docker executor'
{{ $cmd }}
--url "{{ .glurl }}" \
--token $GITLAB_RUNNER_TOKEN \
--executor "docker" \
--docker-image "{{ .runnerImage }}" || echo 'Runner registration failed'
{{- else if index . "baseVM" }}
### Acquire token
logg info 'Acquiring runner token'
GITLAB_RUNNER_TOKEN=$(curl -sSL -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" -d runner_type=group_type -d group_id={{ .groupID }} \
-d description="{{ .runnerDescription }} - on {{ $.chezmoi.hostname }}" \
{{ if and .runnerTags (gt (len .runnerTags) 0) }}-d tag_list="{{ .runnerTags }},{{ $.chezmoi.hostname }},virtualbox"
{{- else }}-d tag_list="{{ $.chezmoi.hostname }},virtualbox" -d run_untagged=true{{ end }} \
{{ .glurl }}api/v4/user/runners | cat | jq -r .token)
### Register VirtualBox based runners if VirtualBox is installed
logg info 'Registering GitLab Runner(s) that use VirtualBox executor'
{{ $cmd }}
--url "{{ .glurl }}" \
--token $GITLAB_RUNNER_TOKEN \
--executor "virtualbox" --ssh-user "{{ $.chezmoi.username }}" --ssh-identity-file "/home/{{ $.chezmoi.username }}/.ssh/id_rsa" \
--virtualbox-base-name "{{ .baseVM }}" || echo 'Runner registration failed'
{{ end }}
{{ end }}
fi

### Register VirtualBox based runners if VirtualBox is installed
if [ $HAS_VIRTUALBOX -eq 1 ]; then
logg info 'Registering GitLab Runner(s) that use VirtualBox executor'
{{- range .host.gitlabRunners }}
{{- if .baseVM }}
{{- $cmd }}
--url {{ .glurl }} \
--executor "virtualbox" \
--description "{{ .runnerDescription }} - on {{ .chezmoi.hostname }}" \
--virtualbox-base-name "{{ .baseVM }}" \
{{ if and .runnerTags (gt (len .runnerTags) 0) }}--tag-list "{{ .runnerTags }},{{ .chezmoi.hostname }},virtualbox"
{{ else }}--tag-list "{{ .chezmoi.hostname }},virtualbox" --run-untagged{{ end }} || echo 'Runner registration failed"
{{ end -}}
{{ end }}
else
logg warn 'jq is required by the Gitlab runner configuration script'
fi
else
logg warn 'GITLAB_RUNNER_TOKEN is not set. Not registering runner(s)'
logg warn 'GITLAB_TOKEN is not set. Not registering runner(s)'
fi
else
logg warn 'gitlab-runner is not installed or is not available in PATH'
@@ -5272,12 +5274,12 @@ softwarePackages:
if command -v gitomatic > /dev/null; then
### Copy bin to /usr/local/bin
logg info "Copying $HOME/.local/bin/gitomatic-service to /usr/local/bin/gitomatic-service" && sudo cp -f "$HOME/.local/bin/gitomatic-service" /usr/local/bin/gitomatic-service

### Copy gitomatic to global directory
if [ ! -f /usr/local/bin/gitomatic ]; then
logg info 'Copying gitomatic executable to /usr/local/bin/gitomatic' && sudo cp -f "$(which gitomatic)" /usr/local/bin/gitomatic
fi

if [ -d /Applications ] && [ -d /System ]; then
### macOS
logg info 'Copying gitomatic plist file to /Library/LaunchDaemons' && sudo cp -f "${XDG_CONFIG_HOME:-$HOME/.config}/gitomatic/com.github.muesli.gitomatic.plist" /Library/LaunchDaemons/com.github.muesli.gitomatic.plist
@@ -7817,7 +7819,7 @@ softwarePackages:

### netdata-claim.sh must be run as netdata user
sudo -H -u netdata bash -c 'export NETDATA_ROOM="{{- if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "NETDATA_ROOM")) -}}{{- includeTemplate "secrets/NETDATA_ROOM" | decrypt | trim -}}{{- else -}}{{- env "NETDATA_ROOM" -}}{{- end -}}" && export NETDATA_TOKEN="{{- if (stat (joinPath .chezmoi.sourceDir ".chezmoitemplates" "secrets" "NETDATA_TOKEN")) -}}{{- includeTemplate "secrets/NETDATA_TOKEN" | decrypt | trim -}}{{- else -}}{{- env "NETDATA_TOKEN" -}}{{- end -}}" && yes | netdata-claim.sh -token="$NETDATA_TOKEN" -rooms="$NETDATA_ROOM" -url="https://app.netdata.cloud"'

### Kernel optimizations
# These are mentioned while installing via the kickstart.sh script method. We are using Homebrew for the installation though.
# Assuming these optimizations do not cause any harm.
@@ -7849,7 +7851,7 @@ softwarePackages:
logg error 'No etc location found for netdata' && exit 1
fi
logg info "Copying ${XDG_CONFIG_HOME:-$HOME/.config}/netdata/health.d/ to $NETDATA_ETC" && sudo cp -rf "${XDG_CONFIG_HOME:-$HOME/.config}/netdata/health.d/" "$NETDATA_ETC"

if command -v gsed > /dev/null; then
SED_UTIL="gsed"
else
@@ -7859,7 +7861,7 @@ softwarePackages:
### Blocky
logg info "Adding Blocky metrics collection to $NETDATA_ETC/go.d/prometheus.conf"
sudo "$SED_UTIL" -i "/jobs:/a\ - name: blocky_local \n url: 'http://127.0.0.1:4000/metrics'" "$NETDATA_ETC/go.d/prometheus.conf"

### SFTPGo
logg info "Adding SFTPGo metrics collection to $NETDATA_ETC/go.d/prometheus.conf"
sudo "$SED_UTIL" -i "/jobs:/a\ - name: sftpgo_local \n url: 'http://127.0.0.1:57500/metrics'" "$NETDATA_ETC/go.d/prometheus.conf"
@@ -10798,7 +10800,7 @@ softwarePackages:
_home: https://github.com/skeeto/endlessh
_name: Endlessh
_service: endlessh
_post: |
_post: |
#!/usr/bin/env bash
# @file Endlessh Configuration
# @brief Applies the Endlessh configuration and starts the service on Linux systems
@@ -11638,7 +11640,7 @@ softwarePackages:
yay: termius
dagu:
_bin: dagu
_github:
_github:
_name: Dagu
_post: |
#!/usr/bin/env bash
@@ -12018,7 +12020,7 @@ softwarePackages:
# ## Links
#
# * [Privoxy configuration](https://github.com/megabyte-labs/install.doctor/tree/master/home/dot_config/privoxy/config)

### Define the Privoxy configuration location based on whether system is macOS or Linux
if [ -d /Applications ] && [ -d /System ]; then
### macOS
@@ -12044,7 +12046,7 @@ softwarePackages:
sudo add-usergroup "$USER" privoxy
fi
sudo chown privoxy:privoxy "$PRIVOXY_CONFIG" 2> /dev/null || sudo chown privoxy:$(id -g -n) "$PRIVOXY_CONFIG"

### Restart Privoxy after configuration is applied
if [ -d /Applications ] && [ -d /System ]; then
### macOS
@@ -13583,7 +13585,7 @@ softwarePackages:
# ## Links
#
# * [`fail2ban` configuration folder](https://github.com/megabyte-labs/install.doctor/tree/master/home/private_dot_ssh/fail2ban)

### Notify of script start
logg info 'Configuring fail2ban'