feat: GPG commit signature verification (#2492) (#3242)

* Add initial primitives and tests for GPG related operations

* More tests and test documentation

* Move gpg primitives to own module

* Add initial primitives for running git verify-commit and tests

* Improve and better comment test

* Implement VerifyCommitSignature() primitive for metrics wrapper

* More commentary

* Make reposerver verify gpg signatures when generating manifests

* Make signature validation optional

* Forbid use of local manifests when signature verification is enabled

* Introduce new signatureKeys field in project CRD

* Initial support for only syncing against signed revisions

* Updates to GnuPG primitives and more test cases

* Move signature verification to correct place and add tests

* Add signature verification result to revision metadata and display it in UI

* Add more primitives and move out some stuff to common module

* Add more testdata

* Add key management primitives to ArgoDB

* Move type GnuPGPublicKey to appsv1 package

* Add const ArgoCDGPGKeysConfigMapName

* Handle key operations with appsv1.GnuPGPublicKey

* Add initial API for managing GPG keys

* Remove deprecated code

* Add primitives for adding public keys to configuration

* Change semantics of ValidateGPGKeys to return more key information

* Add key import functionality to public key API

* Fix code quirks reported by linter

* More code quirks fixes

* Fix test

* Add primitives for deleting keys from configuration

* Add delete key operation to API and CLI

* Cosmetics

* Implement logic to sync configuration to keyring in repo-server

* Add IsGPGEnabled() primitive and also update trustdb on ownertrust changes

* Use gpg.IsGPGEnabled() instead of custom test

* Remove all keyring manipulating methods from DB

* Cosmetics/comments

* Require grpc methods from argoproj pkg

* Enable setting config path via ARGOCD_GPG_DATA_PATH

* Allow "no" and any cases in ARGOCD_GPG_ENABLED

* Enable GPG feature on start and start-e2e and set required environment

* Cosmetics/comments

* Cosmetics and commentary

* Update API documentation

* Fix comment

* Only run GPG related operations if GPG is enabled

* Allow setting ARGOCD_GPG_ENABLE from the environment

* Create GPG ConfigMap resource during installation

* Use function instead of constant to get the watcher path

* Re-watch source path in case it gets recreated. Also, error on finish

* Add End-to-End tests for GPG commit verification

* Introduce SignatureKey type for AppProject CRD

* Fix merge error from previous commit

* Adapt test for additional manifest (argocd-gpg-keys-cm.yaml)

* Fix linter issues

* Adapt CircleCI configuration to enable running tests

* Add wrapper scripts for git and gpg

* Sigh.

* Display gpg version in CircleCI

* Install gnupg2 and link it to gpg in CI

* Try to install gnupg2 in CircleCI image

* More CircleCI tweaks

* # This is a combination of 10 commits.
# This is the 1st commit message:

Containerize tests - test cycle

# This is the commit message #2:

adapt working directory

# This is the commit message #3:

Build before running tests (so we might have a cache)

# This is the commit message #4:

Test limiting parallelism

# This is the commit message #5:

Remove unbound variable

# This is the commit message #6:

Decrease parallelism to find out limit

# This is the commit message #7:

Use correct flag

# This is the commit message #8:

Update Docker image

# This is the commit message #9:

Remove build phase and increase parallelism

# This is the commit message #10:

Further increase parallelism

* Dockerize toolchain

* Add new targets to Makefile

* Codegen

* Properly handle permissions for E2E tests

* Remove gnupg2 installation from CircleCI configuration

* Limit parallelism of build

* Fix Yarn lint

* Retrigger CI for possible flaky test

* Codegen

* Remove duplicate target in Makefile

* Pull in pager from dep ensure -v

* Adapt to gitops-engine changes and codegen

* Use new health package for health status constants

* Add GPG methods to ArgoDB mock module

* Fix possible nil pointer dereference

* Fix linter issue in imports

* Introduce RBAC resource type 'gpgkeys' and adapt policies

* Use ARGOCD_GNUPGHOME instead of GNUPGHOME for subsystem configuration

Also remove some deprecated unit tests.

* Also register GPG keys API with gRPC-GW

* Update from codegen

* Update GPG key API

* Add web UI to manage GPG keys

* Lint updates

* Change wording

* Add some plausibility checks for supplied data on key creation

* Update from codegen

* Re-allow binary keys and move check for ASCII armoured to UI

* Make yarn lint happy

* Add editing signature keys for projects in UI

* Add ability to configure signature keys for project in CLI

* Change default value to use for GNUPGHOME

* Do not include data section in default gpg keys CM

* Adapt Docker image for GnuPG feature

* Add required configuration to installation manifests

* Add add-signature-key and remove-signature-key commands to project CLI

* Fix typo

* Add initial user documentation for GnuPG verification

* Fix role name - oops

* Mention required RBAC roles in docs

* Support GPG verification of git annotated tags as well

* Ensure CLI can build succesfully

* Better support verification on tags

* Print key type in upper case

* Update user documentation

* Correctly disable GnuPG verification if ARGOCD_GPG_ENABLE=false

* Clarify that this feature is only available with Git repositories

* codegen

* Move verification code to own function

* Remove deprecated check

* Make things more developer friendly when running locally

* Enable GPG feature by default, and don't require ARGOCD_GNUPGHOME to be set

* Revert changes to manifests to reflect default enable state

* Codegen

Dockerfile

@@ -50,12 +50,14 @@ RUN groupadd -g 999 argocd && \
     chmod g=u /home/argocd && \
     chmod g=u /etc/passwd && \
     apt-get update && \
-    apt-get install -y git git-lfs python3-pip tini && \
+    apt-get install -y git git-lfs python3-pip tini gpg && \
     apt-get clean && \
     pip3 install awscli==1.18.80 && \
     rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 
 COPY hack/git-ask-pass.sh /usr/local/bin/git-ask-pass.sh
+COPY hack/gpg-wrapper.sh /usr/local/bin/gpg-wrapper.sh
+COPY hack/git-verify-wrapper.sh /usr/local/bin/git-verify-wrapper.sh
 COPY --from=builder /usr/local/bin/ks /usr/local/bin/ks
 COPY --from=builder /usr/local/bin/helm2 /usr/local/bin/helm2
 COPY --from=builder /usr/local/bin/helm /usr/local/bin/helm
@@ -71,6 +73,10 @@ RUN mkdir -p /app/config/ssh && \
     ln -s /app/config/ssh/ssh_known_hosts /etc/ssh/ssh_known_hosts 
 
 RUN mkdir -p /app/config/tls
+RUN mkdir -p /app/config/gpg/source && \
+    mkdir -p /app/config/gpg/keys && \
+    chown argocd /app/config/gpg/keys && \
+    chmod 0700 /app/config/gpg/keys
 
 # workaround ksonnet issue https://github.com/ksonnet/ksonnet/issues/298
 ENV USER=argocd

Makefile

@@ -98,6 +98,8 @@ IMAGE_NAMESPACE?=
 STATIC_BUILD?=true
 # build development images
 DEV_IMAGE?=false
+ARGOCD_GPG_ENABLED?=true
+ARGOCD_E2E_APISERVER_PORT?=8080
 
 override LDFLAGS += \
   -X ${PACKAGE}.version=${VERSION} \
@@ -159,6 +161,8 @@ codegen:
 
 .PHONY: cli
 cli: clean-debug
+	rm -f ${DIST_DIR}/${CLI_NAME}
+	mkdir -p ${DIST_DIR}
 	CGO_ENABLED=0 ${PACKR_CMD} build -v -i -ldflags '${LDFLAGS}' -o ${DIST_DIR}/${CLI_NAME} ./cmd/argocd
 
 .PHONY: cli-docker
@@ -331,7 +335,7 @@ test-e2e:
 test-e2e-local: cli
 	# NO_PROXY ensures all tests don't go out through a proxy if one is configured on the test system
 	export GO111MODULE=off
-	NO_PROXY=* ./hack/test.sh -timeout 15m -v ./test/e2e
+	ARGOCD_GPG_ENABLED=true NO_PROXY=* ./hack/test.sh -timeout 15m -v ./test/e2e
 
 # Spawns a shell in the test server container for debugging purposes
 debug-test-server:
@@ -354,9 +358,17 @@ start-e2e-local:
 	kubectl create ns argocd-e2e || true
 	kubectl config set-context --current --namespace=argocd-e2e
 	kustomize build test/manifests/base | kubectl apply -f -
+	# Create GPG keys and source directories
+	if test -d /tmp/argo-e2e/app/config/gpg; then rm -rf /tmp/argo-e2e/app/config/gpg/*; fi
+	mkdir -p /tmp/argo-e2e/app/config/gpg/keys && chmod 0700 /tmp/argo-e2e/app/config/gpg/keys
+	mkdir -p /tmp/argo-e2e/app/config/gpg/source && chmod 0700 /tmp/argo-e2e/app/config/gpg/source
+	if test "$(USER_ID)" != ""; then chown -R "$(USER_ID)" /tmp/argo-e2e; fi
 	# set paths for locally managed ssh known hosts and tls certs data
 	ARGOCD_SSH_DATA_PATH=/tmp/argo-e2e/app/config/ssh \
 	ARGOCD_TLS_DATA_PATH=/tmp/argo-e2e/app/config/tls \
+	ARGOCD_GPG_DATA_PATH=/tmp/argo-e2e/app/config/gpg/source \
+	ARGOCD_GNUPGHOME=/tmp/argo-e2e/app/config/gpg/keys \
+	ARGOCD_GPG_ENABLED=true \
 	ARGOCD_E2E_DISABLE_AUTH=false \
 	ARGOCD_ZJWT_FEATURE_FLAG=always \
 	ARGOCD_IN_CI=$(ARGOCD_IN_CI) \
@@ -383,8 +395,13 @@ start-local: mod-vendor-local
 	# check we can connect to Docker to start Redis
 	killall goreman || true
 	kubectl create ns argocd || true
+	rm -rf /tmp/argocd-local
+	mkdir -p /tmp/argocd-local
+	mkdir -p /tmp/argocd-local/gpg/keys && chmod 0700 /tmp/argocd-local/gpg/keys
+	mkdir -p /tmp/argocd-local/gpg/source
 	ARGOCD_ZJWT_FEATURE_FLAG=always \
 	ARGOCD_IN_CI=false \
+	ARGOCD_GPG_ENABLED=true \
 	ARGOCD_E2E_TEST=false \
 		goreman -f $(ARGOCD_PROCFILE) start ${ARGOCD_START}
 

Procfile

@@ -1,8 +1,8 @@
-controller: sh -c "FORCE_LOG_COLORS=1 ARGOCD_FAKE_IN_CLUSTER=true ARGOCD_TLS_DATA_PATH=${ARGOCD_TLS_DATA_PATH:=/tmp/argocd/tls} ARGOCD_SSH_DATA_PATH=${ARGOCD_SSH_DATA_PATH:=/tmp/argocd/ssh} go run ./cmd/argocd-application-controller/main.go --loglevel debug --redis localhost:${ARGOCD_E2E_REDIS_PORT:-6379} --repo-server localhost:${ARGOCD_E2E_REPOSERVER_PORT:-8081}"
-api-server: sh -c "FORCE_LOG_COLORS=1 ARGOCD_FAKE_IN_CLUSTER=true ARGOCD_TLS_DATA_PATH=${ARGOCD_TLS_DATA_PATH:=/tmp/argocd/tls} ARGOCD_SSH_DATA_PATH=${ARGOCD_SSH_DATA_PATH:=/tmp/argocd/ssh} go run ./cmd/argocd-server/main.go --loglevel debug --redis localhost:${ARGOCD_E2E_REDIS_PORT:-6379} --disable-auth=${ARGOCD_E2E_DISABLE_AUTH:-'true'} --insecure --dex-server http://localhost:${ARGOCD_E2E_DEX_PORT:-5556} --repo-server localhost:${ARGOCD_E2E_REPOSERVER_PORT:-8081} --port ${ARGOCD_E2E_APISERVER_PORT:-8080} --staticassets ui/dist/app"
+controller: sh -c "FORCE_LOG_COLORS=1 ARGOCD_FAKE_IN_CLUSTER=true ARGOCD_TLS_DATA_PATH=${ARGOCD_TLS_DATA_PATH:-/tmp/argocd-local/tls} ARGOCD_SSH_DATA_PATH=${ARGOCD_SSH_DATA_PATH:-/tmp/argocd-local/ssh} go run ./cmd/argocd-application-controller/main.go --loglevel debug --redis localhost:${ARGOCD_E2E_REDIS_PORT:-6379} --repo-server localhost:${ARGOCD_E2E_REPOSERVER_PORT:-8081}"
+api-server: sh -c "FORCE_LOG_COLORS=1 ARGOCD_FAKE_IN_CLUSTER=true ARGOCD_TLS_DATA_PATH=${ARGOCD_TLS_DATA_PATH:-/tmp/argocd-local/tls} ARGOCD_SSH_DATA_PATH=${ARGOCD_SSH_DATA_PATH:-/tmp/argocd-local/ssh} go run ./cmd/argocd-server/main.go --loglevel debug --redis localhost:${ARGOCD_E2E_REDIS_PORT:-6379} --disable-auth=${ARGOCD_E2E_DISABLE_AUTH:-'true'} --insecure --dex-server http://localhost:${ARGOCD_E2E_DEX_PORT:-5556} --repo-server localhost:${ARGOCD_E2E_REPOSERVER_PORT:-8081} --port ${ARGOCD_E2E_APISERVER_PORT:-8080} --staticassets ui/dist/app"
 dex: sh -c "go run github.com/argoproj/argo-cd/cmd/argocd-util gendexcfg -o `pwd`/dist/dex.yaml && docker run --rm -p ${ARGOCD_E2E_DEX_PORT:-5556}:${ARGOCD_E2E_DEX_PORT:-5556} -v `pwd`/dist/dex.yaml:/dex.yaml quay.io/dexidp/dex:v2.22.0 serve /dex.yaml"
 redis: docker run --rm --name argocd-redis -i -p ${ARGOCD_E2E_REDIS_PORT:-6379}:${ARGOCD_E2E_REDIS_PORT:-6379} redis:5.0.8-alpine --save "" --appendonly no --port ${ARGOCD_E2E_REDIS_PORT:-6379}
-repo-server: sh -c "FORCE_LOG_COLORS=1 ARGOCD_FAKE_IN_CLUSTER=true ARGOCD_TLS_DATA_PATH=${ARGOCD_TLS_DATA_PATH:=/tmp/argocd/tls} ARGOCD_SSH_DATA_PATH=${ARGOCD_SSH_DATA_PATH:=/tmp/argocd/ssh} go run ./cmd/argocd-repo-server/main.go --loglevel debug --port ${ARGOCD_E2E_REPOSERVER_PORT:-8081} --redis localhost:${ARGOCD_E2E_REDIS_PORT:-6379}"
+repo-server: sh -c "FORCE_LOG_COLORS=1 ARGOCD_FAKE_IN_CLUSTER=true ARGOCD_GNUPGHOME=${ARGOCD_GNUPGHOME:-/tmp/argocd-local/gpg/keys} ARGOCD_GPG_DATA_PATH=${ARGOCD_GPG_DATA_PATH:-/tmp/argocd-local/gpg/source} ARGOCD_TLS_DATA_PATH=${ARGOCD_TLS_DATA_PATH:-/tmp/argocd-local/tls} ARGOCD_SSH_DATA_PATH=${ARGOCD_SSH_DATA_PATH:-/tmp/argocd-local/ssh} go run ./cmd/argocd-repo-server/main.go --loglevel debug --port ${ARGOCD_E2E_REPOSERVER_PORT:-8081} --redis localhost:${ARGOCD_E2E_REDIS_PORT:-6379}"
 ui: sh -c 'cd ui && ${ARGOCD_E2E_YARN_CMD:-yarn} start'
 git-server: test/fixture/testrepos/start-git.sh
-dev-mounter: [[ "$ARGOCD_E2E_TEST" != "true" ]] && go run hack/dev-mounter/main.go --configmap argocd-ssh-known-hosts-cm=${ARGOCD_SSH_DATA_PATH:-/tmp/argocd/ssh} --configmap argocd-tls-certs-cm=${ARGOCD_TLS_DATA_PATH:=/tmp/argocd/tls}
+dev-mounter: [[ "$ARGOCD_E2E_TEST" != "true" ]] && go run hack/dev-mounter/main.go --configmap argocd-ssh-known-hosts-cm=${ARGOCD_SSH_DATA_PATH:-/tmp/argocd-local/ssh} --configmap argocd-tls-certs-cm=${ARGOCD_TLS_DATA_PATH:-/tmp/argocd-local/tls} --configmap argocd-gpg-keys-cm=${ARGOCD_GPG_DATA_PATH:-/tmp/argocd-local/gpg/source}

assets/builtin-policy.csv

@@ -12,6 +12,7 @@ p, role:readonly, clusters, get, *, allow
 p, role:readonly, repositories, get, *, allow
 p, role:readonly, projects, get, *, allow
 p, role:readonly, accounts, get, *, allow
+p, role:readonly, gpgkeys, get, *, allow
 
 p, role:admin, applications, create, */*, allow
 p, role:admin, applications, update, */*, allow
@@ -32,6 +33,8 @@ p, role:admin, projects, create, *, allow
 p, role:admin, projects, update, *, allow
 p, role:admin, projects, delete, *, allow
 p, role:admin, accounts, update, *, allow
+p, role:admin, gpgkeys, create, *, allow
+p, role:admin, gpgkeys, delete, *, allow
 
 g, role:admin, role:readonly
 g, admin, role:admin