Skip to content

Commit

Permalink
feat: new dev/test environment
Browse files Browse the repository at this point in the history
This PR simplifies and unifies the process of bringing up a hcloud k8s
environment that is suitable for development/testing purposes. It
builds off the work originally started in #226

The central script is `hack/dev-up.sh`. The script requires a
`HCLOUD_TOKEN`, and will build a 1+ k8s cluster using `k3sup` +
`hcloud` CLI tools. It's typically quite fast: a cold execution of the
script should take less <1min before a fully operational cluster is
ready for use.

The `dev-down.sh` script will delete all resources created by
`dev-up.sh`.

One of the primary objectives in this work was to simplify the e2e
testing process.

Skaffold is plumbed into the e2e test pipelines on GitHub. This way,
the process of building and deploying a test build of csi-driver to a
cluster is unified for development and test.

Once this work has landed and stabilized here, we expect to package it up a bit further and use it in csi-driver and anywhere else we operate Kubernetes integrations that need automated testing.

Co-authored-by: Sam Day <[email protected]>
  • Loading branch information
apricote and samcday committed Apr 6, 2023
1 parent 7c194ce commit fd93df9
Show file tree
Hide file tree
Showing 16 changed files with 310 additions and 982 deletions.
25 changes: 10 additions & 15 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,15 +1,10 @@
.dockerignore
Dockerfile
.git/
.github/
.gitlab-ci.yml
.idea/
CHANGES.md
LICENSE
README.md
e2etests/
deploy/
mock/
script/
**/*_test.go
kustomization.yaml
*
!api/
!app/
!cmd/
!csi/
!driver/
!metrics/
!volumes/
!go.mod
!go.sum
44 changes: 32 additions & 12 deletions .github/workflows/test_e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,14 @@ jobs:
# causes all other currently running jobs to abort and all need to be restarted.
fail-fast: false
matrix:
k8s: [ k8s-1.23.15, k8s-1.24.9, k8s-1.25.5, k8s-1.26.0 ]
name: k8s ${{ matrix.k8s }}
include:
- k3s: v1.24
k8s-test: v1.24.12
- k3s: v1.25
k8s-test: v1.25.8
- k3s: v1.26
k8s-test: v1.26.3
name: k3s ${{ matrix.k3s }}
steps:
- uses: actions/setup-go@v2
with:
Expand All @@ -34,17 +40,31 @@ jobs:
echo "::error ::Couldn't determine HCLOUD_TOKEN. Check that repository secrets are setup correctly."
exit 1
fi
- uses: actions/cache@v2
- uses: 3bit/setup-hcloud@v2
- uses: yokawasa/[email protected]
with:
path: |
~/go/pkg/mod
~/.cache/go-build
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-
setup-tools: |
helm
kubectl
skaffold
helm: v3.11.2
kubectl: v1.26.3
skaffold: v2.3.0

- name: Run tests
env:
K8S_VERSION: ${{ matrix.k8s }}
K3S_CHANNEL: ${{ matrix.k3s }}
K8S_TEST_VERSION: ${{ matrix.k8s-test }}
SCOPE: gha-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.k3s }}
run: |
go test $(go list ./... | grep e2etests) -v -timeout 60m
./script/delete-token.sh $HCLOUD_TOKEN
curl -sLS https://get.k3sup.dev | sh
trap "hack/dev-down.sh; ./script/delete-token.sh $HCLOUD_TOKEN" EXIT
source <(hack/dev-up.sh)
skaffold build --tag="e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_NUMBER}"
tag=$(skaffold build --tag="e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_NUMBER}" --quiet --output="{{ (index .Builds 0).Tag }}")
skaffold deploy --images=hetznercloud/hcloud-csi-driver=$tag
e2etests/run-e2e-tests.sh
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# rootfs for docker plugin
deploy/docker-swarm/pkg/plugin
hack/.*
5 changes: 1 addition & 4 deletions e2etests/.gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1 @@
ci-hcloud-csi-driver.tar
ssh_key
kubeconfig
join.txt
test-binaries/
48 changes: 0 additions & 48 deletions e2etests/e2e_test.go

This file was deleted.

30 changes: 30 additions & 0 deletions e2etests/run-e2e-tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
set -uex -o pipefail
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

k8s_test_version="${K8S_TEST_VERSION:-v1.26.3}"

mkdir -p "${SCRIPT_DIR}/test-binaries"
# TODO: Read linux-amd64 from env
curl --location "https://dl.k8s.io/${k8s_test_version}/kubernetes-test-linux-amd64.tar.gz" | \
tar --strip-components=3 -C "${SCRIPT_DIR}/test-binaries" -zxf - kubernetes/test/bin/e2e.test kubernetes/test/bin/ginkgo

ginkgo="${SCRIPT_DIR}/test-binaries/ginkgo"
ginkgo_flags="-v --flakeAttempts=2"

e2e="${SCRIPT_DIR}/test-binaries/e2e.test"
e2e_flags="-storage.testdriver=${SCRIPT_DIR}/testdriver-1.23.yaml"

echo "Executing parallel tests"
${ginkgo} ${ginkgo_flags} \
-nodes=6 \
-focus='External.Storage' \
-skip='\[Feature:|\[Disruptive\]|\[Serial\]' \
"${e2e}" -- ${e2e_flags}

echo "Executing serial tests"
${ginkgo} ${ginkgo_flags} \
-focus='External.Storage.*(\[Feature:|\[Serial\])' \
-skip='\[Feature:SELinuxMountReadWriteOncePod\]' \
"${e2e}" -- ${e2e_flags}

Loading

0 comments on commit fd93df9

Please sign in to comment.