-
Notifications
You must be signed in to change notification settings - Fork 103
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This PR simplifies and unifies the process of bringing up a hcloud k8s environment that is suitable for development/testing purposes. It builds off the work originally started in #226 The central script is `hack/dev-up.sh`. The script requires a `HCLOUD_TOKEN`, and will build a 1+ k8s cluster using `k3sup` + `hcloud` CLI tools. It's typically quite fast: a cold execution of the script should take less <1min before a fully operational cluster is ready for use. The `dev-down.sh` script will delete all resources created by `dev-up.sh`. One of the primary objectives in this work was to simplify the e2e testing process. Skaffold is plumbed into the e2e test pipelines on GitHub. This way, the process of building and deploying a test build of csi-driver to a cluster is unified for development and test. Once this work has landed and stabilized here, we expect to package it up a bit further and use it in csi-driver and anywhere else we operate Kubernetes integrations that need automated testing. Co-authored-by: Sam Day <[email protected]>
- Loading branch information
Showing
16 changed files
with
310 additions
and
982 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,15 +1,10 @@ | ||
.dockerignore | ||
Dockerfile | ||
.git/ | ||
.github/ | ||
.gitlab-ci.yml | ||
.idea/ | ||
CHANGES.md | ||
LICENSE | ||
README.md | ||
e2etests/ | ||
deploy/ | ||
mock/ | ||
script/ | ||
**/*_test.go | ||
kustomization.yaml | ||
* | ||
!api/ | ||
!app/ | ||
!cmd/ | ||
!csi/ | ||
!driver/ | ||
!metrics/ | ||
!volumes/ | ||
!go.mod | ||
!go.sum |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,8 +9,14 @@ jobs: | |
# causes all other currently running jobs to abort and all need to be restarted. | ||
fail-fast: false | ||
matrix: | ||
k8s: [ k8s-1.23.15, k8s-1.24.9, k8s-1.25.5, k8s-1.26.0 ] | ||
name: k8s ${{ matrix.k8s }} | ||
include: | ||
- k3s: v1.24 | ||
k8s-test: v1.24.12 | ||
- k3s: v1.25 | ||
k8s-test: v1.25.8 | ||
- k3s: v1.26 | ||
k8s-test: v1.26.3 | ||
name: k3s ${{ matrix.k3s }} | ||
steps: | ||
- uses: actions/setup-go@v2 | ||
with: | ||
|
@@ -34,17 +40,31 @@ jobs: | |
echo "::error ::Couldn't determine HCLOUD_TOKEN. Check that repository secrets are setup correctly." | ||
exit 1 | ||
fi | ||
- uses: actions/cache@v2 | ||
- uses: 3bit/setup-hcloud@v2 | ||
- uses: yokawasa/[email protected] | ||
with: | ||
path: | | ||
~/go/pkg/mod | ||
~/.cache/go-build | ||
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} | ||
restore-keys: | | ||
${{ runner.os }}-go- | ||
setup-tools: | | ||
helm | ||
kubectl | ||
skaffold | ||
helm: v3.11.2 | ||
kubectl: v1.26.3 | ||
skaffold: v2.3.0 | ||
|
||
- name: Run tests | ||
env: | ||
K8S_VERSION: ${{ matrix.k8s }} | ||
K3S_CHANNEL: ${{ matrix.k3s }} | ||
K8S_TEST_VERSION: ${{ matrix.k8s-test }} | ||
SCOPE: gha-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.k3s }} | ||
run: | | ||
go test $(go list ./... | grep e2etests) -v -timeout 60m | ||
./script/delete-token.sh $HCLOUD_TOKEN | ||
curl -sLS https://get.k3sup.dev | sh | ||
trap "hack/dev-down.sh; ./script/delete-token.sh $HCLOUD_TOKEN" EXIT | ||
source <(hack/dev-up.sh) | ||
skaffold build --tag="e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_NUMBER}" | ||
tag=$(skaffold build --tag="e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_NUMBER}" --quiet --output="{{ (index .Builds 0).Tag }}") | ||
skaffold deploy --images=hetznercloud/hcloud-csi-driver=$tag | ||
e2etests/run-e2e-tests.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# rootfs for docker plugin | ||
deploy/docker-swarm/pkg/plugin | ||
hack/.* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1 @@ | ||
ci-hcloud-csi-driver.tar | ||
ssh_key | ||
kubeconfig | ||
join.txt | ||
test-binaries/ |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
#!/usr/bin/env bash | ||
set -uex -o pipefail | ||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" | ||
|
||
k8s_test_version="${K8S_TEST_VERSION:-v1.26.3}" | ||
|
||
mkdir -p "${SCRIPT_DIR}/test-binaries" | ||
# TODO: Read linux-amd64 from env | ||
curl --location "https://dl.k8s.io/${k8s_test_version}/kubernetes-test-linux-amd64.tar.gz" | \ | ||
tar --strip-components=3 -C "${SCRIPT_DIR}/test-binaries" -zxf - kubernetes/test/bin/e2e.test kubernetes/test/bin/ginkgo | ||
|
||
ginkgo="${SCRIPT_DIR}/test-binaries/ginkgo" | ||
ginkgo_flags="-v --flakeAttempts=2" | ||
|
||
e2e="${SCRIPT_DIR}/test-binaries/e2e.test" | ||
e2e_flags="-storage.testdriver=${SCRIPT_DIR}/testdriver-1.23.yaml" | ||
|
||
echo "Executing parallel tests" | ||
${ginkgo} ${ginkgo_flags} \ | ||
-nodes=6 \ | ||
-focus='External.Storage' \ | ||
-skip='\[Feature:|\[Disruptive\]|\[Serial\]' \ | ||
"${e2e}" -- ${e2e_flags} | ||
|
||
echo "Executing serial tests" | ||
${ginkgo} ${ginkgo_flags} \ | ||
-focus='External.Storage.*(\[Feature:|\[Serial\])' \ | ||
-skip='\[Feature:SELinuxMountReadWriteOncePod\]' \ | ||
"${e2e}" -- ${e2e_flags} | ||
|
Oops, something went wrong.