Setup automated builds using the release-tools #30
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The repo was created with an HTML version of the build.make file from https://github.com/pohly/csi-build-rules/. Here's the raw file.
It's worth calling out explicitly that only the master branch is maintained.
The actual repository was not named like the prototype repo.
initial content
Copy-and-paste error from the time when the kubernetes-csi/csi-release-tools repo didn't have the code...
README.md: fix repo URL for initial setup
The goal is to enforce that changes get merged upstream first and only get into the local repo via a normal "git subtree merge".
"make test" used to abort after the first test failure. That was partly intentional: if the simple tests already fail (for example, because of a syntax error), then there is no point in continuing to test. However, it also makes it harder to find all errors in a CI system when the errors are unrelated (first error shows up, gets fixed, next error shows up, etc.). Now "make test" still aborts early, but "make -k test" is used in the CI and will run all individual tests because they are split up into different targets.
We don't want to allow local modifications in the subtree. Everything should go to the csi-release-tools repo first.
check subtree for changes
This may or may not work, depending on which packages have tests and whether they contain glog.
Individual repos may have to filter out certain packages from testing. For example, in csi-test the cmd/csi-sanity directory contains a special test that depends on additional parameters that set the CSI driver to test against.
After merging into external-attacher, the next Travis CI run did not push the "canary" image because the check for "canary" only covered the case where "-canary" is used as suffix (https://travis-ci.org/kubernetes-csi/external-attacher/builds/484095261).
The introduction for each individual test looked like an actual command: test-subtree ./release-tools/verify-subtree.sh release-tools Directory 'release-tools' contains non-upstream changes: ... It's better to make it look like a shell comment and increase its visibility with a longer prefix: ### test-subtree: ./release-tools/verify-subtree.sh release-tools ...
build.make: fix pushing of "canary" image from master branch
test enhancements
If for whatever reasons a repo already had a `release-tools` directory before doing a clean import of it with `git subtree`, the check used to fail because it found those old commits. This can be fixed by telling `git log` to stop when the directory disappears from the repo. There has to be a commit with removes the old content, because otherwise `git subtree add` doesn't work. Fixes: kubernetes-csi/external-resizer#21
verify-subtree.sh: relax check and ignore old content
In repos that have a test/e2e, that test suite should be run separately because it depends on a running cluster.
This is an unmodified copy of kubernetes/hack/verify-shellcheck.sh revision d5a3db003916b1d33b503ccd2e4897e094d8af77.
This runs "dep check" to verify that the vendor directory is up-to-date and meets expectations (= done with dep >= 0.5.0).
check vendor directory
In repos that have a test/e2e, that test suite should be run separately because it depends on a running cluster.
build.make: avoid unit-testing E2E test suite
These are the modifications that were necessary to call this outside of Kubernetes. The support for excluding files from checking gets removed to simplify the script. It shouldn't be needed, because linting can be enabled after fixing whatever scripts might fail the check.
By default this only tests the scripts inside the "release-tools" directory, which is useful when making experimental changes to them in a component that uses csi-release-tools. But a component can also enable checking for other directories.
This enables testing of other repos and of this repo itself inside
Prow. Currently supported is unit testing ("make test") and E2E
testing (either via a local test suite or the Kubernetes E2E test
suite applied to the hostpath driver example deployment).
The script passes shellcheck and uses Prow to verify that for future
PRs.
…d-workaround prow.sh: work around "kind build node-image" failure
prow.sh: only run alpha tests for latest Kubernetes release
…_v1_rc Update snapshot CRD version
This PR installs snapshot CRDs and rbac rules from the repo and installs snapshot controller from a local image if it is a PR in external-snapshotter repo. Otherwise it uses main or a stable version.
…t_controller Fix the install of snapshot CRDs and controller
revert part of change
allow export registry name
Builds of external-snapshotter sometimes timed out because it builds three images and that takes almost half an hour (the previous limit) and more if the network is slow. There's no good way to set timeouts per repo and a too high timeout for shouldn't cause any issue, therefore the timeout gets increased everywhere.
cloud build: double the timeout, now 1 hour
Follow the steps here to setup automated builds via release-tools. https://github.com/kubernetes-csi/csi-release-tools - add release-tools via git subtree - update Makefile for arm builds to use the release-tools/make - add cloudbuild and travis links from release-tools - update Dockerfile for amd64 as expected by release tools Signed-off-by: kmova <[email protected]>
k8s-ci-robot
added
approved
msau42
reviewed
Dec 14, 2020
| build-docker build-docker-x86_64: | ||
| GOOS=linux go build -o deploy/docker/x86_64/nfs-provisioner ./cmd/nfs-provisioner | ||
| .PHONY: build-docker build-docker-x86_64 | ||
| IMAGE_ARM = $(REGISTRY)nfs-provisioner-arm:$(VERSION) |
There was a problem hiding this comment.
cloud build and release tools already supports multiarch builds. I'm not sure we need these arm specific params and commands?
There was a problem hiding this comment.
The Dockerfiles for arm and amd64 are a bit different at the moment. We need to work on refactoring them to multi-arch compatible Docker files. Prior to refactoring, we need to add fix the e2e tests.
The current users are building with the existing Docker files and make commands in forked repos - so tried to maintain that backward compatibility for now till the refactoring is completed.
| go test `go list ./... | grep -v 'vendor\|test\|demo'` | ||
| .PHONY: test | ||
|
|
||
| test-e2e: |
There was a problem hiding this comment.
do we have a replacement for test-e2e?
There was a problem hiding this comment.
Not yet, this is coming up in a follow-up PR. Currently, these tests are out of date and not being used.
| @@ -0,0 +1 @@ | |||
| ./release-tools/travis.yml | |||
There was a problem hiding this comment.
we probably don't need travis if we enable all testing via prow
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kmova, msau42 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This was referenced Dec 15, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Followed the steps listed here (https://github.com/kubernetes-csi/csi-release-tools)
to setup automated builds via release-tools.
All the commits except (1311c59) are from
git add subtree.Verified that following commands are working:
make allmakemake containermake container-arm