v0.17.0

Sonobuoy v0.17.0

This release corresponds to the Kubernetes v1.17.0 release. It takes place very close to our previous release, Sonobuoy v0.16.5. The main user-facing change is support for Kubernetes v1.17.x.

It also contains a number of fixes of links in our documentation.

Changelog

93f63ef Prepare v0.17.0 release (#1041)
1af0378 Add results to shortlinks and update link to remove .md suffix (#1039)
28f410a Add blog post describing progress updates from e2e plugin
5884d2a Add FAQ to shortlinks (#1033)
4d9ed5a Merge pull request #1032 from johnSchnake/kubebenchblogImage
9511856 Update blog post about kube-bench image

v0.16.5

Sonobuoy v0.16.5

What's new

• Sonobuoy will now begin tailing logs from new containers as they are created while running sonobuoy logs -f.
• If targeting v1.17 clusters and above, the e2e plugin will now provide the necessary logic in order to enable progress updates from the plugin. At the current time, you can see progress via sonobuoy status --json.
• Sonobuoy can now output the default configuration for the registries used by the e2e plugin using the new command gen default-image-config. This will output the default configuration for the version of Kubernetes that is being tested. This configuration is required for using the images command.
• Sonobuoy will now publish 32-bit Windows/Linux clients and arm64 clients on its Github release page.

What's better

Documentation improvements

We made a number of improvements to our documentation during this release cycle:

• Added an FAQ for common questions around use of the e2e plugin.
• Updated and improved our documentation on how to run Sonobuoy in airgapped environments.
• Made a number of changes to improve our documentation site's accessibility score.
• Fixed a link that pointed to an old version and some typos.

Changelog

99d45a1 Prepare v0.16.5 release (#1031)
4a88c74 Merge pull request #1019 from SDBrett/issue-1018
8a0c2ac Improve site accessibility score (#1017)
a768c66 Add an FAQ (#998)
05dc053 Improve instructions for airgapped environments (#1023)
cff00ca Add command to ouput default image registry config (#1022)
e189def Fixes: #1018
44be251 fix typo in e2eplugin page (#1014)
46ae5a9 Merge pull request #1011 from johnSchnake/e2eProgress
9a40478 Default to enableing progress updates from E2E plugin
fb517c2 Merge pull request #1013 from rbankston/issue1012
e22dd1a Updated to just /docs so this doesn't have to be updated again
ab6125a Updated docs link to point to latest
9e41652 Update to log logic in order to stream from new containers as they come up (#1007)
1b0ea1a Publish 32-bit and arm64 clients (#1009)
36709f5 Fix formatting of commands in CIS blog post (#1004)
7ce6965 Add CIS Benchmark plugin blog post (#1002)

v0.16.4

Sonobuoy v0.16.4

What's new

• The --plugin flag now supports URLs in addition to paths to local files or directories. (#989)
• Daemonset plugins may now specify node affinity in their podSpec without extraneous errors. Until now, this could be specified but Sonobuoy would record errors due to "missing" plugins on the non-targeted nodes. Currently support In/NotIn/Exists/DoesNotExists operators only. (#990)
• The namespace and label selectors used by Sonobuoy to find the DNS pods during the preflight checks can now be customised using the flags --dns-namespace and --dns-pod-labels. (#987)

What's better

• Sonobuoy will no longer treat the first encounter of ErrImagePull or ImagePullBackOff as a final error state for a plugin. It will instead allow a pod up to 5 minutes to recover from this in case the error is transient. (#997)
• Fixed a bug which caused some error files from being properly parsed by Sonobuoy leading to less helpful output from the sonobuoy results command. (#995)

Changelog

cda71c3 Bump version and docs for v0.16.4 (#999)
ad67ceb Add wait period to allow image pull error recovery (#997)
6f983e2 Ensure errors.json with structured data is parsed (#995)
c5ad70d Allow daemonsets to use affinity to limit nodes they run on (#990)
73cdb50 Allow plugins to be specified via a URL (#989)
31b5728 Allow DNS pod selectors to be configured (#987)

v0.16.3

Sonobuoy v0.16.3

What's new

Support for prerelease E2E test images

Sonobuoy will now utilize the prerelease info from the server version when looking up the conformance image. This means that if your cluster is version v1.17.0-beta.0 it will look up that exact version (which the upstream publishes currently). It continues to ignore metadata (e.g. for v1.17.0-beta.0+otherdata the "+otherdata" is dropped to be consistent with current image publishing practice).

What's better

• Sonobuoy now sets the OwnerReference on the Secret resource created when running plugins which is used when communicating with the main aggregator pod. This means that if the main aggregator pod is deleted, this Secret will also be deleted.
• Fixed an issue where the systemd-logs plugin would restart when the e2e tests ran for longer than 1h. This led to confusing messages in the logs.

Changelog

bda80c2 Prepare v0.16.3 release (#983)
871ba62 Use prerelease info when looking up conformance image (#978)
f463b6d Fix broken e2eplugin link (#974)
f673ed7 Set OwnerReference on plugin TLS Secret (#973)
cd942b3 Sleep forever after systemd logs plugin finishes (#971)
f3b4e35 Update github org in one of our scripts (#960)

v0.16.2

Sonobuoy v0.16.2

What's new

Piping data into sonobuoy run

In cases where large changes to YAML from sonobuoy gen need to occur, we've recommended piping the data then to kubectl apply -f - to start the run. However, when you do that you lose out on the ability to use the --wait flag which can be very helpful.

We decided to add a -f flag to sonobuoy run so that you can load files or pipe to it from stdin. For example:

$ sonobuoy gen > my.yaml
$ cat my.yaml | sonobuoy run -f -

This may be a very useful feature for users who routinely need to change parts of the YAML since you can include other tools as well:

sonobuoy gen | <commands to change the yaml> | sonobuoy run -f -

What's better

Improved documentation

Documentation doesn't always get the attention it deserves. As we've been more active with Sonobuoy development in the past few months a number of shortcomings have been noted and we tried to clarify things and improve organization.

Clear timeouts

One relatively common failure mode is that the plugins fail to provide results within the timeout period. Historically when this happens users would get partial results from the e2e plugin which could be misleading since it may appear that N tests passed but it wasn't clear that some were not run at all. In addition, the logs noted the timeout, but the ended up in the middle of all the logs and it was difficult to notice.

As a result, we added more logging and tried to put it in more visible places. In addition, we made changes to the results processing logic that would be aware of problems like this and mark the plugins as having a status of timeout. Hopefully these changes will save hours of unnecessary debugging.

Bug fixes

• Fixed a bug which caused some nodes results from a daemonset plugin to not be considered when doing post-processing.
• Fixed a bug where a plugin which failed to run wouldn't have its errors directory considered during post-processing leading to an "unknown" status instead of a failure.

Other

• Blog post on how to customize the E2E test image and run with unique testing options set: blog
• Moved from Travis to CircleCI
• Moved from gcr.io to DockerHub for image hosting
• Updated imports for the new Github org

Changelog

9b83cac Don't set branch just let it be calculated (#966)
a16e3e4 Tweaks to CI (#964)
3d0a548 Fixing up CI scripts to publish correctly (#963)
fe2646f Adjust our CircleCI to list tags on required job (#961)
7295394 Add older blog posts and author images (#958)
0838111 Prepare v0.16.2 release (#957)
26cb536 Update release process docs to reference new systems (#956)
6077455 Improve docs organization and E2E descriptions (#951)
3d03f98 Make timeouts more clear (#938)
bb1398c Remove .travis.yml to disable TravisCI (#954)
71dd2f9 Update badge to be from CircleCI not Travis (#953)
a673ac1 Unity integration testing and remove Travis (#949)
381d9e9 Make the publish script executable (#948)
be325ce Fixup to path to script (#947)
ff5326e Move image pushing into CircleCI (#945)
03e44e6 Adds new blog post regarding customizing the E2E test image (#944)
6899c87 Add CircleCI config (#940)
e18111d Update README to download release instead of go get (#942)
1e57ed8 Move to go modules (#935)
2686723 Results aggregation should consider the errors directory (#936)
69cc7ff Process all nodes and treat empty status as unknown (#937)
61cb07b Fixup to docker image push (#934)
03fd8b6 Get dockerhub image pushing to work on Travis (#932)
7218d76 Use created namespace name during run --wait (#930)
15ab7f3 Update our registry to dockerhub and use our latest kind-node (#927)
c799770 Fix blockquote rendering (#923)
4ba8b74 Add new GitHub org blog post (#928)
3290552 Allow piping input to sonobuoy run (#925)
3a8f83a Update repo throughout codebase (#924)

v0.16.1

Sonobuoy v0.16.1

Sonobuoy v0.16.1 mitigates a major bug found in the upstream E2E conformance image. Users are highly encouraged to update. See #910 for more details.

What's new

Golang-based test runner for Kubernetes 1.16.0+ clusters.

Within the test image used for end-to-end tests are some wrappers which invoke those tests. Historically those were written in bash but various problems have occurred which caused us to add a Golang-based runner. The benefits of this are:

• avoids a major problem where 1.16.0 clusters with the bash runner would not report test results properly when there are failures
• improved error handling and logging
• ability to further customize the test run by allowing arbitrary flags to be passed via the env vars. This means that many more testing options are supported including provider specific settings that were previously unsettable.

What's better

• sonobuoy results will print results for all plugins by default. This way you don't have to remember which plugins were run or how their names are exactly spelled to figure out what is inside a tarball.
• sonobuoy gen plugin added a --format flag which allows you to specify the result format.

Changelog

c9c2a46 Prepare for v0.16.1 Release (#921)
2c92062 Tweak readme with info about results. (#917)
0e16b5e Default to using the golang-based conformance runner (#916)
bf2fa52 Add a small test which runs a basic plugin (#903)
06f638e Allow the --plugin flag to specify a directory of plugins (#911)
d9ca4ba Prints all plugins when invoking sonobuoy results by default (#909)
aae48ef Adds blog post summarizing improvements made up until 0.16.0 (#908)
6e5dcaa Adds a testImage for our integration tests (#907)
d42d0f1 Adds the results format as a flag parameter for gen plugin (#904)
e30312b Add kind and dep to path in Travis build (#902)
7473b3d Remove unused dependency from Gopkg.toml (#898)
a8e4c1c Add a test to check that version fields are set (#894)

v0.16.0

Sonobuoy v0.16.0

This release corresponds to the Kubernetes v1.16.0 release. It takes place very close to our previous release, Sonobuoy v0.15.4, and does not contain any user-facing changes other than supporting Kubernetes v1.16.x.

Changelog

6db0b0b Prepare v0.16.0 release (#896)
b0fb524 Rename stress test directory to match package name (#893)
0b77329 Fix script to make new versioned docs (#892)

v0.15.4

Sonobuoy v0.15.4

What's new

• Supported modes

The supported "modes" for sonobuoy run have changed. The supported values are now quick, non-disruptive-conformance, and certified-conformance. This is due to Kubernetes changes in v1.16 which will be allowing disruptive tests to be included in Conformance runs. Sonobuoy, by default, will seek to protect your workloads by avoiding those disruptive tests unless you explicitly state that you want to run in certified-conformance mode.

To be clear, sonobuoy run is most appropriate as a safe way to test that your cluster is in a good state.
sonobuoy run --mode certified-conformance will run all the conformance tests for submission to the CNCF Certified Kubernetes program.

• Progress updates

Sonobuoy now has support for collecting and reporting on the progress of plugins while they are still running. This functionality is currently just in Sonobuoy itself and is not yet implemented in the upstream E2E tests, where it is most useful. A new [example][https://github.com/heptio/sonobuoy/tree/master/examples/plugins/progress-reporter] has been added to illustrate how you can utilize this functionality in your own plugins.

What's better

• In v1.16 Kubernetes has a few E2E tests which may evict user workloads from a node (see the above item about certified-conformance mode. To ensure that Sonobuoy and your plugins run appropriately and do not get evicted, a new toleration is added by default.
• When tailing logs via sonobuoy logs -f, if Sonobuoy sees a container but it is not yet running, we will continue to watch the container and start streaming logs once it enters the running state.
• If a plugin fails to report any results and is considered an unknown result, that will bubble up and cause the plugin to properly be marked as unknown rather than passed by default. This reduces the number of false positives possible.

Changelog

3d6c1be Bump version and add v0.15.4 docs (#886)
90a586f Add certified conformance mode and non-disruptive-conformance mode (#885)
cb43587 Progress reports for plugins (#869)
8bd715b Remove 'extended' mode (#884)
9eae93e Minor change to retrigger blog publishing (#882)
06fcf66 Add blog post for Customizing PodSpecs (#879)
f7e2366 Add a disruptive taint to list of default tolerations (#878)
0f9e963 check for container running before following logs (#706)
e03eebd Remove unnecessary legacy branding (#873)
cfde6a0 Update to Go 1.13 (#871)
3975a9e Show logs for a specific plugin (#868)
bc64713 Adjust MASTER_URL to be just the hostname (#867)
a603fdb Bubble up unknown results just like failures (#866)
bfa3ade Fix broken links and reference to conformance image (#864)

v0.15.3

Sonobuoy v0.15.3

What's new

• The results post-processing for JUnit now supports output which contains a testsuites object consisting of multiple test suites rather than just being able to support a single testsuite object.

What's better

• Plugin definitions (e.g. output from sonobuoy gen plugin and input for the --plugin flag) no longer have a result-type field. This was usually identical to the plugin-name field and only led to confusing users or causing issues if those values were not identical.

Changelog

4737048 Update version and add new docs for v0.15.3 (#862)
250dfc1 Update release docs for new build flow (#861)
3b14025 Remove plugin.ResultType due to confusion/lack of need (#857)
2d47854 Add documentation for custom PodSpecs (#859)
a268401 Add junit support for non-e2e tests/multi-suite (#851)
451b313 Use MarshalIndent on json goldenfiles (#856)
973e782 Rename master to aggregator (#847)
58b331c Update gems (#850)

v0.15.2

Sonobuoy v0.15.2

What's new

• PodSpec fields can now be specified for individual plugins.
  • In addition to specifying the container and extra volumes required for a plugin, you can now modify the PodSpec used by Sonobuoy to launch the plugin. You can view the default PodSpec used by Sonobuoy for both Job and DaemonSet plugins by using the new flag --show-default-podspec on both the gen and gen plugin commands. This will allow you to modify the default one, or use it as a basis to provide your own for existing plugins. Sonobuoy will still add resources that it requires to this PodSpec before running the plugin (such as containers to run the plugin image and the Sonobuoy worker) but won't remove or change any settings you provide.
• Plugin results are now placed onto the aggregators status annotation
  • This means you can now see if plugin had any test failures without even downloading the results tarball. You can use sonobuoy status to see the top-level results (e.g. passed/failed). Alternatively, a new --json flag was also added so you can see/parse the full status object.
  • We also added some metadata to the status object so you can relate it to the tarball you download. Fields include filename, sha256, and size.

What's better

• OwnerReference set on plugin pods/daemonsets
  • By setting this metadata field when the aggregator launches plugins, we now ensure that plugin resources are deleted if the aggregator pod is deleted.
• The sonobuoy results command now reports unknown on junit plugins if no xml files were processed instead of passed. This helps properly identify cases where a plugin may not generate the junit properly.
• Added Details field to the results.Item object. This means that sonobuoy results --mode=detailed can not just list the tests which failed but also the failure message and captured stdout.

Bug fixes

• Fixed a bug which caused Sonobuoy to not filter based on namespace properly when running queries after plugins are run.

Changelog

d776707 Bump version of Sonobuoy and add new docs for v0.15.2 (#846)
031d29d Set OwnerReference on Plugin resources (#845)
2bec2b1 Modify some old docs to avoid liquid syntax warnings (#844)
2fb844c Ensure we filter resources by namespace (#842)
c7d0673 Ensure a plugin isn't reported as passed if no results were read (#841)
5238cfa Add Details map to post-processed Items (#838)
0fb9865 Add ability to override pod spec options (#837)
1cce037 Add results info to the status annotation and status cmd output (#829)
3a04f8f Use gcloud auth instead of docker-credential-gcr (#836)
83e39f2 Add image building/pushing back to Travis (#835)
4c07889 Remove use of templates for plugin Jobs and DaemonSets (#834)
94702fb Sync master/root README.md and provide a script to do this (#831)
f68cb44 Remove plugin command (#833)
01a2fcb Fix SEO title output on docs
4cda024 Add SEO functionality to the site
838ea5a Add 0.15.1 blog post (#828)