has_children | nav_order |
---|---|
true |
8 |
{: .no_toc }
Kola is a framework for testing software integration in CoreOS systems across multiple platforms. It is primarily designed to operate within the CoreOS Assembler for testing software that has landed in the OS image.
Kola supports running tests on multiple platforms, currently QEMU, GCE, AWS, VMware VSphere, Packet, and OpenStack. In the future systemd-nspawn and other platforms may be added. Local platforms do not rely on access to the Internet as a design principle of kola, minimizing external dependencies. Any network services required get built directly into kola itself. Machines on cloud platforms do not have direct access to the kola so tests may depend on Internet services such as discovery.etcd.io or quay.io instead.
Kola outputs assorted logs and test data to _kola_temp
for later
inspection.
Kola is still under heavy development and it is expected that its interface will continue to change.
By default, kola uses the qemu-unprivileged
platform with the most recently
built image (assuming it is run from within a CoreOS Assembler working
directory).
- TOC {:toc}
The run command invokes the main kola test harness. It runs any tests whose registered names matches a glob pattern.
kola run <glob pattern>
--denylist-test
can be used if one or more tests in the pattern should be skipped.
This switch may be provided once:
kola --denylist-test linux.nfs.v3 run
multiple times:
kola --denylist-test linux.nfs.v3 --denylist-test linux.nfs.v4 run
and can also be used with glob patterns:
kola --denylist-test linux.nfs* --denylist-test crio.* run
Tests specified in src/config/kola-denylist.yaml
will also be skipped
regardless of whether the switch --denylist-test
was provided.
It's also possible to skip tests based on tags by prefixing
the tag by !
:
kola run --tag '!reprovision'
Example format of the file:
- pattern: test1.blobpattern.*
tracker: https://github.com/coreos/coreos-assembler/pull/123
streams:
# This test will be skipped in these streams
# If no streams are specified, test will be skipped on all streams
- stream1
- stream2
# The test will only be skipped until this date (will resume on the date)
# Format: YYYY-MM-DD
snooze: 2021-07-20
arches:
# This test will be skipped on these arches
# If no arches are specified, test will be skipped on all arches
- s390x
platforms:
# This test will be skipped on these platforms
# If no platforms are specified, test will be skipped on all platforms
- openstack
- aws
- pattern: test2.test
...
The list command lists all of the available tests.
The spawn command launches CoreOS instances.
The bootchart command launches an instance then generates an svg of the boot
process using systemd-analyze
.
Subtests can be parallelized by adding c.H.Parallel()
at the top of the
inline function given to c.Run
. It is not recommended to utilize the
FailFast
flag in tests that utilize this functionality as it can have
unintended results.
The top-level namespace of tests should fit into one of the following categories:
- Groups of tests targeting specific packages/binaries may use that namespace
(ex:
docker.*
) - Tests that target multiple supported distributions may use the
coreos
namespace. - Tests that target singular distributions may use the distribution's namespace.
Registering kola tests currently requires that the tests are registered under the kola package and that the test function itself lives within the mantle codebase.
Groups of similar tests are registered in an init() function inside the kola
package. Register(*Test)
is called per test. A kola Test
struct requires a
unique name, and a single function that is the entry point into the test.
Additionally, userdata (such as an Ignition config) can be supplied. See the
Test
struct in
kola/register/register.go
for a complete list of options.
A kola test is a go function that is passed a platform.TestCluster
to
run code against. Its signature is func(platform.TestCluster)
and must be registered and built into the kola binary.
A TestCluster
implements the platform.Cluster
interface and will give you
access to a running cluster of CoreOS machines. A test writer can interact with
these machines through this interface.
To see test examples look under kola/tests in the mantle codebase.
For a quickstart see kola/adding-tests.md.
For some tests, the Cluster
interface is limited and it is desirable to run
native go code directly on one of the CoreOS machines. This is currently
possible by using the NativeFuncs
field of a kola Test
struct. This like a
limited RPC interface.
NativeFuncs
is used similar to the Run
field of a registered kola test. It
registers and names functions in nearby packages. These functions, unlike the
Run
entry point, must be manually invoked inside a kola test using a
TestCluster
's RunNative
method. The function itself is then run natively on
the specified running CoreOS instances.
For more examples, look at the coretest suite of tests under kola. These tests were ported into kola and make heavy use of the native code interface.
Some tests are light weight and do not involve complex interactions like reboots and multiple machines. Tests that are not expected to conflict with other tests can be marked as "non-exclusive", so that they are run in the same VM to save resources.
External tests can be marked as non-exclusive via kola.json or an inline tag. Note: tests compiled in kola (non external tests) cannot be marked as non-exclusive. This is deliberate as tests compiled in kola should be complex and thus exclusive.
The platform.Manhole()
function creates an interactive SSH session which can
be used to inspect a machine during a test.
The --ssh-on-test-failure
flag can be specified to have the kola runner
automatically SSH into a machine when any MustSSH
calls fail.
kolet is run on kola instances to run native functions in tests. Generally kolet is not invoked manually.
After you run the kola test, you can find more information in tmp/kola/<test-name>
about the test that just ran, as the following file logs. They will help you to debug the problem and will certainly give you hints along the way.
journal.txt
console.txt
ignition.json
journal-raw.txt.gz
- Extended artifacts need additional forms of testing (You can pass the ignition and the path to the artifact you want to test)
cosa kola run -h
(this allows you to see the commands yourself and what syntax is needed)cosa buildextend-"name_of_artifact"
(An example of building an extended artifact)kola run -p <platform>
Is the most generic way of testing extended artifacts, this is mostly useful for the cloud platforms- For running the likes of metal/metal4k artifacts there's not much difference than running
kola run
from the coreos-assembler cd builds/latest/
(This will show your latest build information)cosa list
(This will show you the most recent CoreOS builds that have been made and the artifacts that were created)- In the case of the
testiso
command, you'll see that there is the--qemu-native-4k
option passed tokola testiso
. This instructs thetestiso
test to attempt to install FCOS/RHCOS to a disk that uses 4k sector size. If you don't include that option, thetestiso
command will attempt to install FCOS/RHCOS to a non 4k disk (512b sector size) kola testiso -S --scenarios pxe-install,pxe-offline-install --output-dir tmp/kola-metal
You can also pass some scenarios, in case you do not want to run all of themcosa kola testiso --qemu-native-4k
(This is an example testing the live ISO build for a 4k sectors disk. This tests all of the scenarios.)
Example output:
kola -p qemu-unpriv --output-dir tmp/kola testiso -P --qemu-native-4k
Testing scenarios: [iso-offline-install iso-live-login iso-as-disk miniso-install miniso-install-nm]
Detected development build; disabling signature verification
Successfully tested scenario iso-offline-install for 35.20220217.dev.0 on uefi (metal4k)
Successfully tested scenario iso-live-login for 35.20220217.dev.0 on uefi (metal4k)
Successfully tested scenario iso-as-disk for 35.20220217.dev.0 on uefi (metal4k)
Successfully tested scenario miniso-install for 35.20220217.dev.0 on uefi (metal4k)
Successfully tested scenario miniso-install-nm for 35.20220217.dev.0 on uefi (metal4k with NM keyfile)
cosa kola run 'name_of_test'
This is how to run a single test, This is used to help debug specific tests in order to get a better understanding of the bug that's taking place. Once you run this command this test will be added to the tmp directory
cosa kola run basic
This will just run the basic tests
cosa kola run --parallel=3
This will run tests in parallel, 3 at a time.
In order to see the logs for these tests you must enter the tmp/kola/name_of_the_tests
and there you will find the logs (journal and console files, ignition used and so on)
cosa run
This launches the build you created (in this way you can access the image for troubleshooting). Also check the option -c (console).
cosa run -i ignition_path
You can run it passing your Ignition, or the Ignition used in the the test that failed for troubleshooting reasons.
cosa kola run -p aws --aws-ami ami-0431766f2498820b8 --aws-region us-east-1 basic
This will run the basic tests on AWS using ami-0431766f2498820b8
(fedora-coreos-37.20230227.20.2) with default instance type m5.large
. Add --aws-type <t3.micro>
if you want to use custom type. How to create the credentials refer to https://github.com/coreos/coreos-assembler/blob/main/docs/mantle/credentials.md#aws
kola run -p=gce --gce-image=projects/fedora-coreos-cloud/global/images/fedora-coreos-37-20230227-20-2-gcp-x86-64 --gce-json-key=/data/gce.json --gce-project=fedora-coreos-testing basic
This will run the basic tests on GCP using default machine type n1-standard-1
.
gce-image
is in the format ofprojects/<GCP Image Project>/global/images/<GCP Image Name>
, to find related info refer to https://builds.coreos.fedoraproject.org/browser?stream=testing-devel&arch=x86_64.gce-json-key
is using a service account's JSON key for authentication, how to create service account keys refer to https://github.com/coreos/coreos-assembler/blob/main/docs/mantle/credentials.md#gce.gce-project
is meant for testing in the specified project, or it will use the same as<GCP Image Project>
.