openQA is an automated test tool that makes it possible to test the whole installation process of an operating system. It is free software released under the GPLv2 license. The source code and documentation are hosted in the os-autoinst organization on GitHub.
This document provides the information needed to install and setup the tool, as well as information useful for everyday administration of the system. It is assumed that the reader is already familiar with the concepts of openQA and has already read the Getting Started Guide, also available at the official repository.
Continue with the next section to get a simple, ready-to-use installation, useful for a single user setup. Else, continue with the more advanced section about Custom installation. For a setup suitable to develop openQA itself, have a look at the Development setup section.
To quickly get a working openQA installation, you can use the openqa-bootstrap script. It essentially automates the steps mentioned in the Custom installation section.
This should work on openSUSE Leap and openSUSE Tumbleweed and will setup openQA on your machine.
zypper in openQA-bootstrap
/usr/share/openqa/script/openqa-bootstrapIf you happen to be using an old Leap 15.0 system which does not already have the openQA-bootstrap RPM in the repo you can simply download the openqa-bootstrap script - it will do the rest for you:
# get root
curl -s https://raw.githubusercontent.com/os-autoinst/openQA/master/script/openqa-bootstrap | bash -xopenQA-bootstrap supports to immediately clone an existing job simply by
supplying openqa-clone-job parameters directly for a quickstart:
/usr/share/openqa/script/openqa-bootstrap --from openqa.opensuse.org 12345 SCHEDULE=tests/boot/boot_to_desktop,tests/x11/kontactThe above command will bootstrap an openQA installation and immediately
afterwards start a local test job clone from a test job from a remote instance
with optional, overridden parameters. More information about
openqa-clone-job can be found in
Cloning existing jobs - openqa-clone-job.
NOTE This method is not available on openSUSE Leap older than version 15.1.
You can also setup a systemd-nspawn container with openQA with the following commands. and you need to have no application listening on port 80 yet because the container will share the host system’s network stack.
zypper in openQA-bootstrap
/usr/share/openqa/script/openqa-bootstrap-container
systemd-run -tM openqa1 /bin/bash # start a shell in the containerKeep in mind that there can be disruptive changes between openQA versions. You need to be sure that the webui and the worker that you are using have the same version number or, at least, are compatible.
For example, the packages distributed with older versions of openSUSE Leap are not compatible with the version on Tumbleweed. And the package distributed with Tumbleweed may not be compatible with the version in the development package.
The easiest way to install openQA is from distribution packages.
-
For openSUSE, packages are available for Leap and Tumbleweed
-
For Fedora, packages are available in the official repositories for Fedora 23 and later.
You can find the development version of openQA in OBS in the openQA:devel repository.
To add the development repository to your system, you can use these commands.
# openSUSE Tumbleweed
zypper ar -p 95 -f 'http://download.opensuse.org/repositories/devel:openQA/openSUSE_Tumbleweed' devel_openQA
# openSUSE Leap
zypper ar -p 95 -f 'http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_$releasever' devel_openQA
zypper ar -p 90 -f 'http://download.opensuse.org/repositories/devel:openQA:Leap:$releasever/openSUSE_Leap_$releasever' devel_openQA_Leap|
Note
|
If you installed openQA from the official repository first, you may need to change the vendor of the dependencies. |
# openSUSE Tumbleweed and Leap
zypper dup --from devel_openQA --allow-vendor-change
# openSUSE Leap
zypper dup --from devel_openQA_Leap --allow-vendor-changeYou can install the main openQA server package using these commands.
# openSUSE
zypper in openQA
# Fedora
dnf install openqa openqa-httpdTo install the openQA worker package use the following.
# openSUSE
zypper in openQA-workerDifferent convenience packages exist for convenience in openSUSE, for example:
openQA-local-db to install the server including the setup of a local
PostgreSQL database or openQA-single-instance which sets up a web UI server,
a web proxy as well as a local worker. Install openQA-client if you only
want to interact with existing, external openQA instances.
For a local instance setup you can simply execute the script:
/usr/share/openqa/script/configure-web-proxyThis will automatically setup a local apache http proxy. Read on for more detailed setup instructions with all the details.
It is required to run openQA behind an http proxy (apache, nginx, etc..). See the
openqa.conf.template config file in /etc/apache2/vhosts.d (openSUSE) or
/etc/httpd/conf.d (Fedora). To make everything work correctly on openSUSE, you
need to enable the 'headers', 'proxy', 'proxy_http', 'proxy_wstunnel' and 'rewrite'
modules using the command 'a2enmod'. This is not necessary on Fedora.
# openSUSE Only
# You can check what modules are enabled by using 'a2enmod -l'
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_wstunnel
a2enmod rewriteFor a basic setup, you can copy openqa.conf.template to openqa.conf and modify the ServerName if required
setting. This will direct all HTTP traffic to openQA.
cp /etc/apache2/vhosts.d/openqa.conf.template /etc/apache2/vhosts.d/openqa.confBy default openQA expects to be run with HTTPS. The openqa-ssl.conf.template
Apache config file is available as a base for creating the Apache config; you
can copy it to openqa-ssl.conf and uncomment any lines you like, then
ensure a key and certificate are installed to the appropriate location
(depending on distribution and whether you uncommented the lines for key and
cert location in the config file). On openSUSE, you should also add SSL to the
APACHE_SERVER_FLAGS so it looks like this in /etc/sysconfig/apache2:
APACHE_SERVER_FLAGS="SSL"If you don’t have a TLS/SSL certificate for your host you must turn HTTPS off.
You can do that in /etc/openqa/openqa.ini:
[openid]
httpsonly = 0Since version 4.5.1512500474.437cc1c7 of openQA, PostgreSQL is used as the database.
To configure access to the database in openQA, edit /etc/openqa/database.ini
and change the settings in the [production] section.
The dsn value format technically depends on the database type and is
documented for PostgreSQL at
DBD::Pg
[production]
dsn = dbi:Pg:dbname=openqa;host=db.example.org
user = openqa
password = somepasswordFor older versions of openQA, you can migrate from SQLite to PostgreSQL according to DB migration from SQLite to PostgreSQL
OpenQA supports two different authentication methods - OpenID (default) and
Fake. See auth section in /etc/openqa/openqa.ini.
[auth]
# method name is case sensitive!
method = OpenID|FakeIndependently of method used, the first user that logs in (if there is no admin yet) will automatically get administrator rights!
By default openQA uses OpenID with opensuse.org as OpenID provider.
OpenID method has its own openid section in /etc/openqa/openqa.ini:
[openid]
## base url for openid provider
provider = https://www.opensuse.org/openid/user/
## enforce redirect back to https
httpsonly = 1OpenQA supports only OpenID version up to 2.0. Newer OpenID-Connect and OAuth is not supported currently.
For development purposes only! Fake authentication bypass any authentication and
automatically allow any login requests as 'Demo user' with administrator privileges
and without password. To ease worker testing, API key and secret is created (or updated)
with validity of one day during login.
You can then use following as /etc/openqa/client.conf:
[localhost]
key = 1234567890ABCDEF
secret = 1234567890ABCDEFIf you switch authentication method from Fake to any other, review your API keys! You may be vulnerable for up to a day until Fake API key expires.
To start openQA and enable it to run on each boot call
systemctl enable --now postgresql
systemctl enable --now openqa-webui
systemctl enable --now openqa-scheduler
# openSUSE
systemctl restart apache2
# Fedora
# for now this is necessary to allow Apache to connect to openQA
setsebool -P httpd_can_network_connect 1
systemctl restart httpdThe openQA web UI should be available on http://localhost/ now. To simply
start openQA without enabling it permanently one can simply use systemctl
start instead.
Workers are services running backends to perform the actual testing. The testing is commonly performed by running virtual machines but depending on the specific backend configuration different options exist. The openQA worker is distributed as a separate package which be installed on multiple machines while still using only one web UI.
# openSUSE
zypper in openQA-worker
# Fedora
dnf install openqa-workerTo allow workers to access your instance, you need to log into openQA as
operator and create a pair of API key and secret. Once you are logged in, in the
top right corner, is the user menu, follow the link 'manage API keys'. Click
the 'create' button to generate key and secret. There is also a script
available for creating an admin user and an API key+secret pair
non-interactively, /usr/share/openqa/script/create_admin, which can be useful
for scripted deployments of openQA. Copy and paste the key and secret into
/etc/openqa/client.conf on the machine(s) where the worker is installed. Make
sure to put in a section reflecting your webserver URL. In the simplest case,
your client.conf may look like this:
[localhost]
key = 1234567890ABCDEF
secret = 1234567890ABCDEFTo start the workers you can use the provided systemd files via:
systemctl start openqa-worker@1This will start worker number one. You can start as many workers as you dare, you just need to supply different 'worker id' (number after @).
You can also run workers manually from command line.
install -d -m 0755 -o _openqa-worker /var/lib/openqa/pool/X
sudo -u _openqa-worker /usr/share/openqa/script/worker --instance XThis will run a worker manually showing you debug output. If you haven’t
installed 'os-autoinst' from packages make sure to pass --isotovideo option
to point to the checkout dir where isotovideo is, not to /usr/lib! Otherwise
it will have trouble finding its perl modules.
If you start openQA workers on a different machine than the web UI host make sure to have synchronized clocks, for example using NTP, to prevent inconsistent test results.
From this point on, you can refer to the Getting Started guide to fetch the tests cases and possibly take a look at Test Developer Guide
Editing needles from web can optionally commit new or changed needles automatically to git. To do so, you need to enable git support by setting
[global]
scm = gitin /etc/openqa/openqa.ini. Once you do so and restart the web interface, openQA will
automatically commit new needles to the git repository.
You may want to add some description to automatic commits coming from the web
UI.
You can do so by setting your configuration in the repository
(/var/lib/os-autoinst/needles/.git/config) to some reasonable defaults such as:
[user]
email = [email protected]
name = openQA web UITo enable automatic pushing of the repo as well, you need to add the following to your openqa.ini:
[scm git]
do_push = yesDepending on your setup, you might need to generate and propagate ssh keys for user 'geekotest' to be able to push.
It might also be useful to rebase first. To enable that, add the remote to get the latest updates from and the branch to rebase against to your openqa.ini:
[scm git]
update_remote = origin
update_branch = origin/masterAutomatic cleanup of old results (see GRU jobs) can sometimes render important tests useless. For example bug report with link to openQA job which no longer exists. Job can be manually marked as important to prevent quick cleanup or referer can be set so when job is accessed from particular web page (for example bugzilla), this job is automatically labeled as linked and treated as important.
List of recognized referers is space separated list configured in
/etc/openqa/openqa.ini:
[global]
recognized_referers = bugzilla.suse.com bugzilla.opensuse.orgDefault behavior for all workers is to use the 'Qemu' backend and connect to
'http://localhost'. If you want to change some of those options, you can do so
in /etc/openqa/workers.ini. For example to point the workers to the FQDN of
your host (needed if test cases need to access files of the host) use the
following setting:
[global]
HOST = http://openqa.example.comOnce you got workers running they should show up in the admin section of openQA in the workers section as 'idle'. When you get so far, you have your own instance of openQA up and running and all that is left is to set up some tests.
There are some additional requirements to get remote worker running. First is to
ensure shared storage between openQA web UI and workers.
Directory /var/lib/openqa/share contains all required data and should be
shared with read-write access across all nodes present in openQA cluster.
This step is intentionally left on system administrator to choose proper shared
storage for her specific needs.
Example of NFS configuration:
NFS server is where openQA web UI is running. Content of /etc/exports
/var/lib/openqa/share *(fsid=0,rw,no_root_squash,sync,no_subtree_check)NFS clients are where openQA workers are running. Run following command:
mount -t nfs openQA-webUI-host:/var/lib/openqa/share /var/lib/openqa/shareYou can configure openQA to send events (new comments, tests finished, …) to an AMQP message bus. The messages consist of a topic and a body. The body contains json encoded info about the event. See amqp_infra.md for more info about the server and the message topic format. There you will find instructions how to configure the AMQP server as well.
To let openQA send messages to an AMQP message bus,
first make sure that the perl-Mojo-RabbitMQ-Client RPM is installed.
Then you will need to configure amqp in /etc/openqa/openqa.ini:
# Configuration for AMQP plugin
[amqp]
heartbeat_timeout = 60
reconnect_timeout = 5
# guest/guest is the default anonymous user/pass for RabbitMQ
url = amqp://guest:guest@localhost:5672/
exchange = pubsub
topic_prefix = suseFor a TLS connection use amqps:// and port 5671.
When there are multiple openQA web interfaces (openQA instances) available a worker can be configured to register and accept jobs from all of them.
Requirements:
-
/etc/openqa/client.confmust contain API keys and secrets to all instances -
Shared storage from all instances must be properly mounted
In the /etc/openqa/workers.ini enter space-separated instance hosts and optionally
configure where the shared storage is mounted. Example:
[global]
HOST = openqa.opensuse.org openqa.fedora.fedoraproject.org
[openqa.opensuse.org]
SHARE_DIRECTORY = /var/lib/openqa/opensuse
[openqa.fedoraproject.org]
SHARE_DIRECTORY = /var/lib/openqa/fedoraConfiguring SHARE_DIRECTORY is not a hard requirement. Worker will try following
directories prior registering with openQA instance:
-
SHARE_DIRECTORY -
/var/lib/openqa/$instance_host -
/var/lib/openqa/share -
/var/lib/openqa -
fail if none of above is available
Once worker registers to openQA instance it checks for available job and starts accepting websockets commands. Worker accepts jobs as they will come in, there is no priority, or other ordering, support at the moment. It is possible to mix local openQA instance with remote instances or use only remote instances.
If your network is slow or you experience long time to load needles you might
want to consider to enable caching in your remote workers. To enable caching,
/var/lib/openqa/cache must exist, and right permissions given to the
'_openqa-worker' user to read everything under this path. If you install
openQA through the repositories, said directory will be created for you.
It is suggested to have the cache and pool directories on the same filesystem
to ensure assets used by tests are available as long as needed. This is
achieved by using hard links, resorting to symlinks in other cases with the
risk of assets being deleted from the cache before tests relying on these
assets end.
Start and enable the Cache Service:
systemctl enable --now openqa-worker-cacheserviceEnable and start the Cache Worker:
systemctl enable --now openqa-worker-cacheservice-minionIn the /etc/openqa/workers.ini
[global]
HOST=http://webui
CACHEDIRECTORY = $cache_location
CACHELIMIT = 50 # GB, default is 50.
CACHEWORKERS = 5 # Number of parallel cache minion workers, defaults to 5
[http://webui]
TESTPOOLSERVER = rsync://yourlocation/testsSetup and run rsync server daemon on HOST machine, in /etc/rsyncd.conf should be:
gid = users
read only = true
use chroot = true
transfer logging = true
log format = %h %o %f %l %b
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
slp refresh = 300
use slp = false
#[Example]
# path = /home/Example
# comment = An Example
# auth users = user
# secrets file = /etc/rsyncd.secrets
[tests]
path = /var/lib/openqa/share/tests
comment = OpenQA Test Distributionsand
systemctl enable --now rsyncdThis will allow the workers to download the assets from the web UI and use them
locally. If TESTPOOLSERVER is set tests and needles will also be cached by the
worker.
Auditing plugin enables openQA administrators to maintain overview about what is happening with the system. Plugin records what event was triggered by whom, when and what the request looked like. Actions done by openQA workers are tracked under user whose API keys are workers using.
Audit log is directly accessible from Admin menu.
Auditing, by default enabled, can be disabled by global configuration option in /etc/openqa/openqa.ini:
[global]
audit_enabled = 0The audit section of /etc/openqa/openqa.ini allows to exclude some events from logging using
a space separated blacklist:
[audit]
blacklist = job_grab job_doneThe audit/storage_duration section of /etc/openqa/openqa.ini allows to set the retention policy for
different audit event types:
[audit/storage_duration]
startup = 10
jobgroup = 365
jobtemplate = 365
table = 365
iso = 60
user = 60
asset = 30
needle = 30
other = 15In this example events of the type startup would be cleaned up after 10 days, events related to
job groups after 365 days and so on. Events which do not fall into one of these categories would be
cleaned after 15 days. By default, cleanup is disabled.
Use systemctl enable --now openqa-enqueue-audit-event-cleanup.timer to schedule the cleanup
automatically every day. It is also possible to trigger the cleanup manually by invoking
/usr/share/openqa/script/openqa minion job -e limit_audit_events.
-
Assets:
-
asset_register asset_delete
-
-
Workers:
-
worker_register command_enqueue
-
-
Jobs:
-
iso_create iso_delete iso_cancel
-
jobtemplate_create jobtemplate_delete
-
job_create job_grab job_delete job_update_result job_done jobs_restart job_restart job_cancel job_duplicate
-
jobgroup_create jobgroup_connect
-
-
Tables:
-
table_create table_update table_delete
-
-
Users:
-
user_new_comment user_update_comment user_delete_comment user_login
-
-
Needles:
-
needle_delete needle_modify
-
Some of these events are very common and may clutter audit database. For this reason job_grab and job_done
events are blacklisted by default.
|
Note
|
Upgrading openQA does not automatically update /etc/openqa/openqa.ini. Review your configuration after upgrade.
|
Tests, needles, assets, results and working directories (a.k.a. "pool directories") are located in certain
subdirectories within /var/lib/openqa. This directory is configurable (see
Customize base directory). Here we assume the default is in place.
Note that the sub directories within /var/lib/openqa must be accessible by the user that runs the openQA web UI
(by default 'geekotest') or by the user that runs the worker/isotovideo (by default '_openqa-worker').
These are the most important sub directories within /var/lib/openqa:
-
dbcontains the web UI’s database lockfile -
imagesis where the web UI stores test screenshots and thumbnails -
testresultsis where the web UI stores test logs and test-generated assets -
webuiis where the web UI stores miscellaneous files -
poolcontains working directories of the workers/isotovideo -
sharecontains directories shared between the web UI and (remote) workers, can be owned by root -
share/factorycontains test assets and temp directory, can be owned by root but sysadmin must create subdirs -
share/factory/isoandshare/factory/iso/fixedcontain ISOs for tests -
share/factory/hddandshare/factory/hdd/fixedcontain hard disk images for tests -
share/factory/repoandshare/factory/repo/fixedcontain repositories for tests -
share/factory/otherandshare/factory/other/fixedcontain miscellaneous test assets (e.g. kernels and initrds) -
share/factory/tmpis used as a temporary directory (openQA will create it if it ownsshare/factory) -
share/testscontains the tests themselves
Each of the asset directories (factory/iso, factory/hdd, factory/repo and
factory/other) may contain a fixed/ subdirectory, and assets of the same
type may be placed in that directory. Placing an asset in the fixed/
subdirectory indicates that it should not be deleted to save space: the GRU
task which removes old assets when the size of all assets for a given job
group is above a specified size will ignore assets in the fixed/
subdirectories.
It also contains several symlinks which are necessary due to various things moving around over the course of openQA’s development. All the symlinks can of course be owned by root:
-
script(symlink to/usr/share/openqa/script/) -
tests(symlink toshare/tests) -
factory(symlink toshare/factory)
It is always best to use the canonical locations, not the compatibility
symlinks - so run scripts from /usr/share/openqa/script, not
/var/lib/openqa/script.
You only need the asset directories for the asset types you will actually use,
e.g. if none of your tests refer to openQA-stored repositories, you will need
no factory/repo directory. The distribution packages may not create all
asset directories, so make sure the ones you need are created if necessary.
Packages will likewise usually not contain any tests; you must create your
own tests, or use existing tests for some distribution or other piece of
software.
The worker needs to own /var/lib/openqa/pool/$INSTANCE, e.g.
-
/var/lib/openqa/pool/1 -
/var/lib/openqa/pool/2 -
… - add more if you have more worker instances
You can also give the whole pool directory to the _openqa-worker user and let
the workers create their own instance directories.
-
the "base directory"
-
by default
/var/lib -
configurable via environment variable
OPENQA_BASEDIR -
referred as
$basedirwithin openQA
-
-
the "project directory"
-
defined as
$basedir/openqa, by default/var/lib/openqa -
referred as
$prjdirwithin openQA
-
-
the "share directory": contains directories shared between web UI and (remote) workers
-
defined as
$prjdir/share, by default/var/lib/openqa/share -
referred as
$sharedirwithin openQA
-
-
the "test case directory": contains a test distribution
-
by default
$sharedir/tests/$distrior$sharedir/tests/$distri-$version -
configurable via the test variable
CASEDIR(see backend variables documentation) -
this default is provided by openQA; when starting isotovideo manually the
CASEDIRvariable must be initialized by hand -
might contain the sub directory
libfor placing Perl modules used by the tests
-
-
the "product directory": contains the test schedule (
main.pm) for a certain product within a test distribution-
by default identical to the "test case directory"
-
usually a directory
products/$distriwithin the "test case directory" -
configurable via the test variable
PRODUCTDIR(see backend variables documentation)
-
-
the "needles directory": contains reference images for a certain product within a test distribution
-
by default
$PRODUCTDIR/needles -
configurable via the test variable
NEEDLES_DIR(see backend variables documentation)
-
-
Setting the test variables has only an influence on os-autoinst. The web UI on the other hand always relies on the directory structure described above. For the exact details how these paths are computed by the web UI have a look at
lib/OpenQA/Utils.pm. -
When enabling the worker cache parts of the usual "share directory" are located in the specified cache directory on the worker host.
-
make sure you have a machine with kvm support
-
make sure
kvm_intelorkvm_amdmodules are loaded -
make sure you do have virtualization enabled in BIOS
-
make sure the '_openqa-worker' user can access
/dev/kvm -
make sure you are not already running other hypervisors such as VirtualBox
-
when running inside a vm make sure nested virtualization is enabled (pass nested=1 to your kvm module)
www.opensuse.org’s openid provider may have trouble with IPv6. openQA shows a message like this:
no_identity_server: Could not determine ID provider from URL.
To avoid that switch off IPv6 or add a special route that prevents the system from trying to use IPv6 with www.opensuse.org:
ip -6 r a to unreachable 2620:113:8044:66:130:57:66:6/128