design and implement generic unit tests for all rocks

[ca-scribner]

Context

When updating rocks, it would be useful to have simple unit or sanity tests for our rocks. These are tests that can be executed quickly (either linting the rockcraft.yaml itself, or examining a rock that has already been built). For example, being able to test:

• certain files exist in the image (eg: ["/home/username/my_data", "/bin/my_executable", ...])
• the version follows our agreed upon version nomenclature (?)
• ...
  Out of scope here would be integration tests or anything that actually executes the rock.

We have a few examples of this, like seldon which implements the tests here, but not all rocks use these. Some later charms attempted to simplify the tests a bit, but it could still be done further. We should have these tests consistently applied across our repos.

Our current tests do share some common code, but the tests themselves are defined as code rather than configuration (each rock needs its own test_rock.py file) which adds a lot of boilerplate. We could simplify these to use common test tools and just add configuration via config (for example, replace test_rock.py with assert_file_in_rock.py --file "/home/username/my_data" --file "/bin/my_executable", assert_rock_version_scheme.py, ...)

What needs to get done

1. define what unit/sanity tests we want for all our rocks
2. refactor the tools for this to improve reuse across our repos. The goal here should be as little configuration/code required per repo
3. implement these tests in all our rocks repos

Definition of Done

1. we have documentation that defines what unit/sanity tests all our rocks should have
2. we have implemented these tests in all our rocks repos

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5303.

This message was autogenerated

[ca-scribner]

A really easy mistake in a rockcraft.yaml file is to do:

    override-build:
      command1
      command2

when you mean:

    override-build: |
      command1
      command2

Should we warn for this sort of thing? Same with any override-* block

[ca-scribner]

Another useful test would be a negative file match, for example to assert /x/y/z is NOT in the final rock. This is useful to ensure we do not prime things accidentally, for example some artifact that is used during building but not needed in the final product

[ca-scribner]

This is similar to #783

[orfeas-k]

Another issue with current ("standardised) unit tests we have in ROCKs e.g. here is that we have a fixture which is not used and thus container is not cleaned up after running sanity environment. This fixture was originially designed to be used in that manner but this was dismissed somewhere along the line.

EDIT: A good implementation of this can be seen here.