CLI Tools Testing Use Case

NGS Use Case: Testing Command Line Tools

Run your command line tool(s) one step at a time, checking the output. Exit codes are checked automatically and exceptions are thrown whenever an exit code is "bad", failing the tests. (Parallel testing is work in progress).

• See also all use cases

Pre-flight checks

Before running your tests, you might want to assert certain assumptions about the environment.

Check that Environment Variable is Unset

if 'BEAME_DIR' in ENV {
    throw Error("BEAME_DIR environment variable must be unset for tests")
}

# or shorter alternative
'BEAME_DIR' in ENV throws Error("BEAME_DIR environment variable must be unset for tests")

Output would be the following message or nothing:

...
+--------------------------------------------------------+
| BEAME_DIR environment variable must be unset for tests |
+--------------------------------------------------------+

Exit code will be 240 (exception) in case of the message above.

Check that Environment Variable is Set

ENV.MY_VAR

Output would be the following message or nothing:

...
+-------------------------------------------+
| Environment variable 'MY_VAR' is not set. |
+-------------------------------------------+

Exit code will be 240 (exception) in case of the message above.

Check that a File Exists

Here the file is the CLI tool to be tested. Can't test if it's not there.

BEAME_BIN = '../../src/cli/beame.js'
if not(File(BEAME_BIN)) {
   die("Could not find beame binary. Please make sure you are running this script from the tests/cli_tests directory")
}

Output would be the following message or nothing:

...
+-------------------------------------...
| Could not find beame binary. Please ...
+-------------------------------------...

Exit code will be 1 in case of the message above.

Run a Command, Parse JSON Output

token = ``$BEAME_BIN token create --fqdn $BEAME_AUTH_FQDN --data NONE --format json``

After execution of the above command, token would be a data structure parsed from the JSON that $BEAME_BIN external program returned.

Assertions

WIP

Assertion (for our purposes) is a claim about something that is expected to be true. Examples:

• The data should be a Hash (aka Map, aka dict).
• The data should be an Hash with at least keys k1 and k2.
• The data should be a string that contains substring foo.

Typically, one would run a command and then analyze the output to see whether it's correct.

This section describes the assert_*(...) family of methods. Typical use would be to run a command and assert things about the output. Example:

test("Delete DNS record") with {
	deleted_fqdn = ``$BEAME_BIN creds deleteDns --fqdn $fqdn --format json``
	assert_eq(deleted_fqdn, fqdn, "Deleted fqdn matched to requested")
}

• assert_string(token, "Token is a string") - assert token is a string
• assert_hash(registration, "Registration must be hash")
• assert_resolvable(fqdn, check={A == [ip]}) - assert fqdn is resolvable and the result is exactly one IP, ip. {A == [ip]} is a callback function roughly equivalent in this case to F(resolve_results:Arr) resolve_results==[ip]. Note that assert_resolvable has built-in retry... because we all know DNS updates immediately :)
• assert_eq(deleted_fqdn, fqdn, "Deleted fqdn matched to requested") - mmm important :)

Beyond Asserts

Some situations will not be covered by existing assert functions. (The following example could have used assert_eq() but we either forgot it existed or it was added later.) In that case you have the option to "manually" fail the test:

test("Decrypt data / json", {
	dec = ``$BEAME_BIN creds decrypt --encryptedData $encrypted_data``
	dec != TEST_DATA throws TestFail("Decrypted data is not the same as the original data")
})