Add basic IOPs check

[micahlee]

Desired Outcome

The desired outcome of this PR is to include disk performance checks in Conjur preflight. Namely this includes:

• Read, write, and sync latency
• Read and write operations per second (IOPs)

This is to help A) begin collecting better information from support cases related to disk performance, and B) provide a mechanism to warn on insufficient performance before attempting to run an auto-failover Conjur cluster.

Implemented Changes

This PR is organized into discrete commits and is best reviewed by-commit. This includes:

• A commit for adding debug logs to the CLI
• A commit to improve the testing of some existing code
• A commit to add the disk performance checks and tests for them.

Here is an example output of the new disk performance checks:

========================================
Conjur Enterprise Preflight Qualification
Version: 0.2.0-6835dc5 (Build 31)
========================================

...

Disk
----
...
INFO - FIO - read iops (/): 878.84 (Min: 626, Max: 1008, StdDev: 24.33)
INFO - FIO - write iops (/): 914.83 (Min: 674, Max: 1080, StdDev: 24.87)
INFO - FIO - read latency (99%, /): 0.00 ms
INFO - FIO - write latency (99%, /): 0.00 ms
INFO - FIO - sync latency (99%, /): 1.22 ms
...

Additional example outputs are available in the artifacts for each Jenkins build.

Connected Issue/Story

CyberArk internal issue ID: ONYX-29736

Definition of Done

At least 1 todo must be completed in the sections below for the PR to be
merged.

Changelog

• The CHANGELOG has been updated, or
• This PR does not include user-facing changes and doesn't require a
  CHANGELOG update

Test coverage

• This PR includes new unit and integration tests to go with the code
  changes, or
• The changes in this PR do not require tests

Documentation

• Docs (e.g. READMEs) were updated in this PR
• A follow-up issue to update official docs has been filed here: ONYX-31672
• This PR does not require updating any documentation

Behavior

• This PR changes product behavior and has been reviewed by a PO, or
• These changes are part of a larger initiative that will be reviewed later, or
• No behavior was changed with this PR

Security

• Security architect has reviewed the changes in this PR,
• These changes are part of a larger initiative with a separate security review, or
• There are no security aspects to these changes

[imheresamir]

LGTM

[imheresamir]

Just curious, did we investigate any Go libraries for the FIO disk/iops checks? So we could call a function in a library instead of delegating the checks to external executables?

[micahlee]

Just curious, did we investigate any Go libraries for the FIO disk/iops checks? So we could call a function in a library instead of delegating the checks to external executables?

Thanks @imheresamir,

Yes, an there's not a lot out there. I did consider using https://github.com/cunnie/gobonniego, which would have covered IOPs but not latency. There are some latency libraries, but I wasn't able to find a license compatible one (e.g. https://github.com/koct9i/ioping is GPL).

The main reason to use fio, though, is that job parameters to evaluate performance for etcd are already well documented:

• https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-fast-enough-for-etcd
• https://www.suse.com/support/kb/doc/?id=000020100

So this gives us more assurance that what we're capturing matches what we need to evaluate suitability for auto-failover.

That said, if we do learn about a library solution that would work for us, I am not opposed to finding a way to push toward that instead.