Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[testlib] Add fs mount options check to hpctstlib #2958

Merged
merged 11 commits into from
Oct 26, 2023
65 changes: 37 additions & 28 deletions docs/hpctestlib.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,34 @@ ReFrame Test Library (experimental)
This is a collection of generic tests that you can either run out-of-the-box by specializing them for your system using the :option:`-S` option or create your site-specific tests by building upon them.


Data Analytics
==============

.. automodule:: hpctestlib.data_analytics.spark.spark_checks
:members:
:show-inheritance:


Interactive Computing
=====================

.. automodule:: hpctestlib.interactive.jupyter.ipcmagic
:members:
:show-inheritance:


Machine Learning
================

.. automodule:: hpctestlib.ml.tensorflow.horovod
:members:
:show-inheritance:

.. automodule:: hpctestlib.ml.pytorch.horovod
:members:
:show-inheritance:


Microbenchmarks
===============

Expand Down Expand Up @@ -55,25 +83,6 @@ GPU benchmarks
:show-inheritance:


Scientific Applications
=======================

.. automodule:: hpctestlib.sciapps.amber.nve
:members:
:show-inheritance:

.. automodule:: hpctestlib.sciapps.gromacs.benchmarks
:members:
:show-inheritance:

Data Analytics
==============

.. automodule:: hpctestlib.data_analytics.spark.spark_checks
:members:
:show-inheritance:


Python
======

Expand All @@ -82,21 +91,21 @@ Python
:show-inheritance:


Interactive Computing
=====================
Scientific Applications
=======================

.. automodule:: hpctestlib.interactive.jupyter.ipcmagic
.. automodule:: hpctestlib.sciapps.amber.nve
:members:
:show-inheritance:


Machine Learning
================

.. automodule:: hpctestlib.ml.tensorflow.horovod
.. automodule:: hpctestlib.sciapps.gromacs.benchmarks
:members:
:show-inheritance:

.. automodule:: hpctestlib.ml.pytorch.horovod

System
=======================

.. automodule:: hpctestlib.system.fs.mnt_opts
:members:
:show-inheritance:
110 changes: 110 additions & 0 deletions hpctestlib/system/fs/mnt_opts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Copyright 2016-2023 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
# ReFrame Project Developers. See the top-level LICENSE file for details.
#
# SPDX-License-Identifier: BSD-3-Clause

import os

from string import Template

import reframe as rfm
import reframe.utility.sanity as sn
import reframe.utility.typecheck as typ


@rfm.simple_test
class filesystem_options_check(rfm.RunOnlyRegressionTest):
'''filesystem mount options check

Check if the mounted filesystems have been configured appropriately
based on their type
'''

#: Reference mount options
#:
#: :type: `Dict[str, str]`. The key should be the file system type.
#: and the value should be a string with mount options.
#: E.g., {'xfs: 'nosuid,logbsize=32k'}
#: :default: ``{}``
fs_ref_opts = variable(typ.Dict, value={}, loggable=True)
vkarak marked this conversation as resolved.
Show resolved Hide resolved
fs_ref_opts = required
vkarak marked this conversation as resolved.
Show resolved Hide resolved

#: Fail if the test finds a filesystem type that is not in the
#: reference dictionary
#:
#: :type: `Bool`.
#: :value: ``False``
fail_unknown_fs = variable(typ.Bool, value=False, loggable=True)

executable = 'cat'
executable_opts = ['/proc/mounts']
tags = {'system', 'fs'}

@run_before('sanity')
def process_system_fs_opts(self):
self.filesystem = []
stdout = os.path.join(self.stagedir, sn.evaluate(self.stdout))
with open(stdout, 'r') as fp:
for line in fp:
_, mnt_point, type, options, _, _ = line.split()
self.filesystem.append((mnt_point, type, options))

@run_before('sanity')
def print_test_variables_to_output(self):
'''
Write the reference mount point options used by the test
at the time of execution.
victorusu marked this conversation as resolved.
Show resolved Hide resolved
'''
stdout = os.path.join(self.stagedir, sn.evaluate(self.stdout))
with open(stdout, 'a') as fp:
fp.write('\n---- Reference mount options ----\n')
for mnt_type, options in self.fs_ref_opts.items():
fp.write(f'{mnt_type} {options}\n')

def explode_opts_str(self, opts_str):
result=dict()
victorusu marked this conversation as resolved.
Show resolved Hide resolved
for opt in opts_str.split(','):
if opt == '':
continue
opt_parts = opt.split('=', maxsplit=2)
victorusu marked this conversation as resolved.
Show resolved Hide resolved
keystr = opt_parts[0]
valstr = opt_parts[1] if len(opt_parts) > 1 else ''
result[keystr] = valstr
return result
victorusu marked this conversation as resolved.
Show resolved Hide resolved

@sanity_function
def assert_mnt_options(self):
msg = Template('Found filesystem type(s) - "$mnt_type" - that are not '
'compatible with this test')
msg1 = Template('The mount point "$mnt_point" of type "$mnt_type" does'
' not have the "$opt" option.')
msg2 = Template('The "$variable" variable value of "$value" does not '
'match the reference "$ref_value" for the "$mnt_point"'
' mount point ("$mnt_type" type)')

errors = []
unsupported_types = set()
for mnt_point, mnt_type, options in self.filesystem:
opts = self.explode_opts_str(options)
if mnt_type not in self.fs_ref_opts:
unsupported_types.add(mnt_type)
continue

ref_opts = self.explode_opts_str(self.fs_ref_opts[mnt_type])
for ref_opt, ref_value in ref_opts.items():
if ref_opt not in opts:
errors.append(msg1.substitute(opt=ref_opt,
mnt_point=mnt_point,
mnt_type=mnt_type))
elif ref_value != opts[ref_opt]:
errors.append(msg2.substitute(value=opts[ref_opt],
ref_value=ref_value,
variable=ref_opt,
mnt_point=mnt_point,
mnt_type=mnt_type))

if self.fail_unknown_fs:
errors.append(msg.substitute(
mnt_type=', '.join(unsupported_types)))

return sn.assert_true(errors == [], msg='\n'.join(errors))