6. Pipeline Python API

This guide documents how to use Jarvis within a python script. To this point, we have demonstrated the Jarvis CLI. However, the Python API can be used for building more complex benchmarks.

6.1. Importing Jarvis CD

from jarvis_cd.basic.pkg import Pipeline

6.2. Creating a Pipeline

To create a pipeline and save the environment for the pipeline:

USAGE:

pipeline = Pipeline().create(pipeline_id).build_env().save()

For example:

pipeline = Pipeline().create('gs-hermes').build_env().save()

NOTE: create() will not override any data if the pipeline already exists.

6.3. Loading an Existing Pipeline

USAGE:

pipeline = Pipeline().load(pipeline_id=None)

The following will load the currently-focused pipeline:

pipeline = Pipeline().load()

The following will load the pipeline with a particular name

pipeline = Pipeline().load('gs-hermes')

6.5. Append Pkgs to a Pipeline

USAGE:

pipeline.append(pkg_type, pkg_id=None, do_configure=True, **kwargs)
"""
Create and append a pkg to the pipeline

:param pkg_type: The type of pkg to create (e.g., OrangeFS)
:param pkg_id: Semantic name of the pkg to create
:param do_configure: Whether to configure while appending
:param kwargs: Any parameters the user want to configure in the pkg
:return: self
"""

The following will add Hermes to the pipeline with a sleep of 10

pipeline.append('hermes', 'sleep'=10)
pipeline.append('hermes_mpiio')
pipeline.append('gray_scott')
pipeline.save()

6.6. Configure a Pkg in the Pipeline

USAGE:

pkg = pipeline.get_pkg(pkg_id)
pkg.configure(**kwargs)

For example:

pkg = pipline.get_pkg('hermes')
pkg.configure(sleep=5).save()

6.7. Unlink / Remove Pkgs from a Pipeline

Unlink will simply remove the program from the Jarvis config, but not destroy its contents. Unlinked pkgs can be re-linked using append without losing the configuration data.

Remove ereases the pkg from the filesystem entirely.

USAGE:

pipeline.remove(pkg_id).save()
pipeline.unlink(pkg_id).save()

For example:

pipeline.remove('hermes').save()

6.8. Run a Pipeline

To run the Pipeline end-to-end:

pipeline.run()

6.9. Destroy a Pipeline

To destroy a Pipeline:

pipeline.destroy()