feat: support environment templates (#30)

Environment Templates are a part of the spec that allows an Environment
to be defined outside of a Job Template, but still be applied when
running Job(s) created from the Job Template. The CLI doesn't support
these yet.

This commit teaches the CLI about environment templates:
1. In the 'check' command, to verify a template syntax;
2. In the 'schema' command, to generate json schema for
   EnvironmentTemplates; and
3. In the 'run' command, to apply Environments when running a Job
   locally.

The implementation of this also applies a few small cleanups and
refactors to make the implementation more straightforward.

Signed-off-by: Daniel Neilson <[email protected]>

README.md

@@ -1,22 +1,51 @@
-## Open Job Description CLI
+## Open Job Description - CLI
 
-This CLI enables interactive support for writing Job templates using the Open Job Description specification.
+Open Job Description (OpenJD) is a flexible open specification for defining render jobs which are portable
+between studios and render solutions. This package provides a command-line interface that can be used
+to: Verifiy that OpenJD templates are syntactically correct; Run OpenJD jobs locally; and more.
+
+For more information about Open Job Description and our goals with it, please see the
+Open Job Description [Wiki on GitHub](https://github.com/OpenJobDescription/openjd-specifications/wiki).
+
+## Compatibility
+
+This library requires:
+
+1. Python 3.9 or higher;
+2. Linux, MacOS, or Windows operating system;
+3. On Linux/MacOS:
+    * `sudo`
+4. On Windows:
+    * PowerShell 7+
+
+## Versioning
+
+This package's version follows [Semantic Versioning 2.0](https://semver.org/).
+
+1. The MAJOR version is currently 0.
+2. The MINOR version is incremented when backwards incompatible changes are introduced to the public API.
+3. The PATCH version is incremented when bug fixes or backwards compatible changes are introduced to the public API.
 
 ## Commands
 
+### Getting Help
+
+The main `openjd` command and all subcommands support a `--help` option to display
+information on how to use the command.
+
 ### `check`
 Reports any syntax errors that appear in the schema of a Job Template file.
 
 #### Arguments
 
 |Name|Type|Required|Description|Example|
 |---|---|---|---|---|
-|`path`|path|yes|A path leading to a Job template file, or a bundle containing a `.template.json`/`.template.yaml`/`.template.yml` file.|`/path/to/job.template.json`|
+|`path`|path|yes|A path leading to a Job or Environment template file.|`/path/to/template.json`|
 |`--output`|string|no|How to display the results of the command. Allowed values are `human-readable` (default), `json`, and `yaml`.|`--output json`, `--output yaml`|
 
 #### Example
 ```sh
-$ openjd-cli check /path/to/job.template.json
+$ openjd check /path/to/job.template.json
 
 Template at '/path/to/job.template.json' passes validation checks!
 ```
@@ -28,7 +57,7 @@ Displays summary information about a sample Job or a Step therein. The user may
 
 |Name|Type|Required|Description|Example|
 |---|---|---|---|---|
-|`path`|path|yes|A path leading to a Job template file, or a bundle containing a `.template.json`/`.template.yaml`/`.template.yml` file.|`/path/to/job.template.json`|
+|`path`|path|yes|A path leading to a Job template file.|`/path/to/job.template.json`|
 |`--job-param`, `-p`|string, path|no|A key-value pair representing a parameter in the template and the value to use for it, provided as a string or a path to a JSON/YAML document prefixed with 'file://'. Can be specified multiple times.|`--job-param MyParam=5`, `-p file://parameter_file.json`|
 |`--step-name`|string|no|The name of the Step to summarize.|`--step-name Step1`|
 |`--output`|string|no|How to display the results of the command. Allowed values are `human-readable` (default), `json`, and `yaml`.|`--output json`, `--output yaml`|
@@ -63,14 +92,20 @@ Total environments: 0
 ```
 
 ### `run`
-Runs a sample Session on the user’s local machine. The user should provide the Step to run Tasks from, along with any desired Job or Task parameters.
+
+Given a Job Template, Job Parameters, and optional Environment Templates this will run a set of the Tasks
+from the constructed Job locally within an OpenJD Sesssion.
+
+Please see [How Jobs Are Run](https://github.com/OpenJobDescription/openjd-specifications/wiki/How-Jobs-Are-Run) for
+details on how Open Job Description's Jobs are run within Sessions.
 
 #### Arguments
 
 |Name|Type|Required|Description|Example|
 |---|---|---|---|---|
-|`path`|path|yes|A path leading to a Job template file, or a bundle containing a `.template.json`/`.template.yaml`/`.template.yml` file.|`/path/to/job.template.json`|
+|`path`|path|yes|A path leading to a Job template file.|`/path/to/job.template.json`|
 |`--step-name`|string|yes|The name of the Step to run in a local Session.|`--step-name Step1`|
+|`--environment`|paths|no|Path to a file containing Environment Template definitions. Can be provided multiple times.|`--environment /path/to/env.template1.json --environment /path/to/env.template2.yaml`|
 |`--job-param`, `-p`|string, path|no|A key-value pair representing a parameter in the template and the value to use for it, provided as a string or a path to a JSON/YAML document prefixed with 'file://'. Can be specified multiple times.|`--job-param MyParam=5`, `-p file://parameter_file.json`|
 |`--task-params`, `-tp`|string, path|no|A list of key-value pairs representing a Task parameter set for the Step, provided as a string or a path to a JSON/YAML document prefixed with 'file://'. If present, the Session will run one Task per parameter set supplied with `--task-params`. Can be specified multiple times.|`--task-params PingCount=20 PingDelay=30`, `-tp file://parameter_set_file.json`|
 |`--maximum-tasks`|integer|no|A maximum number of Tasks to run from this Step. Unless present, the Session will run all Tasks defined in the Step's parameter space, or one Task per `--task-params` argument.|`--maximum-tasks 5`|

pyproject.toml

@@ -8,9 +8,26 @@ dynamic = ["version"]
 readme = "README.md"
 license = "Apache-2.0"
 requires-python = ">=3.9"
-
+description = "Provides a command-line interface for working with Open Job Description templates."
+# https://pypi.org/classifiers/
+classifiers = [
+  "Development Status :: 5 - Production/Stable",
+  "Programming Language :: Python",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Operating System :: POSIX :: Linux",
+  "Operating System :: Microsoft :: Windows",
+  "Operating System :: MacOS",
+  "License :: OSI Approved :: Apache Software License",
+  "Intended Audience :: Developers",
+  "Intended Audience :: End Users/Desktop"
+]
 dependencies = [
-  "openjd-sessions == 0.3.*"
+  "openjd-sessions == 0.3.*",
+  "openjd-model == 0.2.*"
 ]
 
 [project.scripts]
@@ -142,3 +159,6 @@ source = [
 [tool.coverage.report]
 show_missing = true
 fail_under = 92
+omit= [
+  "src/openjd/cli/_version.py"
+]

src/openjd/cli/_check/_check_command.py

@@ -1,6 +1,12 @@
 # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 
 from argparse import Namespace
+from openjd.model import (
+    DecodeValidationError,
+    SchemaVersion,
+    decode_job_template,
+    decode_environment_template,
+)
 
 from .._common import read_template, OpenJDCliResult, print_cli_result
 
@@ -11,10 +17,36 @@ def do_check(args: Namespace) -> OpenJDCliResult:
 
     try:
         # Raises: RuntimeError
-        filepath, _ = read_template(args)
+        template_object = read_template(args.path)
+
+        # Raises: KeyError
+        document_version = template_object["specificationVersion"]
+
+        # Raises: ValueError
+        template_version = SchemaVersion(document_version)
+
+        # Raises: DecodeValidationError
+        if SchemaVersion.is_job_template(template_version):
+            decode_job_template(template=template_object)
+        elif SchemaVersion.is_environment_template(template_version):
+            decode_environment_template(template=template_object)
+        else:
+            return OpenJDCliResult(
+                status="error",
+                message=f"Unknown template 'specificationVersion' ({document_version}).",
+            )
+
+    except KeyError:
+        return OpenJDCliResult(
+            status="error", message="ERROR: Missing field 'specificationVersion'"
+        )
     except RuntimeError as exc:
         return OpenJDCliResult(status="error", message=f"ERROR: {str(exc)}")
+    except DecodeValidationError as exc:
+        return OpenJDCliResult(
+            status="error", message=f"ERROR: '{str(args.path)}' failed checks: {str(exc)}"
+        )
 
     return OpenJDCliResult(
-        status="success", message=f"Template at '{str(filepath)}' passes validation checks."
+        status="success", message=f"Template at '{str(args.path)}' passes validation checks."
     )

src/openjd/cli/_common/__init__.py

@@ -14,14 +14,21 @@
     get_job_params,
     get_task_params,
 )
-from ._validation_utils import get_doc_type, read_template
+from ._validation_utils import (
+    get_doc_type,
+    read_template,
+    read_job_template,
+    read_environment_template,
+)
 from openjd.model import DecodeValidationError, Job
 
 __all__ = [
     "get_doc_type",
     "get_job_params",
     "get_task_params",
     "read_template",
+    "read_job_template",
+    "read_environment_template",
     "validate_task_parameters",
 ]
 
@@ -61,7 +68,7 @@ def add_common_arguments(
             "path",
             type=Path,
             action="store",
-            help="The path to the template file or Job Bundle to use.",
+            help="The path to the template file.",
         )
     if CommonArgument.JOB_PARAMS in common_arg_options:
         parser.add_argument(
@@ -101,12 +108,12 @@ def add(self, name: str, description: str, **kwargs) -> ArgumentParser:
 def generate_job(args: Namespace) -> Job:
     try:
         # Raises: RuntimeError, DecodeValidationError
-        template_file, template = read_template(args)
+        template = read_job_template(args.path)
         # Raises: RuntimeError
         return job_from_template(
             template,
             args.job_params if args.job_params else None,
-            Path(os.path.abspath(template_file.parent)),
+            Path(os.path.abspath(args.path.parent)),
             Path(os.getcwd()),
         )
     except RuntimeError as rte:

src/openjd/cli/_common/_validation_utils.py

@@ -1,14 +1,16 @@
 # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 
-from argparse import Namespace
+from typing import Any
 from pathlib import Path
 
 from openjd.model import (
     DecodeValidationError,
     DocumentType,
+    EnvironmentTemplate,
     JobTemplate,
-    decode_template,
     document_string_to_object,
+    decode_environment_template,
+    decode_job_template,
 )
 
 
@@ -20,49 +22,60 @@ def get_doc_type(filepath: Path) -> DocumentType:
     raise RuntimeError(f"'{str(filepath)}' is not JSON or YAML.")
 
 
-def _find_template_in_directory(bundle_dir: Path) -> Path:
-    """Search a directory for a Job Template file,
-    stopping at the first instance of a `template.json` or `template.yaml` file in the top level."""
-    for file in bundle_dir.glob("*template.*"):
-        if file.is_file() and file.suffix.lower() in (".json", ".yaml", ".yml"):
-            return file
-
-    raise RuntimeError(
-        f"Couldn't find 'template.json' or 'template.yaml' in the folder '{str(bundle_dir)}'."
-    )
-
-
-def read_template(args: Namespace) -> tuple[Path, JobTemplate]:
+def read_template(template_file: Path) -> dict[str, Any]:
     """Open a JSON or YAML-formatted file and attempt to parse it into a JobTemplate object.
     Raises a RuntimeError if the file doesn't exist or can't be opened, and raises a
     DecodeValidationError if its contents can't be parsed into a valid JobTemplate.
     """
 
-    if not args.path.exists():
-        raise RuntimeError(f"'{str(args.path)}' does not exist.")
+    if not template_file.exists():
+        raise RuntimeError(f"'{str(template_file)}' does not exist.")
 
-    if args.path.is_file():
+    if template_file.is_file():
         # Raises: RuntimeError
-        filepath = args.path
-        filetype = get_doc_type(filepath)
-    elif args.path.is_dir():
-        # Raises: RuntimeError
-        filepath = _find_template_in_directory(args.path)
-        filetype = get_doc_type(filepath)
+        filetype = get_doc_type(template_file)
     else:
-        raise RuntimeError(f"'{str(args.path)}' is not a file or directory.")
+        raise RuntimeError(f"'{str(template_file)}' is not a file.")
 
     try:
-        template_string = filepath.read_text(encoding="utf-8")
+        template_string = template_file.read_text(encoding="utf-8")
     except OSError as exc:
-        raise RuntimeError(f"Could not open file '{str(filepath)}': {str(exc)}")
+        raise RuntimeError(f"Could not open file '{str(template_file)}': {str(exc)}")
 
     try:
+        # Raises: DecodeValidationError
         template_object = document_string_to_object(
             document=template_string, document_type=filetype
         )
-        template = decode_template(template=template_object)
     except DecodeValidationError as exc:
-        raise RuntimeError(f"'{str(filepath)}' failed checks: {str(exc)}")
+        raise RuntimeError(f"'{str(template_file)}' failed checks: {str(exc)}")
+
+    return template_object
+
+
+def read_job_template(template_file: Path) -> JobTemplate:
+    """Open a JSON or YAML-formatted file and attempt to parse it into a JobTemplate object.
+    Raises a RuntimeError if the file doesn't exist or can't be opened, and raises a
+    DecodeValidationError if its contents can't be parsed into a valid JobTemplate.
+    """
+    # Raises RuntimeError
+    template_object = read_template(template_file)
+
+    # Raises: DecodeValidationError
+    template = decode_job_template(template=template_object)
+
+    return template
+
+
+def read_environment_template(template_file: Path) -> EnvironmentTemplate:
+    """Open a JSON or YAML-formatted file and attempt to parse it into an EnvironmentTemplate object.
+    Raises a RuntimeError if the file doesn't exist or can't be opened, and raises a
+    DecodeValidationError if its contents can't be parsed into a valid EnvironmentTemplate.
+    """
+    # Raises RuntimeError
+    template_object = read_template(template_file)
+
+    # Raises: DecodeValidationError
+    template = decode_environment_template(template=template_object)
 
-    return filepath, template
+    return template

src/openjd/cli/_run/_local_session/_session_manager.py

@@ -10,6 +10,7 @@
 from ._actions import EnterEnvironmentAction, ExitEnvironmentAction, RunTaskAction, SessionAction
 from ._logs import LocalSessionLogHandler, LogEntry
 from openjd.model import (
+    EnvironmentTemplate,
     Job,
     ParameterValue,
     ParameterValueType,
@@ -49,6 +50,7 @@ class LocalSession:
     _current_action: Optional[SessionAction]
     _action_ended: Event
     _path_mapping_rules: Optional[list[PathMappingRule]]
+    _environments: Optional[list[EnvironmentTemplate]]
     _log_handler: LocalSessionLogHandler
     _cleanup_called: bool
 
@@ -58,13 +60,15 @@ def __init__(
         job: Job,
         session_id: str,
         path_mapping_rules: Optional[list[PathMappingRule]] = None,
+        environments: Optional[list[EnvironmentTemplate]] = None,
         should_print_logs: bool = True,
     ):
         self.session_id = session_id
         self.ended = Event()
         self._action_ended = Event()
         self._job = job
         self._path_mapping_rules = path_mapping_rules
+        self._environments = environments
 
         # Evaluate Job parameters, if applicable
         if job.parameters:
@@ -145,6 +149,11 @@ def initialize(
         self.ended.clear()
 
         session_environment_ids: list[str] = []
+        # Enqueue "Enter Environment" actions for the given environments
+        if self._environments:
+            envs = [environ.environment for environ in self._environments]
+            session_environment_ids += self._add_environments(envs)
+
         # Enqueue "Enter Environment" actions for root level environments
         if self._job.jobEnvironments:
             session_environment_ids += self._add_environments(self._job.jobEnvironments)