Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat!: update to the 2023-09 job template schema
Browse files Browse the repository at this point in the history
The job template schema is being updated for service release. With it,
the implementation library is being updated in a breaking way and so
this code needs to be updated to that revision. This includes a rename
of the job template's specification name from the codename openjobio
to Open Job Description.

Signed-off-by: Daniel Neilson <neilsd@amazon.com>
Signed-off-by: Mark Wiebe <markw@amazon.com>
Daniel Neilson committed Sep 11, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
1 parent 9ecf26c commit 5fb06ce
Showing 14 changed files with 85 additions and 83 deletions.
10 changes: 5 additions & 5 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
@@ -22,11 +22,11 @@ WARNING: This workflow installs additional Python packages into your Maya's pyth

1. Create a development location within which to do your git checkouts. For example `~/deadline-clients`.
Clone packages from this directory with commands like
`git clone git@github.com:casillas2/deadline-cloud-for-maya.git`. You'll also want the `deadline-cloud` and `openjobio` repos.
`git clone git@github.com:casillas2/deadline-cloud-for-maya.git`. You'll also want the `deadline-cloud` and `openjd` repos.
2. Switch to your Maya directory, like `cd "C:\Program Files\Autodesk\Maya2023"`.
3. Run `.\mayapy -m pip install -e C:\Users\<username>\deadline-clients\deadline-cloud` to install the Amazon Deadline Cloud Client
Library in edit mode.
4. Run `.\mayapy -m pip install -e C:\Users\<username>\deadline-clients\openjobio` to install the OpenJobIO
4. Run `.\mayapy -m pip install -e C:\Users\<username>\deadline-clients\openjd` to install the Open Job Description
Library in edit mode.
5. Run `.\mayapy -m pip install -e C:\Users\<username>\deadline-clients\deadline-cloud-for-maya` to install the Maya Submtiter
in edit mode.
@@ -48,21 +48,21 @@ your build of the adaptor for the one in the service.

1. Use the development location from the Submitter Development Workflow.
Make sure you're running Maya with `set DEADLINE_ENABLE_DEVELOPER_OPTIONS=true` enabled.
2. Build wheels for `openjobio`, `deadline` and `deadline-cloud-for-maya`.
2. Build wheels for `openjd`, `deadline` and `deadline-cloud-for-maya`.
```
# If you don't have the build package installed already
$ pip install build
...
$ mkdir wheels; \
rm wheels/*; \
for dir in ../openjobio ../deadline-cloud ../deadline-cloud-for-maya; \
for dir in ../openjd ../deadline-cloud ../deadline-cloud-for-maya; \
do python -m build --wheel --outdir ./wheels --skip-dependency-check $dir; \
done
...
$ ls ./wheels
deadline_cloud_for_maya-<version>-py3-none-any.whl
deadline-<version>-py3-none-any.whl
openjobio-<version>-py3-none-any.whl
openjd-<version>-py3-none-any.whl
```
3. Open the Maya integrated submitter, and in the Job-Specific Settings tab, enable the option 'Include Adaptor Wheels'. This
option is only visible when the environment variable `DEADLINE_ENABLE_DEVELOPER_OPTIONS` is set to `true`.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@ This package has two active branches:
- `mainline` -- For active development. This branch is not intended to be consumed by other packages. Any commit to this branch may break APIs, dependencies, and so on, and thus break any consumer without notice.
- `release` -- The official release of the package intended for consumers. Any breaking releases will be accompanied with an increase to this package's interface version.

The deadline.maya_adaptor package is an adaptor that renders maya scenes through MayaPy. It uses the openjobio adaptor_runtime and supports job stickiness.
The deadline.maya_adaptor package is an adaptor that renders maya scenes through MayaPy. It uses the Open Job Description adaptor_runtime and supports job stickiness.

## Development

4 changes: 2 additions & 2 deletions depsBundle.py
Original file line number Diff line number Diff line change
@@ -35,8 +35,8 @@ def _get_dependencies(pyproject_dict: dict[str, Any]) -> list[str]:
raise Exception("pyproject.toml is missing dependencies section")

dependencies = pyproject_dict["project"]["dependencies"]
deps_noopenjobio = filter(lambda dep: not dep.startswith("openjobio"), dependencies)
return list(map(lambda dep: dep.replace(" ", ""), deps_noopenjobio))
deps_noopenjd = filter(lambda dep: not dep.startswith("openjd"), dependencies)
return list(map(lambda dep: dep.replace(" ", ""), deps_noopenjd))


def _get_package_version_regex(package: str) -> re.Pattern:
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
specificationVersion: '2022-09-01'
specificationVersion: jobtemplate-2023-09
name: cube.ma
parameters:
parameterDefinitions:
- name: MayaSceneFile
type: PATH
objectType: FILE
@@ -101,15 +101,15 @@ parameters:
steps:
- name: masterLayer
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.Frames}}'
- name: Camera
type: STRING
range:
- persp
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
24 changes: 12 additions & 12 deletions job_bundle_output_tests/layers/expected_job_bundle/template.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
specificationVersion: '2022-09-01'
specificationVersion: jobtemplate-2023-09
name: layers.ma
parameters:
parameterDefinitions:
- name: MayaSceneFile
type: PATH
objectType: FILE
@@ -236,15 +236,15 @@ parameters:
steps:
- name: layerDefaultFrames
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.layerDefaultFramesFrames}}'
- name: Camera
type: STRING
range:
- camera1
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
@@ -309,7 +309,7 @@ steps:
mode: NOTIFY_THEN_TERMINATE
- name: layerFrameRange1
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.layerFrameRange1Frames}}'
@@ -318,7 +318,7 @@ steps:
range:
- camera1
- persp
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
@@ -382,7 +382,7 @@ steps:
mode: NOTIFY_THEN_TERMINATE
- name: masterLayer
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.masterLayerFrames}}'
@@ -391,7 +391,7 @@ steps:
range:
- camera1
- persp
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
@@ -456,7 +456,7 @@ steps:
mode: NOTIFY_THEN_TERMINATE
- name: renderSetupLayer2
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.renderSetupLayer2Frames}}'
@@ -466,7 +466,7 @@ steps:
- camera1
- camera2
- camera3
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
@@ -531,7 +531,7 @@ steps:
mode: NOTIFY_THEN_TERMINATE
- name: renderSetupLayer4
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.renderSetupLayer4Frames}}'
@@ -540,7 +540,7 @@ steps:
range:
- camera1
- persp
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
specificationVersion: '2022-09-01'
specificationVersion: jobtemplate-2023-09
name: layers_no_variation.ma
parameters:
parameterDefinitions:
- name: MayaSceneFile
type: PATH
objectType: FILE
@@ -112,15 +112,15 @@ parameters:
steps:
- name: layerTheFIrst
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.Frames}}'
- name: Camera
type: STRING
range:
- persp
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
@@ -185,15 +185,15 @@ steps:
mode: NOTIFY_THEN_TERMINATE
- name: layerTheSecond
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.Frames}}'
- name: Camera
type: STRING
range:
- persp
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
@@ -258,15 +258,15 @@ steps:
mode: NOTIFY_THEN_TERMINATE
- name: masterLayer
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.Frames}}'
- name: Camera
type: STRING
range:
- persp
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -10,8 +10,8 @@ license = ""
requires-python = ">=3.7"

dependencies = [
"deadline == 0.20.*",
"openjobio == 0.8.*",
"deadline == 0.23.*",
"openjd == 0.10.*",
]

[project.scripts]
2 changes: 1 addition & 1 deletion src/deadline/maya_adaptor/MayaAdaptor/__main__.py
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@
import logging
import sys

from openjobio.adaptor_runtime import EntryPoint
from openjd.adaptor_runtime import EntryPoint

from .adaptor import MayaAdaptor

24 changes: 12 additions & 12 deletions src/deadline/maya_adaptor/MayaAdaptor/adaptor.py
Original file line number Diff line number Diff line change
@@ -14,13 +14,13 @@
from pathlib import Path
from typing import Callable

from openjobio.adaptor_runtime.adaptors import Adaptor, AdaptorDataValidators
from openjobio.adaptor_runtime_client import Action
from openjobio.adaptor_runtime.adaptors.configuration import AdaptorConfiguration
from openjobio.adaptor_runtime.process import LoggingSubprocess
from openjobio.adaptor_runtime.app_handlers import RegexCallback, RegexHandler
from openjobio.adaptor_runtime.application_ipc import ActionsQueue, AdaptorServer
from openjobio.adaptor_runtime._utils import secure_open
from openjd.adaptor_runtime.adaptors import Adaptor, AdaptorDataValidators
from openjd.adaptor_runtime_client import Action
from openjd.adaptor_runtime.adaptors.configuration import AdaptorConfiguration
from openjd.adaptor_runtime.process import LoggingSubprocess
from openjd.adaptor_runtime.app_handlers import RegexCallback, RegexHandler
from openjd.adaptor_runtime.application_ipc import ActionsQueue, AdaptorServer
from openjd.adaptor_runtime._utils import secure_open

_logger = logging.getLogger(__name__)

@@ -299,16 +299,16 @@ def _start_maya_client(self) -> None:
mayapy_exe = "mayapy"
regexhandler = RegexHandler(self._get_regex_callbacks())

# Add the OpenJobIO namespace directory to PYTHONPATH, so that adaptor_runtime_client
# Add the openjd namespace directory to PYTHONPATH, so that adaptor_runtime_client
# will be available directly to the adaptor client.
import openjobio.adaptor_runtime_client
import openjd.adaptor_runtime_client
import deadline.maya_adaptor

openjobio_namespace_dir = os.path.dirname(
os.path.dirname(openjobio.adaptor_runtime_client.__file__)
openjd_namespace_dir = os.path.dirname(
os.path.dirname(openjd.adaptor_runtime_client.__file__)
)
deadline_namespace_dir = os.path.dirname(os.path.dirname(deadline.maya_adaptor.__file__))
python_path_addition = f"{openjobio_namespace_dir}{os.pathsep}{deadline_namespace_dir}"
python_path_addition = f"{openjd_namespace_dir}{os.pathsep}{deadline_namespace_dir}"
if "PYTHONPATH" in os.environ:
os.environ[
"PYTHONPATH"
4 changes: 2 additions & 2 deletions src/deadline/maya_adaptor/MayaClient/maya_client.py
Original file line number Diff line number Diff line change
@@ -6,15 +6,15 @@
from types import FrameType
from typing import Optional

# The Maya Adaptor adds the `openjobio` namespace directory to PYTHONPATH,
# The Maya Adaptor adds the `openjd` namespace directory to PYTHONPATH,
# so that importing just the adaptor_runtime_client should work.
try:
from adaptor_runtime_client import HTTPClientInterface # type: ignore[import]
from maya_adaptor.MayaClient.render_handlers import ( # type: ignore[import]
get_render_handler,
)
except (ImportError, ModuleNotFoundError):
from openjobio.adaptor_runtime_client import HTTPClientInterface # type: ignore[import]
from openjd.adaptor_runtime_client import HTTPClientInterface # type: ignore[import]
from deadline.maya_adaptor.MayaClient.render_handlers import ( # type: ignore[import]
get_render_handler,
)
12 changes: 6 additions & 6 deletions src/deadline/maya_submitter/adaptor_override_environment.yaml
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
specificationVersion: '2022-09-01'
parameters:
specificationVersion: 'jobtemplate-2023-09'
parameterDefinitions:
- name: OverrideAdaptorWheels
type: PATH
objectType: DIRECTORY
dataFlow: IN
description: A directory that contains wheels for openjobio, deadline, and the overridden adaptor.
description: A directory that contains wheels for openjd, deadline, and the overridden adaptor.
- name: OverrideAdaptorName
type: STRING
description: The name of the adaptor to override, for example NukeAdaptor or MayaAdaptor.
@@ -40,7 +40,7 @@ environment:
echo ""
echo "Installing adaptor into the venv"
pip install {{Param.OverrideAdaptorWheels}}/openjobio*.whl
pip install {{Param.OverrideAdaptorWheels}}/openjd*.whl
pip install {{Param.OverrideAdaptorWheels}}/deadline*.whl
echo ""
@@ -79,6 +79,6 @@ environment:
for k, v in put.items():
print(f"updating {k}={v}")
print(f"openjobio_env: {k}={v}")
print(f"openjd_env: {k}={v}")
for k in delete:
print(f"openjobio_unset_env: {k}")
print(f"openjd_unset_env: {k}")
8 changes: 4 additions & 4 deletions src/deadline/maya_submitter/default_maya_job_template.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
specificationVersion: '2022-09-01'
specificationVersion: 'jobtemplate-2023-09'
name: Default Maya Job Template
parameters:
parameterDefinitions:
- name: MayaSceneFile
type: PATH
objectType: FILE
@@ -78,11 +78,11 @@ parameters:
steps:
- name: Render
parameterSpace:
parameters:
taskParameterDefinitions:
- name: Frame
type: INT
range: '{{Param.Frames}}'
environments:
stepEnvironments:
- name: Maya
description: Runs Maya in the background.
script:
48 changes: 25 additions & 23 deletions src/deadline/maya_submitter/maya_render_submitter.py
Original file line number Diff line number Diff line change
@@ -71,24 +71,24 @@ def _get_job_template(
# If there are multiple frame ranges, split up the Frames parameter by layer
if render_layers[0].frames_parameter_name:
# Extract the Frames parameter definition
frame_param = [param for param in job_template["parameters"] if param["name"] == "Frames"][
0
]
job_template["parameters"] = [
param for param in job_template["parameters"] if param["name"] != "Frames"
frame_param = [
param for param in job_template["parameterDefinitions"] if param["name"] == "Frames"
][0]
job_template["parameterDefinitions"] = [
param for param in job_template["parameterDefinitions"] if param["name"] != "Frames"
]

# Create layer-specific Frames parameters
for layer_data in render_layers:
layer_frame_param = deepcopy(frame_param)
layer_frame_param["name"] = layer_data.frames_parameter_name
layer_frame_param["userInterface"]["groupLabel"] = layer_data.ui_group_label
job_template["parameters"].append(layer_frame_param)
job_template["parameterDefinitions"].append(layer_frame_param)

# If there are multiple output image formats, split that up by layer
if render_layers[0].output_file_prefix_parameter_name:
for layer_data in render_layers:
job_template["parameters"].append(
job_template["parameterDefinitions"].append(
{
"name": layer_data.output_file_prefix_parameter_name,
"type": "STRING",
@@ -101,7 +101,7 @@ def _get_job_template(
}
)
else:
job_template["parameters"].append(
job_template["parameterDefinitions"].append(
{
"name": "OutputFilePrefix",
"type": "STRING",
@@ -117,7 +117,7 @@ def _get_job_template(
# If there are multiple output image resolutions, split that up by layer
if render_layers[0].image_width_parameter_name:
for layer_data in render_layers:
job_template["parameters"].append(
job_template["parameterDefinitions"].append(
{
"name": layer_data.image_width_parameter_name,
"type": "INT",
@@ -130,7 +130,7 @@ def _get_job_template(
"description": f"The image width for layer {layer_data.display_name}.",
}
)
job_template["parameters"].append(
job_template["parameterDefinitions"].append(
{
"name": layer_data.image_height_parameter_name,
"type": "INT",
@@ -144,7 +144,7 @@ def _get_job_template(
}
)
else:
job_template["parameters"].append(
job_template["parameterDefinitions"].append(
{
"name": "ImageWidth",
"type": "INT",
@@ -157,7 +157,7 @@ def _get_job_template(
"description": "The image width of the output.",
}
)
job_template["parameters"].append(
job_template["parameterDefinitions"].append(
{
"name": "ImageHeight",
"type": "INT",
@@ -187,7 +187,7 @@ def _get_job_template(
"description": "Select which camera to render.",
"allowedValues": selectable_cameras,
}
job_template["parameters"].append(camera_param)
job_template["parameterDefinitions"].append(camera_param)

# Replicate the default step, once per render layer, and adjust its settings
default_step = job_template["steps"][0]
@@ -201,12 +201,12 @@ def _get_job_template(
parameter_space = step["parameterSpace"]
# Update the 'Param.Frames' reference in the Frame task parameter
if layer_data.frames_parameter_name:
parameter_space["parameters"][0]["range"] = (
parameter_space["taskParameterDefinitions"][0]["range"] = (
"{{Param." + layer_data.frames_parameter_name + "}}"
)
# If we're submitting all cameras, create another parameter space dimension
if settings.camera_selection == ALL_CAMERAS:
parameter_space["parameters"].append(
parameter_space["taskParameterDefinitions"].append(
{
"name": "Camera",
"type": "STRING",
@@ -217,7 +217,7 @@ def _get_job_template(
run_data["data"] += "camera: '{{Task.Param.Camera}}'\n"

# Update the init data of the step
init_data = step["environments"][0]["script"]["embeddedFiles"][0]
init_data = step["stepEnvironments"][0]["script"]["embeddedFiles"][0]
init_data["data"] = (
f"renderer: {layer_data.renderer_name}\nrender_layer: {layer_data.display_name}\n"
+ init_data["data"]
@@ -243,7 +243,7 @@ def _get_job_template(

# If Arnold is one of the renderers, add Arnold-specific parameters
if "arnold" in renderers:
job_template["parameters"].append(
job_template["parameterDefinitions"].append(
{
"name": "ArnoldErrorOnLicenseFailure",
"type": "STRING",
@@ -273,28 +273,28 @@ def _get_job_template(
wheels_path_package_names = {
path.split("-", 1)[0] for path in os.listdir(wheels_path) if path.endswith(".whl")
}
if wheels_path_package_names != {"openjobio", "deadline", "deadline_cloud_for_maya"}:
if wheels_path_package_names != {"openjd", "deadline", "deadline_cloud_for_maya"}:
raise RuntimeError(
"The Developer Option 'Include Adaptor Wheels' is enabled, but the wheels directory contains the wrong wheels:\n"
+ "Expected: openjobio, deadline, and deadline_cloud_for_maya\n"
+ "Expected: openjd, deadline, and deadline_cloud_for_maya\n"
+ f"Actual: {wheels_path_package_names}"
)

override_adaptor_wheels_param = [
param
for param in override_environment["parameters"]
for param in override_environment["parameterDefinitions"]
if param["name"] == "OverrideAdaptorWheels"
][0]
override_adaptor_wheels_param["default"] = str(wheels_path)
override_adaptor_name_param = [
param
for param in override_environment["parameters"]
for param in override_environment["parameterDefinitions"]
if param["name"] == "OverrideAdaptorName"
][0]
override_adaptor_name_param["default"] = "MayaAdaptor"

# There are no parameter conflicts between these two templates, so this works
job_template["parameters"].extend(override_environment["parameters"])
job_template["parameterDefinitions"].extend(override_environment["parameterDefinitions"])

# Add the environment to the end of the template's job environments
if "environments" not in job_template:
@@ -429,7 +429,9 @@ def show_maya_render_submitter(parent, f=Qt.WindowFlags()) -> "Optional[SubmitJo

# Get the RezPackages parameter definition, and use the default set there
rez_package_param = [
param for param in default_job_template["parameters"] if param["name"] == "RezPackages"
param
for param in default_job_template["parameterDefinitions"]
if param["name"] == "RezPackages"
]
if rez_package_param:
default_rez_packages = render_settings.rez_packages = rez_package_param[0].get(
Original file line number Diff line number Diff line change
@@ -11,7 +11,7 @@

import pytest
import jsonschema # type: ignore
from openjobio.adaptor_runtime_client import PathMappingRule
from openjd.adaptor_runtime_client import PathMappingRule

import deadline.maya_adaptor.MayaAdaptor.adaptor as adaptor_module
from deadline.maya_adaptor.MayaAdaptor import MayaAdaptor

0 comments on commit 5fb06ce

Please sign in to comment.