Feature request: Customizing DCC Submitters (UI, settings, hooks, custom steps)

[Ahuge]

Use Case

This is a discussion around scoping out a feature request that I initially asked in #292 around adding a "pre-submit" and "post-submit" callback.

After discussing with @epmog we have determined that some additional scoping around this is needed and I'd like to solicit opinions on it.

Proposed Solution

Initially the PR focused around two hooks that ran after the UI had been created but just before and just after the job was being submitted.

This was designed to solve my own use case of modifying a Nuke scene before a job submission and then reverting the changes for the artist just afterwards.
Our studios had been doing this around the "SubmissionHook.py" in the old Deadline application to strip out ShotGrid Toolkit Write Nodes from Nuke and convert them to generic write nodes so that the farm didn't have to deal with SGTK stuff.

epmog suggested potential features around the ability to create/embed custom gui controls, pre-populating values/overriding certain sticky settings, translate assets into openjd compatible ones, add custom steps/modify existing ones.

I can see these being grouped into "Before UI" and "After UI" options, with the "After UI" one being another chance to modify the scene.

[Ahuge]

Custom GUI Controls

I think the first thing we need to determine here is if it is ok to have all custom controls on a separate tab, or if we need to provide the ability to add custom elements to each existing tab (Shared Job Settings, Job-Specific Settings, Job Attachments, Host Requirements)

Shared Job Settings

I don't have any suggestions for what may be added to this section. Please suggest ideas.

Job-Specific Settings

This is a DCC submitter provided QWidget object however we could allow for a developer provided widget to replace the existing widget.

Job Attachments

I don't have any suggestions for what may be added to this section. Please suggest ideas.
I think it is more likely that developers will be adding to the translated OpenJD assets and those will be automatically shown inline here as well.

Host Requirements

The main modification that I anticipate seeing here would be setting up default settings within this Host Requirements widget and I see that either via passing this widget and implementing a programmatic api on it that can be set, or allowing these settings to be pre-defined by some sort of developer provided "settings override".

Based on my initial look into the potential UI overrides, I think the most likely place a developer may want to override UI controls is in the Job-Specific Settings. I think we will want to allow them to provide their own UI Settings object as well.

[Ahuge]

Overriding Settings / Default Settings

The deadline-cloud already provides settings for everything except the host_requirements, so adding those would be valuable.

We should just need an interface that provides the settings objects, allowing them to be changed

[Ahuge]

Translation of Assets into OpenJD

I feel like providing the AssetReferences instance to a developer would allow them to provide additional assets as required,
I feel like all we would need here is an interface that sends over the AssetReferences

[Ahuge]

Custom OpenJD steps

This is where my lack of OpenJD knowledge starts to show.
I would appreciate any insight folks have on what this interface may look like.

I feel like we would have to provide some array of steps and allow the developer to return a potentially modified array usable for the submission.

[Ahuge]

Post Submission

This would be some interface that allows calling a developer provided function.
This function may update some external database, perform some actions in the DCC scene, or do something completely different.

I don't see a use for myself to have any UI based settings, however it may be valuable to have a "JobID" or some summary of the action that was just performed.

My personal use case would be limited to modifying the DCC scene

[Ahuge]

Interfaces

Is this going to be a single "mega interface" that allows adding UI widgets, modifying settings, providing additional assets, and changing the OpenJD steps? Or do we see this as a few different optional hooks that can be called.

I lean towards the latter, allowing a UI_CALLBACK, which would contain providing UI elements and potentially modifying settings, a JOB_CALLBACK, which would allow you to provide additional assets and modify the OpenJD steps, and then some sort of POST_SUBMIT_CALLBACK which gets the newly submitted JobID and does whatever it needs to with it.

That way developers and studios could choose to subscribe to as few or as many of those custom steps as they need without having to build a huge callback that takes a bunch of different unrelated parameters.

Would love any thoughts around this proposed grouping as well.

[epmog]

Thanks for writing all this up! I've been thinking on this for a little while, and the critical piece I want to focus would be the mechanism in which a user brings and applies their customization to 1 or many applications. Once we have this part figured out, actually adding the specific places in which a user can then hook into doesn't seem too particularly difficult.

Aside: Even with all of this, at some point a user may decide that it's easier to just write their own submitter using our library/parts, and that's totally fine and definitely encouraged. But if we can ease them into that with something here, it's probably a better experience than saying, "if you want anything different, fork and maintain it yourself".

I know in the original example PR, you leveraged an environment variable as a way to load the custom code for the specific DCC+callback combination. And I can see the appeal there since it's easy to manage configurations environments that have their specific settings/callbacks set. Do we see use-cases where multiple customizations for the DCC+callback combination could be applied?

This leads me to the next thought where an alternative to only leveraging environment variables for loading, we could implement a plugin system with deadline-cloud to allow users to inform deadline-cloud about their customizations. Given a say a default plugin folder (customizable via environment variable?) within the deadline namespace, users could install their plugins and register their code to run at certain timings.

I see a few benefits with this:

1. it's easy to share code amongst other users. ie. you could publish customizations to pypi and others could leverage them.
2. your installation can choose to leverage environments (ie. python, conda, rez, etc.) for versioning but it also enables the simple workflow of not doing any of that

Enforcing an ordering to modifications that are affecting the same thing would definitely be something to consider. There's probably more to think about that I haven't touched upon here, but that's the way I'm leaning.

Thoughts?

[Ahuge]

Hi @epmog

So I have a WIP POC version up that I have been using internally here with one customer that is using environment variables.

In the DCC submitter layer (deadline-cloud-for-nuke for example) here is a blurb of the code I am using:

...
        callback_kwargs = {}
        if os.path.exists(os.environ.get("DEADLINE_NUKE_UI_CALLBACK", "")):
            try:
                on_ui_callback = ui_callback.load_ui_callback(
                    module_path=os.environ.get("DEADLINE_NUKE_UI_CALLBACK")
                )
            except Exception:
                import traceback
                raise DeadlineOperationError(
                    "Error while loading on_pre_submit_callback at {path}. {trace}".format(
                        path=os.environ.get("DEADLINE_NUKE_UI_CALLBACK"),
                        trace=traceback.format_exc()
                    )
                )

            callback_kwargs["on_ui_callback"] = on_ui_callback

        if os.path.exists(os.environ.get("DEADLINE_NUKE_POST_SUBMIT_CALLBACK", "")):
            try:
                on_post_submit_callback = post_submit_callback.load_post_submit_callback(
                    module_path=os.environ.get("DEADLINE_NUKE_POST_SUBMIT_CALLBACK"),
                )
            except Exception:
                import traceback
                raise DeadlineOperationError(
                    "Error while loading on_post_submit_callback at {path}. {trace}".format(
                        path=os.environ.get("DEADLINE_NUKE_POST_SUBMIT_CALLBACK"),
                        trace=traceback.format_exc()
                    )
                )
            callback_kwargs["on_post_submit_callback"] = on_post_submit_callback

As you can see above, the structure I have proposed is DEADLINE_<DCC>_<CALLBACK_NAME>.
Those callback_kwargs are then passed into the SubmitJobToDeadlineDialog constructore and used by the core deadline-cloud library.
The nice thing about that is that the DCC submitters define the environment variable name, so you can share plugins (if applicable) or provide per-DCC ones.
I've attached one example ui_callback.py file that I have to this so you can see how it works.
ui_callback.py.txt

This leads me to the next thought where an alternative to only leveraging environment variables for loading, we could implement a plugin system with deadline-cloud to allow users to inform deadline-cloud about their customizations. Given a say a default plugin folder (customizable via environment variable?) within the deadline namespace, users could install their plugins and register their code to run at certain timings.

Your suggestion of a plugin folder is interesting as well. I have a few thoughts about that, firstly, are you planning on just adding that DEADLINE_PLUGIN_PATH to the PYTHONPATH and allowing python to load deadline.callbacks.ui_callback (for example)? I think that could work quite well and then still support environment variables for overrides.

Secondly, are you thinking about additional plugin/integration points beyond the several we have defined above? I'd be interested to hear what else you're planning.

I know in the original example PR, you leveraged an environment variable as a way to load the custom code for the specific DCC+callback combination. And I can see the appeal there since it's easy to manage configurations environments that have their specific settings/callbacks set. Do we see use-cases where multiple customizations for the DCC+callback combination could be applied?

I could imagine a world where two departments at a larger studio each have their own deadline submitter plugin potentially and the desire is to make both run. I don't personally have a use case for that but I could potentially see someone asking for that down the road.
I think in that case we could make the environment variable a PATH style one where we resolve each file within it.
It would complicate the DEADLINE_PLUGIN_PATH one as the first deadline.callback.ui_callback found in the path would be the only one loaded. You would have to implement some callback registration system where each callback in DEADLINE_PLUGIN_PATH would register itself with a deadline callback manager and then it knows about each.
My opinion is that supporting multiple ui_callback's within the same DCC at this point feels like over complicating it.
I see a potential way to add it in the future if we need to but it feels to me like unless someone has expressed a need for that, we should hold off for now on it. Technically a user could implement their own lazy-load/multi-load system within their single ui_callback hook.

Once we have this part figured out, actually adding the specific places in which a user can then hook into doesn't seem too particularly difficult.
I agree, you probably have a better idea of initial places to call these hooks than I do. My naive implementation has put ui_callback in the __init__ and refresh of SubmitJobToDeadlineDialog and post_submit_callback after job_progress_dialog.start_submission in submit (although I also need to support it potentially on dialog closing without a submit)

Aside: Even with all of this, at some point a user may decide that it's easier to just write their own submitter using our library/parts, and that's totally fine and definitely encouraged. But if we can ease them into that with something here, it's probably a better experience than saying, "if you want anything different, fork and maintain it yourself".

Regarding users writing their own submitters, I agree I think some power users/studios will end up writing their own submitters but making the Deadline reference implementation as flexible as possible will allow a central community/repository for people to start from.

I'll be honest, starting to write my own OpenJD jobs and digging down into the internals of this implementation has been more difficult than I anticipated and I think that other developers that either have less experience or less time may also feel the same.
I think having a strong "generic" implementation of the submitters that can be at least partially customized is a valuable thing to have for the service and then because it is built on OpenJD, it can empower studios and very customized workflows to build their own solutions but still look to the submitters for reference.

I'm going to continue with my fork in the short term as it is helping me learn more about the codebase and respond with more relevant answers.

Thanks for your thought on this @epmog. Let me know if you have any thoughts on anything I wrote above.