Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore: Set environment variables for farm #1023

Merged
merged 8 commits into from
Nov 26, 2024
+35 −0
Merged

Chore: Set environment variables for farm #1023

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9cd354e
added constant to define store key for env variables
iLLiCiTiT Nov 25, 2024
fae4eed
added new plugin collecting environment variables to context
iLLiCiTiT Nov 25, 2024
8111b4d
Merge branch 'develop' into enhancement/farm-env-variables
iLLiCiTiT Nov 25, 2024
85be6b2
remove unnecessary attribute
iLLiCiTiT Nov 25, 2024
fa88c29
Merge branch 'develop' into enhancement/farm-env-variables
iLLiCiTiT Nov 25, 2024
cd5f89a
remove OpenPype env key
iLLiCiTiT Nov 26, 2024
c7940b4
added comment to workdir env
iLLiCiTiT Nov 26, 2024
4d86202
Merge branch 'develop' into enhancement/farm-env-variables
iLLiCiTiT Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions client/ayon_core/pipeline/publish/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
ValidateContentsOrder,
ValidateSceneOrder,
ValidateMeshOrder,
FARM_JOB_ENV_DATA_KEY,
)

from .publish_plugins import (
Expand Down Expand Up @@ -59,6 +60,7 @@
"ValidateContentsOrder",
"ValidateSceneOrder",
"ValidateMeshOrder",
"FARM_JOB_ENV_DATA_KEY",

"AbstractMetaInstancePlugin",
"AbstractMetaContextPlugin",
Expand Down
2 changes: 2 additions & 0 deletions client/ayon_core/pipeline/publish/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,5 @@
DEFAULT_PUBLISH_TEMPLATE = "default"
DEFAULT_HERO_PUBLISH_TEMPLATE = "default"
TRANSIENT_DIR_TEMPLATE = "default"

FARM_JOB_ENV_DATA_KEY: str = "farmJobEnv"
33 changes: 33 additions & 0 deletions client/ayon_core/plugins/publish/collect_farm_env_variables.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import os

import pyblish.api

from ayon_core.pipeline.publish import FARM_JOB_ENV_DATA_KEY


class CollectCoreJobEnvVars(pyblish.api.ContextPlugin):
"""Collect set of environment variables to submit with deadline jobs"""
order = pyblish.api.CollectorOrder - 0.45
label = "AYON core Farm Environment Variables"
targets = ["local"]

def process(self, context):
env = context.data.setdefault(FARM_JOB_ENV_DATA_KEY, {})
for key in [
# AYON
"AYON_BUNDLE_NAME",
"AYON_DEFAULT_SETTINGS_VARIANT",
"AYON_PROJECT_NAME",
"AYON_FOLDER_PATH",
"AYON_TASK_NAME",
"AYON_WORKDIR",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's technically not related to this PR - but why do we need to pass the workdir? Isn't this workdir path also local to the current machine and not rootless? I have a feeling we should maybe, after this PR investigate this and potentially clean that up?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be used here by the way:

  • ayon-core/client/ayon_core/plugins/publish/collect_rendered_files.py

    Lines 176 to 187 in a08a1af

    # Remap workdir if it's set
    workdir = os.getenv("AYON_WORKDIR")
    remapped_workdir = None
    if workdir:
    remapped_workdir = anatomy.roots_obj.path_remapper(
    os.getenv("AYON_WORKDIR")
    )
    if remapped_workdir:
    os.environ["AYON_WORKDIR"] = remapped_workdir
    except Exception as e:
    self.log.error(e, exc_info=True)
    raise Exception("Error") from e

But I guess that should still be set by the "launch context" regardless?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did try to find out where it is used, and I didn't find anything. If you can create issue it would be nice.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be used here by the way:

Yes, but is it then used for anything?

"AYON_LOG_NO_COLORS",
"AYON_IN_TESTS",
# backwards compatibility
"IS_TEST",
iLLiCiTiT marked this conversation as resolved.
Show resolved Hide resolved
]:
value = os.getenv(key)
if value:
self.log.debug(f"Setting job env: {key}: {value}")
env[key] = value