feat: integrate with job attachment download cli as a openjd action run #476
Conversation
godobyte
changed the title
| # =========================== TO BE DELETED =========================== | ||
| path_mapping_file_path: str = os.path.join( | ||
| session._session.working_directory, "path_mapping" | ||
| ) | ||
| for rule in job_attachment_path_mappings: | ||
| rule["source_path"] = rule["destination_path"] | ||
|
|
||
| with open(path_mapping_file_path, "w", encoding="utf8") as f: | ||
| f.write(json.dumps([rule for rule in job_attachment_path_mappings])) | ||
| # =========================== TO BE DELETED =========================== |
There was a problem hiding this comment.
Keeping it for now, pending changes and testing with attachment upload cli action integration.
There was a problem hiding this comment.
I actually think writing out a path mapping file is super useful for users? They can easily cat this file and use jq Or are we depending on the environment variable for path mapping for customers?
There was a problem hiding this comment.
Openjd session already writes the path mapping files, it's in a slightly different format tho, that's why I was thinking whether still keep this path mapping file or reuse the path written by openjd. Openjd would need to change to return the file path, and the attachment CLI would need slight change to read the following file.
{
"version": "pathmapping-1.0",
"path_mapping_rules": [{
"source_path_format": "POSIX",
"source_path": "/local/bea/bealine-clients/deadline-cloud-worker-agent/scripts/submit_jobs/asset_example",
"destination_path": "/sessions/session-39d74575600f47eb80d31d440721a27b7mrqf9ip/assetroot-a7714e87e776d9f1c179"
}]
}
There was a problem hiding this comment.
There was a problem hiding this comment.
I think that's a good idea if it fits the data flow, need to make sure the Session.PathMappingRulesFile can be used for both attachment download and attachment upload. Let's chat more offline.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ) | ||
| if "stepId" not in action_definition: | ||
| action_queue_entry = cast( |
Check notice
Code scanning / CodeQL
Unused local variable Note
| job_attachment_details=job_attachment_details, | ||
| ) | ||
| else: | ||
| action_queue_entry = cast( |
Check notice
Code scanning / CodeQL
Unused local variable Note
src/deadline_worker_agent/sessions/actions/run_attachment_download.py
Outdated
Show resolved
Hide resolved
src/deadline_worker_agent/sessions/actions/run_attachment_download.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Godot Bian <[email protected]>
godobyte
changed the title
godobyte
changed the title
|
|
||
|
|
||
| class AttachmentDownloadAction(OpenjdAction): | ||
| """Action to synchronize input job attachments for a AWS Deadline Cloud job |
There was a problem hiding this comment.
nit, for Deadline Cloud Session? Since a session has job and step-step dependency.
| ) | ||
| ) | ||
|
|
||
| vfs_handled = self._vfs_handling( |
There was a problem hiding this comment.
nit, is this starting VFS? If yes, maybe self._start_vfs(...
godobyte
added
enhancement
godobyte
changed the title
godobyte
force-pushed
the
sync-input
branch
2 times, most recently
from
2aa7cff to
83d67b7
Compare
| if ASSET_SYNC_JOB_USER_FEATURE: | ||
| action_definition = cast(AttachmentDownloadActionApiModel, action_definition) | ||
| if "stepId" not in action_definition: | ||
| action_queue_entry = cast( |
Check notice
Code scanning / CodeQL
Unused local variable Note
| job_attachment_details=job_attachment_details, | ||
| ) | ||
| else: | ||
| action_queue_entry = cast( |
Check notice
Code scanning / CodeQL
Unused local variable Note
| # =========================== TO BE DELETED =========================== | ||
| path_mapping_file_path: str = os.path.join( | ||
| session._session.working_directory, "path_mapping" | ||
| ) | ||
| for rule in job_attachment_path_mappings: | ||
| rule["source_path"] = rule["destination_path"] | ||
|
|
||
| with open(path_mapping_file_path, "w", encoding="utf8") as f: | ||
| f.write(json.dumps([rule for rule in job_attachment_path_mappings])) | ||
| # =========================== TO BE DELETED =========================== |
There was a problem hiding this comment.
godobyte
force-pushed
the
sync-input
branch
2 times, most recently
from
2088943 to
7d2c4d6
Compare
Signed-off-by: Godot Bian <[email protected]>
|
| session._session._path_mapping_rules.sort(key=lambda rule: -len(rule.source_path.parts)) | ||
|
|
||
| # =========================== TO BE DELETED =========================== | ||
| path_mapping_file_path: str = os.path.join( |
There was a problem hiding this comment.
nit this will be deleted later right?
There was a problem hiding this comment.
Yes. In order to delete this, I'd need to do some additional changes, but that'd be part of this feature completion.
What was the problem/requirement? (What/Why)
Worker Agent runs sync_input and sync_output callbacks to interact with customer s3 as worker agent user. This brings the following problem
What was the solution? (How)
Worker Agent is in the process of changing Job Attachment sync input and sync output to use attachment and manifest commands running as job user.
What is the impact of this change?
This is still in development and is under a feature flag.
How was this change tested?
Locally set the feature flag, build and install whl files, run
hatch deadline-worker-agentWas this change documented?
It's currently only documented in the code docstring.
Is this a breaking change?
No
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.