Skip to content

Commit

Permalink
fix: Use correct paths on windows for local manifest file
Browse files Browse the repository at this point in the history
Signed-off-by: Caden Marofke <[email protected]>
  • Loading branch information
marofke committed Apr 15, 2024
1 parent a62e0ae commit 15444ba
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 10 deletions.
2 changes: 1 addition & 1 deletion src/deadline/job_attachments/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ These snapshots are encapsulated in one or more [`asset_manifests`](asset_manife

When starting work, the worker downloads the manifest associated with your job, and recreates the file structure of your submission locally, either downloading all files at once, or as needed if using the [virtual][vfs] job attachments filesystem type. When a task completes, the worker creates a new manifest for any outputs that were specified in the job submission, and uploads the manifest and the outputs back to your S3 bucket.

Manifest files are written to a `manifests` directory within each job bundle that is added to the job history if submitted through the GUI (default: `~/.deadline/job_history`). The file path inside the `manifests` directory corresponds to the S3 manifest path in the submitted job's job attachments metadata.
Manifest files are written to a `manifests` directory within each job bundle that is added to the job history if submitted through the GUI (default: `~/.deadline/job_history`). A corresponding `manifest_s3_mapping` file is created alongside manifests, which specifies each local manifest file with the S3 manifest path in the submitted job's job attachments metadata.

[vfs]: https://docs.aws.amazon.com/deadline-cloud/latest/userguide/storage-virtual.html

Expand Down
20 changes: 11 additions & 9 deletions src/deadline/job_attachments/upload.py
Original file line number Diff line number Diff line change
Expand Up @@ -170,15 +170,6 @@ def upload_assets(
)
manifest_name = f"{manifest_name_prefix}_input"

if manifest_write_dir:
local_manifest_file = os.path.join(
manifest_write_dir, "manifests", partial_manifest_prefix, manifest_name
)
logger.info(f"Creating local manifest file: {local_manifest_file}\n")
os.makedirs(os.path.dirname(local_manifest_file), exist_ok=True)
with open(local_manifest_file, "w") as file:
file.write(manifest.encode())

if partial_manifest_prefix:
partial_manifest_key = _join_s3_paths(partial_manifest_prefix, manifest_name)
else:
Expand All @@ -188,6 +179,17 @@ def upload_assets(
partial_manifest_key
)

if manifest_write_dir:
local_manifest_file = Path(manifest_write_dir, "manifests", manifest_name)
logger.info(f"Creating local manifest file: {local_manifest_file}\n")
local_manifest_file.parent.mkdir(parents=True, exist_ok=True)
with open(local_manifest_file, "w") as file:
file.write(manifest.encode())
manifest_map_file = Path(manifest_write_dir, "manifests", "manifest_s3_mapping")
mapping = {"local_file": manifest_name, "s3_key": full_manifest_key}
with open(manifest_map_file, "a") as mapping_file:
mapping_file.write(f"{mapping}\n")

self.upload_bytes_to_s3(
bytes=BytesIO(manifest_bytes),
bucket=job_attachment_settings.s3BucketName,
Expand Down

0 comments on commit 15444ba

Please sign in to comment.