From f8e1f50abf4e87f97c37d7e67eee30dd79e46936 Mon Sep 17 00:00:00 2001 From: Morgan Epp <60796713+epmog@users.noreply.github.com> Date: Sun, 31 Mar 2024 23:16:18 -0500 Subject: [PATCH] docs: update README/DEVELOPMENT Signed-off-by: Morgan Epp <60796713+epmog@users.noreply.github.com> --- DEVELOPMENT.md | 77 +++++++- README.md | 171 +++++++++++------- docs/telemetry.md | 10 + src/deadline/client/cli/_deadline_cli.py | 4 +- src/deadline/client/cli/_groups/auth_group.py | 2 +- src/deadline/job_attachments/README.md | 14 +- 6 files changed, 202 insertions(+), 76 deletions(-) create mode 100644 docs/telemetry.md diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index be03297e7..638c8e831 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -1,4 +1,77 @@ -# The AWS Deadline Cloud Client Library (`deadline.client`) +# Development documentation + +This documentation provides guidance on developer workflows for working with the code in this repository. + +## Code organization + +This repository is split up into two main modules: +1. `src/client` +2. `src/job_attachments` + +The `src/client` organization is laid out below. + +For more information on job attachments, see [here](src/deadline/job_attachments/README.md). + +### `src/client/api` + +This submodule contains utilities to call boto3 in a standardized way +using an aws profile configured for AWS Deadline Cloud, helpers for working with the +AWS Deadline Cloud monitor login/logout, and objects representing AWS Deadline Cloud +resources. + +### `src/client/cli` + +This submodule contains entry points for the CLI applications provided +by the library. + +### `src/client/config` + +This submodule contains an interface to the machine-specific AWS Deadline Cloud +configuration, specifically settings stored in `~/.deadline/*` + +### `src/client/ui` + +This submodule contains Qt GUIs, based on PySide(2/6), for common controls +and widgets used in interactive submitters, and to display the status +of various AWS Deadline Cloud resoruces. + +### `src/client/job_bundle` + +This submodule contains code related to the history of job submissions +performed on the workstation. Its initial functionality is to create +job bundle directories in a standardized manner. + +# Build / Test / Release + +## Build the package. +``` +hatch build +``` + +## Run tests +``` +hatch run test +``` + +## Run integration tests +``` +hatch run integ:test +``` + +## Run linting +``` +hatch run lint +``` + +## Run formating +``` +hatch run fmt +``` + +## Run tests for all supported Python versions. +``` +hatch run all:test +``` ## Qt and Calling AWS (including AWS Deadline Cloud) APIs @@ -90,7 +163,7 @@ class MyCustomWidget(QWidget): ``` -**We recommend you set up your runtimes via `asdf`.** +**We recommend you set up your runtimes via `mise`.** ## Running Docker-based Unit Tests diff --git a/README.md b/README.md index 60f2d8668..8904038a2 100644 --- a/README.md +++ b/README.md @@ -1,111 +1,154 @@ -# The AWS Deadline Cloud Client Library (`deadline.client`) +# AWS Deadline Cloud client -## Overview +[![pypi](https://img.shields.io/pypi/v/deadline.svg)](https://pypi.python.org/pypi/deadline) -This is a shared Python library that implements functionality to support -client applications using AWS Deadline Cloud. -It is divided into the following submodules: +AWS Deadline Cloud client is a multi-purpose python library and command line tool for interacting with and submitting [Open Job Description][openjd] jobs to [AWS Deadline Cloud][deadline-cloud]. -### api -This submodule contains utilities to call boto3 in a standardized way -using an aws profile configured for AWS Deadline Cloud, helpers for working with -Deadline Cloud Monitor Desktop login/logout, and objects representing AWS Deadline Cloud -resources. +[deadline-cloud]: https://docs.aws.amazon.com/deadline-cloud/latest/userguide/what-is-deadline-cloud.html +[deadline-cloud-monitor]: https://docs.aws.amazon.com/deadline-cloud/latest/userguide/working-with-deadline-monitor.html +[deadline-cloud-samples]: https://github.com/aws-deadline/deadline-cloud-samples +[deadline-jobs]: https://docs.aws.amazon.com/deadline-cloud/latest/userguide/deadline-cloud-jobs.html +[openjd]: https://github.com/OpenJobDescription/openjd-specifications/wiki -### cli - -This submodule contains entry points for the CLI applications provided -by the library. - -### config - -This submodule contains an interface to the machine-specific AWS Deadline Cloud -configuration, specifically settings stored in `~/.deadline/*` +## Compatibility -### ui +This library requires: -This submodule contains Qt GUIs, based on PySide(2/6), for common controls -and widgets used in interactive submitters, and to display the status -of various AWS Deadline Cloud resoruces. +1. Python 3.7 or higher; and +2. Linux, Windows, or macOS operating system. -### job_bundle +## Usage -This submodule contains code related to the history of job submissions -performed on the workstation. Its initial functionality is to create -job bundle directories in a standardized manner. +AWS Deadline Cloud client can be installed by the standard python packaging mechanisms: +``` +$ pip install deadline +``` -## Compatibility +or if you want the optional gui dependencies: +``` +pip install deadline[gui] +``` -This library requires: +it can then be used as a command line tool: +``` +$ deadline farm list +- farmId: farm-1234567890abcdefg + displayName: my-first-farm +``` -1. Python 3.7 or higher; and -2. Linux, MacOS, or Windows operating system. +or as a python library: +``` +>>> from deadline.client import api +>>> api.list_farms() +{'farms': [{'farmId': 'farm-1234567890abcdefg', 'displayName': 'my-first-farm', ...},]} +``` -## Versioning +## Job attachments -This package's version follows [Semantic Versioning 2.0](https://semver.org/), but is still considered to be in its -initial development, thus backwards incompatible versions are denoted by minor version bumps. To help illustrate how -versions will increment during this initial development stage, they are described below: +Job attachments enable you to transfer files between your workstations and AWS Deadline Cloud, by using Amazon S3 buckets as [content-addressed storage](https://en.wikipedia.org/wiki/Content-addressable_storage) in your AWS account. -1. The MAJOR version is currently 0, indicating initial development. -2. The MINOR version is currently incremented when backwards incompatible changes are introduced to the public API. -3. The PATCH version is currently incremented when bug fixes or backwards compatible changes are introduced to the public API. +See [job attachments](src/deadline/job_attachments/README.md) for a more in-depth look at how files are uploaded, stored, and retrieved. -## Downloading +## Job bundles -You can download this package from: -- [GitHub releases](https://github.com/casillas2/deadline-cloud/releases) +At minimum a job bundle is a folder that contains an [OpenJD][openjd] template, however it can optionally include: +1. an `asset_references.yaml` - lists relevant inputs and outputs, +2. a `parameter_values.yaml` - contains the selected values for the job template's parameters, +3. and any number of additional files required for the job. -## Development +For example job bundles, visit the [AWS Deadline Cloud samples repository][deadline-cloud-samples] -See instructions in DEVELOPMENT.md +To submit a job bundle, you can run -## Telemetry +``` +$ deadline bundle submit +``` -This library collects telemetry data by default. Telemetry events contain non-personally-identifiable information that helps us understand how users interact with our software so we know what features our customers use, and/or what existing pain points are. +or if you have the optional GUI components installed: -You can opt out of telemetry data collection by either: +``` +$ deadline bundle gui-submit --browse +``` -1. Setting the environment variable: `DEADLINE_CLOUD_TELEMETRY_OPT_OUT=true` -2. Setting the config file: `deadline config set telemetry.opt_out true` +For more information on jobs and job bundles, see [AWS Deadline Cloud jobs][deadline-jobs]. -Note that setting the environment variable supersedes the config file setting. +## Configuration -# Build / Test / Release +You can see the current configuration by running: -## Build the package. ``` -hatch build +$ deadline config show ``` -## Run tests +and change them by running the associated `get` and `set` commands. + +To see a list of settings that can be configured, run: ``` -hatch run test +$ deadline config --help ``` -## Run integration tests +Or you can manage settings by a graphical interface if you have the optional gui dependencies: + ``` -hatch run integ:test +deadline config gui ``` -## Run linting +By default, configuration of AWS Deadline Cloud is provided at `~/.deadline/config`, however this can be overridden by the `DEADLINE_CONFIG_FILE_PATH` environment variable. + +## Authentication + +Standard AWS credential mechanisms are supported within AWS Deadline Cloud client. + +To view the currently configured credentials authentication status, run: + ``` -hatch run lint +deadline auth status ``` -## Run formating +The AWS Profile can be + +If the currently selected AWS Profile is set-up to use AWS Deadline Cloud monitor credentials, you can run: + ``` -hatch run fmt +$ deadline auth login ``` -## Run tests for all supported Python versions. +and vice-versa: + ``` -hatch run all:test +$ deadline auth logout ``` -# Optional Third Party Dependencies - GUI +## Versioning + +This package's version follows [Semantic Versioning 2.0](https://semver.org/), but is still considered to be in its +initial development, thus backwards incompatible versions are denoted by minor version bumps. To help illustrate how +versions will increment during this initial development stage, they are described below: + +1. The MAJOR version is currently 0, indicating initial development. +2. The MINOR version is currently incremented when backwards incompatible changes are introduced to the public API. +3. The PATCH version is currently incremented when bug fixes or backwards compatible changes are introduced to the public API. + +## Contributing + +See [`CONTRIBUTING.md`](https://github.com/aws-deadline/deadline-cloud/blob/mainline/CONTRIBUTING.md) +for information on reporting issues, requesting features, and developer information. + +## Security + +See [security issue notifications](https://github.com/aws-deadline/deadline-cloud/blob/release/CONTRIBUTING.md#security-issue-notifications) for more information. + +## Telemetry + +See [telemetry](https://github.com/aws-deadline/deadline-cloud/blob/release/docs/telemetry.md) for more information. + +## Optional third party dependencies - GUI N.B.: Although this repository is released under the Apache-2.0 license, its optional GUI feature -uses the third party Qt && PySide projects. The Qt and PySide projects' licensing includes the LGPL-3.0 license. +uses the third party Qt and PySide projects. The Qt and PySide projects' licensing includes the LGPL-3.0 license. + +## License + +This project is licensed under the Apache-2.0 License. diff --git a/docs/telemetry.md b/docs/telemetry.md new file mode 100644 index 000000000..91ab49597 --- /dev/null +++ b/docs/telemetry.md @@ -0,0 +1,10 @@ +# Telemetry + +This library collects telemetry data by default. Telemetry events contain non-personally-identifiable information that helps us understand how users interact with our software so we know what features our customers use, and/or what existing pain points are. + +You can opt out of telemetry data collection by either: + +1. Setting the environment variable: `DEADLINE_CLOUD_TELEMETRY_OPT_OUT=true` +2. Setting the config file: `deadline config set telemetry.opt_out true` + +Note that setting the environment variable supersedes the config file setting. diff --git a/src/deadline/client/cli/_deadline_cli.py b/src/deadline/client/cli/_deadline_cli.py index 5d726f585..46d8e7027 100644 --- a/src/deadline/client/cli/_deadline_cli.py +++ b/src/deadline/client/cli/_deadline_cli.py @@ -55,8 +55,8 @@ @click.pass_context def main(ctx: click.Context, log_level: str): """ - The AWS Deadline Cloud CLI provides functionality to work with the AWS Deadline Cloud - closed beta service. + The AWS Deadline Cloud CLI provides functionality to interact with the AWS Deadline Cloud + service. """ logging.basicConfig(level=log_level) if log_level == "DEBUG": diff --git a/src/deadline/client/cli/_groups/auth_group.py b/src/deadline/client/cli/_groups/auth_group.py index 33be42b09..624796010 100644 --- a/src/deadline/client/cli/_groups/auth_group.py +++ b/src/deadline/client/cli/_groups/auth_group.py @@ -85,7 +85,7 @@ def auth_logout(): ) @_handle_error def auth_status(output, **args): - """EXPERIMENTAL - Gets the authentication status for the given AWS profile""" + """Gets the authentication status for the given AWS profile""" # Get a temporary config object with the standard options handled config = _apply_cli_options_to_config(**args) profile_name = get_setting("defaults.aws_profile_name", config=config) diff --git a/src/deadline/job_attachments/README.md b/src/deadline/job_attachments/README.md index cf428c6a9..a29a008a1 100644 --- a/src/deadline/job_attachments/README.md +++ b/src/deadline/job_attachments/README.md @@ -1,12 +1,12 @@ -# AWS Deadline Cloud Job Attachments +# Job attachments -[Job attachments][job-attachments] enable you to transfer files back and forth between your workstation and [AWS Deadline Cloud][deadline-cloud], using an Amazon S3 bucket in your AWS account associated with your [Deadline Cloud queues][queue]. +[Job attachments][job-attachments] enable you to transfer files back and forth between your workstation and [AWS Deadline Cloud][deadline-cloud], using an Amazon S3 bucket in your AWS account associated with your [AWS Deadline Cloud queues][queue]. -Job attachments uses your configured S3 bucket as a [content-addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage), which creates a snapshot of the files used in your job submission in [asset manifests](#asset-manifests), only uploading files that aren't already in S3. This saves you time and bandwidth when iterating on jobs. When an [AWS Deadline Cloud Worker Agent][worker-agent] starts working on a job with job attachments, it recreates the file system snapshot in the worker agent session directory, and uploads any outputs back to your S3 bucket. +Job attachments uses your configured S3 bucket as a [content-addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage), which creates a snapshot of the files used in your job submission in [asset manifests](#asset-manifests), only uploading files that aren't already in S3. This saves you time and bandwidth when iterating on jobs. When an [AWS Deadline Cloud worker agent][worker-agent] starts working on a job with job attachments, it recreates the file system snapshot in the worker agent session directory, and uploads any outputs back to your S3 bucket. -You can then easily download your outputs with the [Deadline client](../client/) `deadline job download-output` command, or using the [protocol handler](#protocol-handler) to download from a click of a button in the [Deadline Cloud Monitor][monitor]. +You can then easily download your outputs with the [deadline cli](../client/) `deadline job download-output` command, or using the [protocol handler](#protocol-handler) to download from a click of a button in the [AWS Deadline Cloud monitor][monitor]. -Job attachments also works as an auxiliary storage when used with [AWS Deadline Cloud Storage Profiles][shared-storage], allowing you to flexibly upload files to your Amazon S3 bucket that aren't on your configured shared storage. +Job attachments also works as an auxiliary storage when used with [AWS Deadline Cloud storage profiles][shared-storage], allowing you to flexibly upload files to your Amazon S3 bucket that aren't on your configured shared storage. See the [`examples`](../../../examples/) directory for some simple examples on how to use job attachments. @@ -53,7 +53,7 @@ In order to further improve submission time, there are currently two local [`cac ## Protocol Handler -On Windows and Linux operating systems, you can choose to install the [Deadline client](../client/) protocol handler in order to run AWS Deadline Cloud commands sent from a web browser. Of note is the ability to download job attachments outputs from your jobs through the [Deadline Cloud monitor][downloading-output]. +On Windows and Linux operating systems, you can choose to install the [Deadline client](../client/) protocol handler in order to run AWS Deadline Cloud commands sent from a web browser. Of note is the ability to download job attachments outputs from your jobs through the [AWS Deadline Cloud monitor][downloading-output]. You can install the protocol handler by running the command: `deadline handle-web-url --install` @@ -61,4 +61,4 @@ You can install the protocol handler by running the command: `deadline handle-we ## Security -When creating a queue, provide the name of the an S3 bucket in the same account and region as the queue you are creating, and provide a 'root prefix' name for files to be uploaded to. You also must provide an IAM role that has access to the S3 bucket. See the [security best practices][ja-security] documentation for more information on securely configuring job attachments. \ No newline at end of file +When creating a queue, provide the name of the an S3 bucket in the same account and region as the queue you are creating, and provide a 'root prefix' name for files to be uploaded to. You also must provide an IAM role that has access to the S3 bucket. See the [security best practices][ja-security] documentation for more information on securely configuring job attachments.