Merge branch 'main' of github.com:latchbio/latch into kenny/nextflow

docs/source/index.md

@@ -6,11 +6,11 @@ The Latch SDK is an open-source toolchain to define serverless bioinformatics wo
 
 Bioinformatics workflows developed with the SDK automatically receive:
 
-- Instant no-code interfaces for accessibility and publication
-- First class static typing
-- Containerization and versioning of every registered change
-- Reliable and scalable managed cloud infrastructure
-- Single line definition of arbitrary resource requirements (eg. CPU, GPU, Storage) for serverless execution
+* Instant no-code interfaces for accessibility and publication
+* First class static typing
+* Containerization and versioning of every registered change
+* Reliable and scalable managed cloud infrastructure
+* Single line definition of arbitrary resource requirements (eg. CPU, GPU, Storage) for serverless execution
 
 ![SDK Overview](./assets/sdk-intro.png)
 
@@ -24,15 +24,15 @@ Lacth SDK allows developers to upload workflows to the full-featured [Latch Plat
 
 With Latch SDK, developers can write the description to their workflow and customize input parameters using plain Markdown. Latch automatically parses the written text and Python function headers to compile a type-safe UI.
 
-**Specifying arbitrary cloud compute and storage resources for bioinformatics pipelines is difficult.** With the Latch SDK, there are several Python task decorators that easily allow you to define the resources available at runtime. The framework starts at 2 CPUs and 4 GiBs of memory and goes all the way to 95 CPUs, 490 GiBs of memory, 4949 GiB of storage, and 1 GPU (24 GiBs of VRAM, 9,216 CUDA cores) to easily handle all processing needs.
+**Specifying arbitrary cloud compute and storage resources for bioinformatics pipelines is difficult.** With the Latch SDK, there are several Python task decorators that easily allow you to define the resources available at runtime. The framework starts at 2 CPUs and 4 GiBs of memory and goes all the way to 95 CPUs, 490 GiBs of memory, 4949 GiB of storage, and 1 GPU (24 GiBs of VRAM, 9, 216 CUDA cores) to easily handle all processing needs.
 
 **Bioinformatics tools face the challenges of irreproducibility.** The lack of proper versioning and dependencies management results in a long tail of poorly documented and unusable bioinformatics software tools.
 
 Latch SDK containerizes and versions workflow code in the background each time a workflow is registered to the Latch platform. Container images are constructed by parsing autogenerated or user-defined [Dockerfiles](https://docs.docker.com/engine/reference/builder/). Similarly, versions are user-specified as any unique plaintext string. The resulting workflows are portable and scalable, and their behavior is reproducible.
 
 ## What the Latch SDK is not
 
-- **A self-hosted solution**: Currently, you cannot write your workflow using Latch SDK and host it in your own AWS instance or an HPC. The infrastructure serving bioinformatics pipelines is fully managed by Latch. This allows us to rapidly iterate to bring on high quality features, give cost and performance guarantees, and ensure that security is offered out-of-the-box.
+* **A self-hosted solution**: Currently, you cannot write your workflow using Latch SDK and host it in your own AWS instance or an HPC. The infrastructure serving bioinformatics pipelines is fully managed by Latch. This allows us to rapidly iterate to bring on high quality features, give cost and performance guarantees, and ensure that security is offered out-of-the-box.
 
 ## Examples
 
@@ -44,20 +44,21 @@ Visit [Examples](../examples/workflows_examples.md) to see real-world bioinforma
 
 To get started with Latch SDK, view the following resources:
 
-- **[Quickstart](./getting_started/quick_start.md)** is the fastest way to get started with the Latch SDK.
-- **[Concepts](./basics/what_is_a_workflow.md)** describes all important Latch SDK concepts.
-- **[Examples](./examples/workflows_examples.md)** show full examples of using Latch SDK for various bioinformatics pipelines.
-- **[Troubleshooting](./troubleshooting/troubleshooting)** provides a guide to debug common errors.
-- **[Reference](./api/modules.rst)** contains detailed API and design documents.
-- **[Subcommands](./subcommands.md)** contains details about the Latch command line toolchain to register workflows and upload data to Latch.
-- Join the <a href="https://forms.gle/sCjr8tdjzx5HjVW27" target="_blank">SDK open-source community</a> on Slack!
+* **[Quickstart](./getting_started/quick_start.md)** is the fastest way to get started with the Latch SDK.
+* **[Concepts](./basics/what_is_a_workflow.md)** describes all important Latch SDK concepts.
+* **[Examples](./examples/workflows_examples.md)** show full examples of using Latch SDK for various bioinformatics pipelines.
+* **[Troubleshooting](./troubleshooting/troubleshooting)** provides a guide to debug common errors.
+* **[Reference](./api/modules.rst)** contains detailed API and design documents.
+* **[Subcommands](./subcommands.md)** contains details about the Latch command line toolchain to register workflows and upload data to Latch.
+* Join the <a href="https://forms.gle/sCjr8tdjzx5HjVW27" target="_blank">SDK open-source community</a> on Slack!
 
 ---
 
 ```{toctree}
 :hidden:
 :maxdepth: 2
 self
+
 ```
 
 ```{toctree}
@@ -88,6 +89,7 @@ basics/caching
 basics/conditional_section
 basics/map_task
 basics/adding_secrets
+
 ```
 
 ```{toctree}
@@ -105,6 +107,7 @@ basics/remote_execution
 :caption: Publishing a Workflow
 basics/uploading_test_data
 basics/remote_registration
+
 ```
 
 ```{toctree}
@@ -125,6 +128,7 @@ registry/record.md
 automation/overview.md
 automation/example-data-addition.md
 automation/example-interval.md
+
 ```
 
 ```{toctree}
@@ -142,6 +146,7 @@ tutorials/overview
 tutorials/rnaseq
 tutorials/metamage
 tutorials/registry_import
+
 ```
 
 ```{toctree}
@@ -157,15 +162,14 @@ cli/mv
 :hidden:
 :maxdepth: 2
 :caption: Snakemake Integration
+snakemake/overview.md
 snakemake/quickstart.md
-snakemake/tutorial.md
-snakemake/metadata.md
-snakemake/environments.md
-snakemake/lifecycle.md
-snakemake/resources.md
 snakemake/cloud.md
+snakemake/configuration.md
+snakemake/tutorial.md
 snakemake/debugging.md
 snakemake/troubleshooting.md
+
 ```
 
 ```{toctree}

docs/source/snakemake/configuration.md

@@ -0,0 +1,7 @@
+# Advanced Configuration
+
+```{toctree}
+:maxdepth: 2
+configuration/resources.md
+configuration/registries.md
+```

docs/source/snakemake/configuration/registries.md

@@ -0,0 +1,32 @@
+# Private Container Registries
+
+When executing Snakemake workflows in containers, the container images may exist in a private registry that the Latch cloud cannot access. Downloading images from private registries at runtime requires two steps:
+
+1. Upload your private container registry's password/access token to the Latch platform. See [Storing and using Secrets](../basics/adding_secrets.md).
+2. Add the `docker_metadata` field to your workflow's `SnakemakeMetadata` object so the workflow engine knows where to pull your credentials. For example:
+
+```
+# latch_metadata.py
+from latch.types.metadata import SnakemakeMetadata, SnakemakeFileParameter, DockerMetadata
+from latch.types.directory import LatchDir
+from latch.types.metadata import LatchAuthor, LatchMetadata, LatchParameter
+from pathlib import Path
+
+SnakemakeMetadata(
+    display_name="snakemake_tutorial_workflow",
+    author=LatchAuthor(
+            name="latchbio",
+    ),
+    env_config=EnvironmentConfig(
+      use_conda=False,
+      use_container=True,
+    ),
+    docker_metadata=DockerMetadata(
+      username="user0",
+      secret_name="LATCH_SECRET_NAME",
+    ),
+    ...
+)
+```
+
+**Note**: the `secret_name` field specifies the name of the Latch Secret uploaded in step #1, NOT the actual registry password.

docs/source/snakemake/resources.md → docs/source/snakemake/configuration/resources.md

@@ -28,7 +28,7 @@ SnakemakeMetadata(
         name="Your Name",
     ),
     parameters=generated_parameters,
-    cores=8, # added
+    cores=8, # updated
 )
 ```