feat: add zarf dev deploy for quickly testing packages and restruct…

…ure `zarf prepare` (#2170)

## Description

Adds a `zarf dev deploy` command that will create + deploy a local
package in one shot.

## Related Issue

Fixes #2169 
Fixes #2098 

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Other (security config, docs update, etc)

## Checklist before merging

- [x] Test, docs, adr added or updated as needed
- [x] [Contributor Guide
Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow)
followed

---------

Signed-off-by: razzle <[email protected]>
Co-authored-by: Wayne Starr <[email protected]>

README.md

@@ -41,7 +41,7 @@ Zarf eliminates the [complexity of air gap software delivery](https://www.itopst
 - Built-in Docker registry
 - Builtin [K9s Dashboard](https://k9scli.io/) for managing a cluster from the terminal
 - [Mutating Webhook](adr/0005-mutating-webhook.md) to automatically update Kubernetes pod's image path and pull secrets as well as [Flux Git Repository](https://fluxcd.io/docs/components/source/gitrepositories/) URLs and secret references
-- Builtin [command to find images](https://docs.zarf.dev/docs/the-zarf-cli/cli-commands/zarf_prepare_find-images) and resources from a Helm chart
+- Builtin [command to find images](https://docs.zarf.dev/docs/the-zarf-cli/cli-commands/zarf_dev_find-images) and resources from a Helm chart
 - Tunneling capability to [connect to Kubernetes resources](https://docs.zarf.dev/docs/the-zarf-cli/cli-commands/zarf_connect) without network routing, DNS, TLS or Ingress configuration required
 
 ## 🛠️ Configurable Features

adr/0022-dev-cmd.md

@@ -0,0 +1,42 @@
+# 22. Introduce `dev` command
+
+Date: 2023-12-03
+
+## Status
+
+Accepted
+
+## Context
+
+> Feature request: <https://github.com/defenseunicorns/zarf/issues/2169>
+
+The current package development lifecycle is:
+
+1. Create a `zarf.yaml` file and add components
+2. Create a package with `zarf package create <dir>`
+3. Debug any create errors and resolve by editing `zarf.yaml` and repeating step 2
+4. Run `zarf init` to initialize the cluster
+5. Deploy the package with `zarf package deploy <tarball>`
+6. Debug any deploy errors and resolve by editing `zarf.yaml` and repeating step 2 or 5
+
+If there are deployment errors, the common pattern is to reset the cluster (ex: `k3d cluster delete && k3d cluster create`) and repeat steps 4-6. Re-initializing the cluster, recreating the package, and redeploying the package is tedious and time consuming; especially when the package is large or the change was small.
+
+`zarf package create` is designed around air-gapped environments where the package is created in one environment and deployed in another. Due to this architecture, a package's dependencies _must_ be retrieved and assembled _each_ and _every_ time.
+
+There already exists the concept of [`YOLO` mode](0010-yolo-mode.md), which can build + deploy a package without the need for `zarf init`, and builds without fetching certain heavy dependencies (like Docker images). However, `YOLO` mode is not exposed via CLI flags, and is meant to develop and deploy packages in fully connected environments.
+
+## Decision
+
+Introduce a `dev deploy` command that will combine the lifecycle of `package create` and `package deploy` into a single command. This command will:
+
+- Not result in a re-usable tarball / OCI artifact
+- Not have any interactive prompts
+- Not require `zarf init` to be run by default (override-able with a CLI flag)
+- Be able to create+deploy a package in either YOLO mode (default) or prod mode (exposed via a CLI flag)
+- Only build + deploy components that _will_ be deployed (contrasting with `package create` which builds _all_ components regardless of whether they will be deployed)
+
+## Consequences
+
+The `dev deploy` command will make it easier to develop and deploy packages in connected **development** environments. It will also make it easier to debug deployment errors by allowing the user to iterate on the package without having to re-initialize the cluster, re-build and re-deploy the package each cycle.
+
+There is a purpose to placing this functionality behind a new command, rather than adding it to `package create`. Commands under `dev` are meant to be used in **development** environments, and are **not** meant to be used in **production** environments. There is still the possibility that a user will use `dev deploy` in a production environment, but the command name and documentation will make it clear that this is not the intended use case.

docs/0-zarf-overview.md

@@ -148,7 +148,7 @@ In this use case, you configure Zarf to initialize a cluster that already exists
 - Builtin Docker registry
 - Builtin [K9s Dashboard](https://k9scli.io/) for managing a cluster from the terminal
 - [Mutating Webhook](adr/0005-mutating-webhook.md) to automatically update Kubernetes pod's image path and pull secrets as well as [Flux Git Repository](https://fluxcd.io/docs/components/source/gitrepositories/) URLs and secret references
-- Builtin [command to find images](./2-the-zarf-cli/100-cli-commands/zarf_prepare_find-images.md) and resources from a Helm chart
+- Builtin [command to find images](./2-the-zarf-cli/100-cli-commands/zarf_dev_find-images.md) and resources from a Helm chart
 - Tunneling capability to [connect to Kuberenetes resources](./2-the-zarf-cli/100-cli-commands/zarf_connect.md) without network routing, DNS, TLS or Ingress configuration required
 
 ### 🛠️ Configurable Features

docs/2-the-zarf-cli/100-cli-commands/zarf.md

@@ -31,8 +31,8 @@ zarf COMMAND [flags]
 * [zarf completion](zarf_completion.md)	 - Generate the autocompletion script for the specified shell
 * [zarf connect](zarf_connect.md)	 - Accesses services or pods deployed in the cluster
 * [zarf destroy](zarf_destroy.md)	 - Tears down Zarf and removes its components from the environment
+* [zarf dev](zarf_dev.md)	 - Commands useful for developing packages
 * [zarf init](zarf_init.md)	 - Prepares a k8s cluster for the deployment of Zarf packages
 * [zarf package](zarf_package.md)	 - Zarf package commands for creating, deploying, and inspecting packages
-* [zarf prepare](zarf_prepare.md)	 - Tools to help prepare assets for packaging
 * [zarf tools](zarf_tools.md)	 - Collection of additional tools to make airgap easier
 * [zarf version](zarf_version.md)	 - Shows the version of the running Zarf binary

docs/2-the-zarf-cli/100-cli-commands/zarf_prepare.md → docs/2-the-zarf-cli/100-cli-commands/zarf_dev.md

@@ -1,12 +1,12 @@
-# zarf prepare
+# zarf dev
 <!-- Auto-generated by hack/gen-cli-docs.sh -->
 
-Tools to help prepare assets for packaging
+Commands useful for developing packages
 
 ## Options
 
 ```
-  -h, --help   help for prepare
+  -h, --help   help for dev
 ```
 
 ## Options inherited from parent commands
@@ -25,9 +25,10 @@ Tools to help prepare assets for packaging
 ## SEE ALSO
 
 * [zarf](zarf.md)	 - DevSecOps for Airgap
-* [zarf prepare find-images](zarf_prepare_find-images.md)	 - Evaluates components in a zarf file to identify images specified in their helm charts and manifests
-* [zarf prepare generate-config](zarf_prepare_generate-config.md)	 - Generates a config file for Zarf
-* [zarf prepare lint](zarf_prepare_lint.md)	 - Verifies the package schema
-* [zarf prepare patch-git](zarf_prepare_patch-git.md)	 - Converts all .git URLs to the specified Zarf HOST and with the Zarf URL pattern in a given FILE.  NOTE:
+* [zarf dev deploy](zarf_dev_deploy.md)	 - [beta] Creates and deploys a Zarf package from a given directory
+* [zarf dev find-images](zarf_dev_find-images.md)	 - Evaluates components in a Zarf file to identify images specified in their helm charts and manifests
+* [zarf dev generate-config](zarf_dev_generate-config.md)	 - Generates a config file for Zarf
+* [zarf dev lint](zarf_dev_lint.md)	 - Verifies the package schema
+* [zarf dev patch-git](zarf_dev_patch-git.md)	 - Converts all .git URLs to the specified Zarf HOST and with the Zarf URL pattern in a given FILE.  NOTE:
 This should only be used for manifests that are not mutated by the Zarf Agent Mutating Webhook.
-* [zarf prepare sha256sum](zarf_prepare_sha256sum.md)	 - Generates a SHA256SUM for the given file
+* [zarf dev sha256sum](zarf_dev_sha256sum.md)	 - Generates a SHA256SUM for the given file

docs/2-the-zarf-cli/100-cli-commands/zarf_dev_deploy.md

@@ -0,0 +1,41 @@
+# zarf dev deploy
+<!-- Auto-generated by hack/gen-cli-docs.sh -->
+
+[beta] Creates and deploys a Zarf package from a given directory
+
+## Synopsis
+
+[beta] Creates and deploys a Zarf package from a given directory, setting options like YOLO mode for faster iteration.
+
+```
+zarf dev deploy [flags]
+```
+
+## Options
+
+```
+      --components string                  Comma-separated list of components to install.  Adding this flag will skip the init prompts for which components to install
+      --create-set stringToString          Specify package variables to set on the command line (KEY=value) (default [])
+      --deploy-set stringToString          Specify deployment variables to set on the command line (KEY=value) (default [])
+  -f, --flavor string                      The flavor of components to include in the resulting package (i.e. have a matching or empty "only.flavor" key)
+  -h, --help                               help for deploy
+      --no-yolo                            Disable the YOLO mode default override and create / deploy the package as-defined
+      --registry-override stringToString   Specify a map of domains to override on package create when pulling images (e.g. --registry-override docker.io=dockerio-reg.enterprise.intranet) (default [])
+```
+
+## Options inherited from parent commands
+
+```
+  -a, --architecture string   Architecture for OCI images and Zarf packages
+      --insecure              Allow access to insecure registries and disable other recommended security enforcements such as package checksum and signature validation. This flag should only be used if you have a specific reason and accept the reduced security posture.
+  -l, --log-level string      Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info")
+      --no-color              Disable colors in output
+      --no-log-file           Disable log file creation
+      --no-progress           Disable fancy UI progress bars, spinners, logos, etc
+      --tmpdir string         Specify the temporary directory to use for intermediate files
+      --zarf-cache string     Specify the location of the Zarf cache directory (default "~/.zarf-cache")
+```
+
+## SEE ALSO
+
+* [zarf dev](zarf_dev.md)	 - Commands useful for developing packages

docs/2-the-zarf-cli/100-cli-commands/zarf_prepare_find-images.md → docs/2-the-zarf-cli/100-cli-commands/zarf_dev_find-images.md

@@ -1,16 +1,16 @@
-# zarf prepare find-images
+# zarf dev find-images
 <!-- Auto-generated by hack/gen-cli-docs.sh -->
 
-Evaluates components in a zarf file to identify images specified in their helm charts and manifests
+Evaluates components in a Zarf file to identify images specified in their helm charts and manifests
 
 ## Synopsis
 
-Evaluates components in a zarf file to identify images specified in their helm charts and manifests.
+Evaluates components in a Zarf file to identify images specified in their helm charts and manifests.
 
 Components that have repos that host helm charts can be processed by providing the --repo-chart-path.
 
 ```
-zarf prepare find-images [ PACKAGE ] [flags]
+zarf dev find-images [ PACKAGE ] [flags]
 ```
 
 ## Options
@@ -37,4 +37,4 @@ zarf prepare find-images [ PACKAGE ] [flags]
 
 ## SEE ALSO
 
-* [zarf prepare](zarf_prepare.md)	 - Tools to help prepare assets for packaging
+* [zarf dev](zarf_dev.md)	 - Commands useful for developing packages

docs/2-the-zarf-cli/100-cli-commands/zarf_prepare_generate-config.md → docs/2-the-zarf-cli/100-cli-commands/zarf_dev_generate-config.md

@@ -1,4 +1,4 @@
-# zarf prepare generate-config
+# zarf dev generate-config
 <!-- Auto-generated by hack/gen-cli-docs.sh -->
 
 Generates a config file for Zarf
@@ -13,7 +13,7 @@ Accepted extensions are json, toml, yaml.
 NOTE: This file must not already exist. If no filename is provided, the config will be written to the current working directory as zarf-config.toml.
 
 ```
-zarf prepare generate-config [ FILENAME ] [flags]
+zarf dev generate-config [ FILENAME ] [flags]
 ```
 
 ## Options
@@ -37,4 +37,4 @@ zarf prepare generate-config [ FILENAME ] [flags]
 
 ## SEE ALSO
 
-* [zarf prepare](zarf_prepare.md)	 - Tools to help prepare assets for packaging
+* [zarf dev](zarf_dev.md)	 - Commands useful for developing packages

docs/2-the-zarf-cli/100-cli-commands/zarf_prepare_lint.md → docs/2-the-zarf-cli/100-cli-commands/zarf_dev_lint.md

@@ -1,4 +1,4 @@
-# zarf prepare lint
+# zarf dev lint
 <!-- Auto-generated by hack/gen-cli-docs.sh -->
 
 Verifies the package schema
@@ -8,7 +8,7 @@ Verifies the package schema
 Verifies the package schema and warns the user if they have variables that won't be evaluated
 
 ```
-zarf prepare lint [ DIRECTORY ] [flags]
+zarf dev lint [ DIRECTORY ] [flags]
 ```
 
 ## Options
@@ -32,4 +32,4 @@ zarf prepare lint [ DIRECTORY ] [flags]
 
 ## SEE ALSO
 
-* [zarf prepare](zarf_prepare.md)	 - Tools to help prepare assets for packaging
+* [zarf dev](zarf_dev.md)	 - Commands useful for developing packages

docs/2-the-zarf-cli/100-cli-commands/zarf_prepare_patch-git.md → docs/2-the-zarf-cli/100-cli-commands/zarf_dev_patch-git.md

@@ -1,11 +1,11 @@
-# zarf prepare patch-git
+# zarf dev patch-git
 <!-- Auto-generated by hack/gen-cli-docs.sh -->
 
 Converts all .git URLs to the specified Zarf HOST and with the Zarf URL pattern in a given FILE.  NOTE:
 This should only be used for manifests that are not mutated by the Zarf Agent Mutating Webhook.
 
 ```
-zarf prepare patch-git HOST FILE [flags]
+zarf dev patch-git HOST FILE [flags]
 ```
 
 ## Options
@@ -30,4 +30,4 @@ zarf prepare patch-git HOST FILE [flags]
 
 ## SEE ALSO
 
-* [zarf prepare](zarf_prepare.md)	 - Tools to help prepare assets for packaging
+* [zarf dev](zarf_dev.md)	 - Commands useful for developing packages

docs/2-the-zarf-cli/100-cli-commands/zarf_prepare_sha256sum.md → docs/2-the-zarf-cli/100-cli-commands/zarf_dev_sha256sum.md

@@ -1,10 +1,10 @@
-# zarf prepare sha256sum
+# zarf dev sha256sum
 <!-- Auto-generated by hack/gen-cli-docs.sh -->
 
 Generates a SHA256SUM for the given file
 
 ```
-zarf prepare sha256sum { FILE | URL } [flags]
+zarf dev sha256sum { FILE | URL } [flags]
 ```
 
 ## Options
@@ -29,4 +29,4 @@ zarf prepare sha256sum { FILE | URL } [flags]
 
 ## SEE ALSO
 
-* [zarf prepare](zarf_prepare.md)	 - Tools to help prepare assets for packaging
+* [zarf dev](zarf_dev.md)	 - Commands useful for developing packages

docs/2-the-zarf-cli/2-zarf-config-files.md

@@ -8,7 +8,7 @@ import FetchFileCodeBlock from '@site/src/components/FetchFileCodeBlock';
 
 Users can use a config file to easily control flags for `zarf init`, `zarf package create`, and `zarf package deploy` commands, as well as global flags (excluding `--confirm`), enabling a more straightforward and declarative workflow.
 
-Zarf supports config files written in common configuration file formats including `toml`, `json`, `yaml`, `ini` and `props`, and by default Zarf will look for a file called `zarf-config` with one of these filenames in the current working directory.  To generate a blank config file you can run `zarf prepare generate-config` with an optional output filename/format.  For example, to create an empty config file with the `my-cool-env` in the yaml format, you can use `zarf prepare generate-config my-cool-env.yaml`.
+Zarf supports config files written in common configuration file formats including `toml`, `json`, `yaml`, `ini` and `props`, and by default Zarf will look for a file called `zarf-config` with one of these filenames in the current working directory.  To generate a blank config file you can run `zarf dev generate-config` with an optional output filename/format.  For example, to create an empty config file with the `my-cool-env` in the yaml format, you can use `zarf dev generate-config my-cool-env.yaml`.
 
 To use a custom config filename, set the `ZARF_CONFIG` environment variable to the config file's path. For example, to use the `my-cool-env.yaml` config file in the current working directory, you can set the `ZARF_CONFIG` environment variable to `my-cool-env.yaml`. The `ZARF_CONFIG` environment variable can be set either in the shell or in a `.env` file in the current working directory. Note that the `ZARF_CONFIG` environment variable takes precedence over the default config file path.
 

docs/2-the-zarf-cli/index.md

@@ -34,6 +34,6 @@ The `zarf package create` command is used to create a Zarf package from a `zarf.
 
 :::tip
 
-When deploying and managing packages you may find the sub-commands under `zarf prepare` useful to find resources and manipulate package definitions as needed.
+When developing packages you may find the sub-commands under `zarf dev` useful to find resources and manipulate package definitions.
 
 :::

docs/3-create-a-zarf-package/1-zarf-packages.md

@@ -68,6 +68,7 @@ When executed, the `zarf package create` command locates the `zarf.yaml` file in
 The process of defining and creating a package is also elaborated on in detail in the [Creating a Zarf Package Tutorial](../5-zarf-tutorials/0-creating-a-zarf-package.md).
 
 ### Creating Differential Packages
+
 If you already have a Zarf package and you want to create an updated package you would normally have to re-create the entire package from scratch, including things that might not have changed. Depending on your workflow, you may  want to create a package that only contains the artifacts that have changed since the last time you built your package. This can be achieved by using the `--differential` flag while running the `zarf package create` command. You can use this flag to point to an already built package you have locally or to a package that has been previously [published](../5-zarf-tutorials/7-publish-and-deploy.md#publish-package) to a registry.
 
 ## Inspecting a Created Package

docs/3-create-a-zarf-package/10-dev.md

@@ -0,0 +1,57 @@
+# Developing Zarf Packages
+
+## `dev` Commands
+
+Zarf contains many commands that are useful while developing a Zarf package to iterate on configuration, discover resources and more!  Below are explanations of some of these commands with the full list discoverable with `zarf dev --help`.
+
+:::caution
+
+The `dev` commands are meant to be used in **development** environments / workflows. They are **not** meant to be used in **production** environments / workflows.
+
+:::
+
+### `dev deploy`
+
+The `dev deploy` command will combine the lifecycle of `package create` and `package deploy` into a single command. This command will:
+
+- Not result in a re-usable tarball / OCI artifact
+- Not have any interactive prompts
+- Not require `zarf init` to be run (by default, but _is required_ if `--no-yolo` is not set)
+- Be able to create+deploy a package in either YOLO mode (default) or prod mode (exposed via `--no-yolo` flag)
+- Only build + deploy components that _will_ be deployed (contrasting with `package create` which builds _all_ components regardless of whether they will be deployed)
+
+```bash
+# Create and deploy dos-games in yolo mode
+$ zarf dev deploy examples/dos-games
+```
+
+```bash
+# If deploying a package in prod mode, `zarf init` must be run first
+$ zarf init --confirm
+# Create and deploy dos-games in prod mode
+$ zarf dev deploy examples/dos-games --no-yolo
+```
+
+### `dev find-images`
+
+Evaluates components in a `zarf.yaml` to identify images specified in their helm charts and manifests.
+
+Components that have `git` repositories that host helm charts can be processed by providing the `--repo-chart-path`.
+
+```bash
+$ zarf dev find-images examples/wordpress
+
+components:
+
+  - name: wordpress
+    images:
+      - docker.io/bitnami/apache-exporter:0.13.3-debian-11-r2
+      - docker.io/bitnami/mariadb:10.11.2-debian-11-r21
+      - docker.io/bitnami/wordpress:6.2.0-debian-11-r18
+```
+
+### Misc `dev` Commands
+
+Not all `dev` commands have been mentioned here.
+
+Further `dev` commands can be discovered by running `zarf dev --help`.

docs/3-create-a-zarf-package/2-zarf-components.md

@@ -116,11 +116,11 @@ Kustomizations are handled a bit differently than normal manifests in that Zarf
 
 <Properties item="ZarfComponent" include={["images"]} />
 
-Images can either be discovered manually, or automatically by using [`zarf prepare find-images`](../2-the-zarf-cli/100-cli-commands/zarf_prepare_find-images.md).
+Images can either be discovered manually, or automatically by using [`zarf dev find-images`](../2-the-zarf-cli/100-cli-commands/zarf_dev_find-images.md).
 
 :::note
 
-`zarf prepare find-images` will find images for most standard manifests, kustomizations, and helm charts, however some images cannot be discovered this way as some upstream resources (like operators) may bury image definitions inside.  For these images, `zarf prepare find-images` also offers support for the draft [Helm Improvement Proposal 15](https://github.com/helm/community/blob/main/hips/hip-0015.md) which allows chart creators to annotate any hidden images in their charts along with the [values conditions](https://github.com/helm/community/issues/277) that will cause those images to be used.
+`zarf dev find-images` will find images for most standard manifests, kustomizations, and helm charts, however some images cannot be discovered this way as some upstream resources (like operators) may bury image definitions inside.  For these images, `zarf dev find-images` also offers support for the draft [Helm Improvement Proposal 15](https://github.com/helm/community/blob/main/hips/hip-0015.md) which allows chart creators to annotate any hidden images in their charts along with the [values conditions](https://github.com/helm/community/issues/277) that will cause those images to be used.
 
 :::