Template files (#1171)

docs/4-user-guide/3-zarf-schema.md

@@ -1518,9 +1518,9 @@ Must be one of:
 | -------- | -------- |
 | **Type** | `string` |
 
-| Restrictions                      |                                                                         |
-| --------------------------------- | ----------------------------------------------------------------------- |
-| **Must match regular expression** | ```^[A-Z_]+$``` [Test](https://regex101.com/?regex=%5E%5BA-Z_%5D%2B%24) |
+| Restrictions                      |                                                                               |
+| --------------------------------- | ----------------------------------------------------------------------------- |
+| **Must match regular expression** | ```^[A-Z0-9_]+$``` [Test](https://regex101.com/?regex=%5E%5BA-Z0-9_%5D%2B%24) |
 
 </blockquote>
 </details>
@@ -1620,9 +1620,9 @@ Must be one of:
 | -------- | -------- |
 | **Type** | `string` |
 
-| Restrictions                      |                                                                         |
-| --------------------------------- | ----------------------------------------------------------------------- |
-| **Must match regular expression** | ```^[A-Z_]+$``` [Test](https://regex101.com/?regex=%5E%5BA-Z_%5D%2B%24) |
+| Restrictions                      |                                                                               |
+| --------------------------------- | ----------------------------------------------------------------------------- |
+| **Must match regular expression** | ```^[A-Z0-9_]+$``` [Test](https://regex101.com/?regex=%5E%5BA-Z0-9_%5D%2B%24) |
 
 </blockquote>
 </details>

examples/data-injection/manifest.yaml

@@ -21,7 +21,7 @@ spec:
             [
               "sh",
               "-c",
-              'while [ ! -f /test/###ZARF_DATA_INJECTON_MARKER### ]; do echo "waiting for zarf data sync" && sleep 1; done; echo "we are done waiting!"',
+              'while [ ! -f /test/###ZARF_DATA_INJECTION_MARKER### ]; do echo "waiting for zarf data sync" && sleep 1; done; echo "we are done waiting!"',
             ]
           resources:
             requests:
@@ -52,7 +52,7 @@ spec:
             exec:
               command:
                 - "cat"
-                - "/test/###ZARF_DATA_INJECTON_MARKER###"
+                - "/test/###ZARF_DATA_INJECTION_MARKER###"
             initialDelaySeconds: 1
             periodSeconds: 1
 

examples/package-variables/README.md

@@ -1,11 +1,17 @@
 # Package Variables
 
-This example demonstrates how to define `variables` and `constants` in your package that will be templated across the manifests and charts that your package uses during `zarf package deploy` with `###ZARF_VAR_*###` and `###ZARF_CONST_*###` respectively.  It also shows how package-level variables can be used in the `zarf.yaml` during `zarf package create` with `###ZARF_PKG_VAR_*###`.
+This example demonstrates how to define `variables` and `constants` in your package that will be templated across the manifests and charts that your package uses during `zarf package deploy` with `###ZARF_VAR_*###` and `###ZARF_CONST_*###` respectively. It also shows how package-level variables can be used in the `zarf.yaml` during `zarf package create` with `###ZARF_PKG_VAR_*###`.
 
 With this templating feature, you can define values in the zarf.yaml file without having to define them in every manifest and chart, and can prompt the deployer for certain information that may be dynamic on `zarf package deploy`.
 
 This becomes useful when you are working with an upstream chart that is often changing, or a lot of charts that have slightly different conventions for their values. Now you can standardize all of that from your zarf.yaml file.
 
+Text files are also templated during `zarf package deploy` so you can use these variables in any text file that you want to be templated.
+
+:::note
+Because files can be deployed without a Kubernetes cluster, some built-in variables such as `###ZARF_REGISTRY###` may not be available if no previous component has required access to the cluster. If you need one of these built-in variables, a prior component will need to have been called that requires access to the cluster, such as `images`, `repos`, `manifests`, `dataInjections`.
+:::
+
 :::info
 
 To view the example source code, select the `Edit this page` link below the article and select the parent folder.
@@ -15,6 +21,7 @@ To view the example source code, select the `Edit this page` link below the arti
  
 
 ## How to Use Deploy-Time Variables and Constants
+
 The 'placeholder' text in the manifest or chart yaml should have your desired key name in all caps with `###ZARF_VAR` prepended and `###` appended for `variables` or `###ZARF_CONST` prepended and `###` appended for `constants`.
 
 For example, if I wanted to create a template for a database username (using the variable `name`: `DATABASE_USERNAME`) I would do something like `###ZARF_VAR_DATABASE_USERNAME###` in the manifest or chart yaml.
@@ -24,7 +31,7 @@ In the zarf.yaml you would add the name of the variable in the `variables` secti
 ```yaml
 variables:
   name: DATABASE_USERNAME
-  description: "The username for the database"
+  description: 'The username for the database'
 ```
 
 :::note
@@ -44,7 +51,7 @@ For variables, you can also specify a `default` value for the variable to take i
 ```yaml
 variables:
   name: DATABASE_USERNAME
-  default: "postgres"
+  default: 'postgres'
   prompt: true
 ```
 
@@ -54,12 +61,12 @@ Variables that do not have a default, are not `--set` and are not prompted for d
 
 :::
 
-For constants, you must specify the value they will use at package create.  These values cannot be overridden with `--set` during `zarf package deploy`, but you can use package variables (described below) to variablize them during create.
+For constants, you must specify the value they will use at package create. These values cannot be overridden with `--set` during `zarf package deploy`, but you can use package variables (described below) to variablize them during create.
 
 ```yaml
 constants:
   name: DATABASE_TABLE
-  value: "users"
+  value: 'users'
 ```
 
 ::note
@@ -70,23 +77,23 @@ constants:
 
 ## How to Use Create-Time Package Variables
 
-You can also specify variables at package create time by including `###_ZARF_PKG_VAR_*###` in your package definition's string values.  These values are discovered during `zarf package create` and will be prompted for if not using `--confirm` or `--set`.  An example of this is below:
+You can also specify variables at package create time by including `###_ZARF_PKG_VAR_*###` in your package definition's string values. These values are discovered during `zarf package create` and will be prompted for if not using `--confirm` or `--set`. An example of this is below:
 
 ```yaml
 kind: ZarfPackageConfig
 metadata:
-  name: "pkg-variables"
-  description: "Prompt for a variables during package create"
+  name: 'pkg-variables'
+  description: 'Prompt for a variables during package create'
 
 constants:
   - name: PROMPT_IMAGE
-    value: "###ZARF_PKG_VAR_PROMPT_ON_CREATE###"
+    value: '###ZARF_PKG_VAR_PROMPT_ON_CREATE###'
 
 components:
   - name: zarf-prompt-image
     required: true
     images:
-      - "###ZARF_PKG_VAR_PROMPT_ON_CREATE###"
+      - '###ZARF_PKG_VAR_PROMPT_ON_CREATE###'
 ```
 
 :::note

examples/package-variables/simple-terraform.tf

@@ -0,0 +1,14 @@
+provider "aws" {
+  region = "###ZARF_VAR_AWS_REGION###"
+}
+
+terraform {
+  required_version = ">= 0.13"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 3.37.0"
+    }
+  }
+}

examples/package-variables/zarf.yaml

@@ -10,15 +10,17 @@ constants:
 
 # Demonstrates injecting custom variables into a K8s resource, e.g. ###ZARF_VAR_DOG###
 variables:
-  - name: "WOLF"
-  - name: "DOG"
+  - name: WOLF
+  - name: DOG
     default: "woof"
-  - name: "CAT"
+  - name: CAT
     description: "What sound does a cat make?"
     prompt: true
-  - name: "FOX"
+  - name: FOX
     default: "###ZARF_PKG_VAR_CONFIG_MAP###"
     prompt: true
+  - name: AWS_REGION
+    default: "us-east-1"
 
 components:
   # Note that you must specify the ACTION and CONFIG_MAP i.e. `--set ACTION=template --set CONFIG_MAP=simple-configmap.yaml` during package create
@@ -29,3 +31,10 @@ components:
       - name: variable-example-configmap
         files:
           - "###ZARF_PKG_VAR_CONFIG_MAP###"
+
+  - name: file-template-variable-example
+    description: "Change a value in a regular file with a Zarf variable. Set AWS_REGION variable to modify the file."
+    required: true
+    files:
+      - source: simple-terraform.tf
+        target: modified-terraform.tf

packages/distros/k3s/common/k3s.service

@@ -25,4 +25,4 @@ RestartSec=5s
 ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
 ExecStartPre=-/sbin/modprobe br_netfilter
 ExecStartPre=-/sbin/modprobe overlay
-ExecStart=/usr/sbin/k3s server --write-kubeconfig-mode=700 --write-kubeconfig /root/.kube/config --disable traefik
+ExecStart=/usr/sbin/k3s server --write-kubeconfig-mode=700 --write-kubeconfig /root/.kube/config ###ZARF_VAR_K3S_ARGS###

packages/distros/k3s/common/zarf.yaml

@@ -2,6 +2,11 @@ kind: ZarfInitConfig
 metadata:
   name: "distro-k3s"
 
+variables:
+  - name: K3S_ARGS
+    description: "Arguments to pass to K3s"
+    default: "--disable traefik"
+
 components:
   - name: k3s
     only:

src/internal/packager/helm/post-render.go

@@ -69,7 +69,9 @@ func (r *renderer) Run(renderedManifests *bytes.Buffer) (*bytes.Buffer, error) {
 	}
 
 	// Run the template engine against the chart output
-	template.ProcessYamlFilesInPath(tempDir, r.options.Component, r.values)
+	if _, err := template.ProcessYamlFilesInPath(tempDir, r.options.Component, r.values); err != nil {
+		return nil, fmt.Errorf("error templating the helm chart: %w", err)
+	}
 
 	// Read back the templated file contents
 	buff, err := os.ReadFile(path)

src/internal/packager/template/template.go

@@ -64,17 +64,23 @@ func (values Values) GetRegistry() string {
 }
 
 // Apply renders the template and writes the result to the given path.
-func (values Values) Apply(component types.ZarfComponent, path string) {
+func (values Values) Apply(component types.ZarfComponent, path string, ignoreReady bool) error {
 	message.Debugf("template.Apply(%#v, %s)", component, path)
 
-	if !values.Ready() {
-		// This should only occur if the state couldn't be pulled or on init if a template is attempted before the pre-seed stage
-		message.Fatalf(nil, "template.Apply() called before template.Generate()")
+	// If Apply() is called before all values are loaded, fail unless ignoreReady is true
+	if !values.Ready() && !ignoreReady {
+		return fmt.Errorf("template.Apply() called before template.Generate()")
 	}
 
 	regInfo := values.config.State.RegistryInfo
 	gitInfo := values.config.State.GitServer
 
+	depMarkerOld := "DATA_INJECTON_MARKER"
+	depMarkerNew := "DATA_INJECTION_MARKER"
+	deprecations := map[string]string{
+		depMarkerOld: depMarkerNew,
+	}
+
 	builtinMap := map[string]string{
 		"STORAGE_CLASS": values.config.State.StorageClass,
 
@@ -91,9 +97,10 @@ func (values Values) Apply(component types.ZarfComponent, path string) {
 	}
 
 	// Include the data injection marker template if the component has data injections
-	// TODO: (@jeff-mccoy) DATA_INJECTON_MARKER is misspelled however it is in use by other packages and would be a breaking change!
 	if len(component.DataInjections) > 0 {
-		builtinMap["DATA_INJECTON_MARKER"] = config.GetDataInjectionMarker()
+		// Preserve existing misspelling for backwards compatibility
+		builtinMap[depMarkerOld] = config.GetDataInjectionMarker()
+		builtinMap[depMarkerNew] = config.GetDataInjectionMarker()
 	}
 
 	// Don't template component-specific variables for every component
@@ -131,5 +138,7 @@ func (values Values) Apply(component types.ZarfComponent, path string) {
 	}
 
 	message.Debugf("templateMap = %#v", templateMap)
-	utils.ReplaceTextTemplate(path, templateMap)
+	utils.ReplaceTextTemplate(path, templateMap, deprecations)
+
+	return nil
 }

src/internal/packager/template/yaml.go

@@ -12,14 +12,16 @@ import (
 )
 
 // ProcessYamlFilesInPath iterates over all yaml files in a given path and performs Zarf templating + image swapping.
-func ProcessYamlFilesInPath(path string, component types.ZarfComponent, values Values) []string {
+func ProcessYamlFilesInPath(path string, component types.ZarfComponent, values Values) ([]string, error) {
 	// Only pull in yml and yaml files
 	pattern := regexp.MustCompile(`(?mi)\.ya?ml$`)
 	manifests, _ := utils.RecursiveFileList(path, pattern)
 
 	for _, manifest := range manifests {
-		values.Apply(component, manifest)
+		if err := values.Apply(component, manifest, false); err != nil {
+			return nil, err
+		}
 	}
 
-	return manifests
+	return manifests, nil
 }

src/internal/packager/validate/validate.go

@@ -154,7 +154,7 @@ func validatePackageName(subject string) error {
 }
 
 func validatePackageVariable(subject types.ZarfPackageVariable) error {
-	isAllCapsUnderscore := regexp.MustCompile(`^[A-Z_]+$`).MatchString
+	isAllCapsUnderscore := regexp.MustCompile(`^[A-Z0-9_]+$`).MatchString
 
 	// ensure the variable name is only capitals and underscores
 	if !isAllCapsUnderscore(subject.Name) {
@@ -165,7 +165,7 @@ func validatePackageVariable(subject types.ZarfPackageVariable) error {
 }
 
 func validatePackageConstant(subject types.ZarfPackageConstant) error {
-	isAllCapsUnderscore := regexp.MustCompile(`^[A-Z_]+$`).MatchString
+	isAllCapsUnderscore := regexp.MustCompile(`^[A-Z0-9_]+$`).MatchString
 
 	// ensure the constant name is only capitals and underscores
 	if !isAllCapsUnderscore(subject.Name) {

src/pkg/packager/deploy.go

@@ -192,7 +192,7 @@ func (p *Packager) deployComponent(component types.ZarfComponent, noImgChecksum
 		return charts, fmt.Errorf("unable to run the 'before' scripts: %w", err)
 	}
 
-	if err := p.processComponentFiles(component.Files, componentPath.Files); err != nil {
+	if err := p.processComponentFiles(component, componentPath.Files); err != nil {
 		return charts, fmt.Errorf("unable to process the component files: %w", err)
 	}
 
@@ -243,16 +243,16 @@ func (p *Packager) deployComponent(component types.ZarfComponent, noImgChecksum
 }
 
 // Move files onto the host of the machine performing the deployment.
-func (p *Packager) processComponentFiles(componentFiles []types.ZarfFile, sourceLocation string) error {
+func (p *Packager) processComponentFiles(component types.ZarfComponent, sourceLocation string) error {
 	// If there are no files to process, return early.
-	if len(componentFiles) < 1 {
+	if len(component.Files) < 1 {
 		return nil
 	}
 
-	spinner := *message.NewProgressSpinner("Copying %d files", len(componentFiles))
+	spinner := *message.NewProgressSpinner("Copying %d files", len(component.Files))
 	defer spinner.Stop()
 
-	for index, file := range componentFiles {
+	for index, file := range component.Files {
 		spinner.Updatef("Loading %s", file.Target)
 		sourceFile := filepath.Join(sourceLocation, strconv.Itoa(index))
 
@@ -267,9 +267,23 @@ func (p *Packager) processComponentFiles(componentFiles []types.ZarfFile, source
 		// Replace temp target directories
 		file.Target = strings.Replace(file.Target, "###ZARF_TEMP###", p.tmp.Base, 1)
 
+		// Check if the file looks like a text file
+		isText, err := utils.IsTextFile(sourceFile)
+		if err != nil {
+			message.Debugf("unable to determine if file %s is a text file: %s", sourceFile, err)
+		}
+
+		// If the file is a text file, template it
+		if isText {
+			spinner.Updatef("Templating %s", file.Target)
+			if err := valueTemplate.Apply(component, sourceFile, true); err != nil {
+				return fmt.Errorf("unable to template file %s: %w", sourceFile, err)
+			}
+		}
+
 		// Copy the file to the destination
 		spinner.Updatef("Saving %s", file.Target)
-		err := copy.Copy(sourceFile, file.Target)
+		err = copy.Copy(sourceFile, file.Target)
 		if err != nil {
 			return fmt.Errorf("unable to copy file %s to %s: %w", sourceFile, file.Target, err)
 		}
@@ -422,7 +436,9 @@ func (p *Packager) installChartAndManifests(componentPath types.ComponentPaths,
 		// zarf magic for the value file
 		for idx := range chart.ValuesFiles {
 			chartValueName := helm.StandardName(componentPath.Values, chart) + "-" + strconv.Itoa(idx)
-			valueTemplate.Apply(component, chartValueName)
+			if err := valueTemplate.Apply(component, chartValueName, false); err != nil {
+				return installedCharts, err
+			}
 		}
 
 		// Generate helm templates to pass to gitops engine