Layer env vars are not processed in layer name order at build time

[edmorley]

Summary

The Buildpack API spec defines what should happen if multiple layers within the same buildpack contribute the same buildpack provided env var.

For example:

Override

...

• For environment variable file contents originating from the same buildpack, file contents that are later (when sorted alphabetically ascending by associated layer name) MUST override file contents that are earlier.

Prepend

...

• Environment variable file contents originating from the same buildpack MUST be sorted alphabetically descending by associated layer name.

See:
https://github.com/buildpacks/spec/blob/main/buildpack.md#environment-variable-modification-rules

lifecycle currently implements the spec correctly at launch time, however, at build time this layer name ordering is not honoured and worse, is actually non-deterministic.

This is a significant issue for buildpacks where it is unavoidable that there are several executables with the same name on PATH. For example, a Python buildpack where there is a main Python install and a virtual environment, where the virtual environment's python needs to be ahead of the main python on PATH.

The cause of this issue is the iteration of a Go map here:

lifecycle/buildpack/build.go

Line 212 in a43d599

for path, layerMetadataFile := range createdLayers {

...since in Go the iteration order of a map is not guaranteed:

The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next.

(see https://go.dev/ref/spec#For_range)

Compare this to the launch time implementation, which iterates over the slice returned from os.ReadDir (which returns its results sorted by filename):

lifecycle/launch/launcher.go

Line 100 in a43d599

if err := eachLayer(bpDir, l.doLayerRoot()); err != nil {

---

Reproduction

Steps

1. mkdir testcase && cd $_ && touch project.toml
2. Edit the project.toml to have the contents below
3. echo "Values during build..." && for i in {1..5}; do pack build testcase --builder heroku/builder:24 --pull-policy if-not-present | grep -E "^(PATH|MYVAR)="; done
4. echo "Values at launch..." && for i in {1..5}; do docker run --rm testcase -- bash -c 'printenv | grep -E "^(PATH|MYVAR)="'; done

project.toml

[_]
schema-version = "0.2"

[[io.buildpacks.group]]
id = "bp"

  [io.buildpacks.group.script]
  api = "0.10"
  shell = "/bin/bash"
  inline = '''
set -euo pipefail
for layer in {a..c}; do
  layer_dir="${CNB_LAYERS_DIR}/${layer}"
  mkdir -p "${layer_dir}/bin"
  mkdir -p "${layer_dir}/env"
  echo -e "[types]\nbuild = true\nlaunch = true" > "${layer_dir}.toml"
  echo "${layer}" > "${layer_dir}/env/MYVAR.override"
done
'''

[[io.buildpacks.group]]
id = "bp2"

  [io.buildpacks.group.script]
  api = "0.10"
  shell = "/bin/bash"
  inline = '''
set -euo pipefail
printenv | grep -E "^(PATH|MYVAR)="
'''

Current behavior

Non-deterministic values that don't match the corresponding values at launch time (or the spec):

Values during build...
MYVAR=b
PATH=/layers/bp/b/bin:/layers/bp/a/bin:/layers/bp/c/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MYVAR=b
PATH=/layers/bp/b/bin:/layers/bp/a/bin:/layers/bp/c/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MYVAR=c
PATH=/layers/bp/c/bin:/layers/bp/b/bin:/layers/bp/a/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MYVAR=a
PATH=/layers/bp/a/bin:/layers/bp/c/bin:/layers/bp/b/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MYVAR=c
PATH=/layers/bp/c/bin:/layers/bp/b/bin:/layers/bp/a/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Values at launch...
MYVAR=c
PATH=/layers/bp/c/bin:/layers/bp/b/bin:/layers/bp/a/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MYVAR=c
PATH=/layers/bp/c/bin:/layers/bp/b/bin:/layers/bp/a/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MYVAR=c
PATH=/layers/bp/c/bin:/layers/bp/b/bin:/layers/bp/a/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MYVAR=c
PATH=/layers/bp/c/bin:/layers/bp/b/bin:/layers/bp/a/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MYVAR=c
PATH=/layers/bp/c/bin:/layers/bp/b/bin:/layers/bp/a/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Expected behavior

For the values during the build to:

• Be deterministic, instead of changing every build
• To match the values seen at launch
• To match the spec

---

Context

lifecycle version

0.20.0

platform version(s)

Pack 0.34.2+git-ce8db3c.build-6005

[natalieparellano]

Thanks for reporting @edmorley. We'll look into fixing this in the next patch