Layering of templated yaml guidance

[eriknelson]

Use Case

I'm working to build several different LXC based container images, which are
defined with a yaml spec. Becuase I'll have a library of these images, often
with a lot of shared configuration, I need a solution for building this shared
library and the ultimate manifests while keeping things DRY. I'm used to
kustomize and other templating solutions, but the real need here is just yaml,
and I need something more sophisticated than yq. ytt looks like it is perfect
for my needs.

Current layout

Project that is stripped down to the core elements of the question with a
reproducible case.

link

make coredns-manifest is what I'm hoping to be my complete target.

make _base-manifest & make _common-manifest will each compile yaml fragments
of each layer independently.

Templates

I'm having some trouble groking the idiomatic means of doing this with ytt
and determining what the correct features would be to use to accomplish this
(functions, modules, templates, or overlays)?

This is not an issue that really has anything to do with distrobuilder, so
anything to do with that is generally out of scope for this issue. Here's
how I'm thinking about it:

• Base: With LXC, there are a handful of base distros that can be utilized as the
  base of the image. The schema of its specification is the same, but the values
  differ, so I believe this is a natural fit for schema+values files, and
  templating.
• Common: With each base image, I'm going to want some basic set of packages that are
  just my preference for working inside of the container, maybe I have some
  scripts I want to install etc.
• Workload: This is the layer that brings in the specifics of the purpose
  of the image, usually configuration and the specific package. CoreDNS falls
  into this bucket.

As i"m putting this together, each layer is probably going to have templated
fields that I'd like to keep in a values file, and verify with a values
schema file. Each layer ends up being composed of a schema + values + template
file. Each of these can, and should get rendered independent of one
another. A lower layer shouldn't have any knowledge of layers that will
be overlayed on top of it.

Overlays

The ultimate file is a composition of these layers on top of one another,
ultimately ending up at the final image definition. I'd like to be able to
overlay them with descending precedence:

workload > common > base

I will also need the ability to annotate my desired merging behavior sometimes
on an array node basis, i.e. concat array items, concat array items but drop
any duplication (this is effectively a set) etc.

Problems & Questions

• First of all, am I thinking about this in the correct manner? I do NOT want
  to "fight the framework", so I'd like to do this in the idiomatic ytt manner.
• It's not clear to me what order these files should be passed when running
  the command; should I be passing all schemas in order, then all values, then
  all the templates, or should I be passing schema > values > template in order
  of each layer?
• I'm getting errors either way, but currently I'm attempting to render each
  "triplicate" in order of least specific to most specific. I'm running into
  the following error with this approach:

# make coredns-manifest
ytt \
        -f base/schema/distro.yml \
        -f base/values/alpine.distro.yml \
        -f base/templates/distro.yml \
        -f common/schema/common.yml \
        -f common/values/alpine.common.yml \
        -f common/templates/common.yml \
        -f img/coredns/img.yml
ytt: Error: Overlaying data values schema (in following order: distro.yml, common.yml):
  Document on line common.yml:2:
    Map item (key 'common_packages') on line common.yml:3:
      Expected number of matched nodes to be 1, but was 0
make: *** [Makefile:4: coredns-manifest] Error 1

The details are in the linked project, but I suspect the issue here is it's
trying to apply the aggregation of all the schemas first, and because the
distro.yml file has none of the nodes that the common.yml schema declares
it's producing the error. I'm not sure about this though since the error
is really quite opaque. I'm also not certain I have a grasp over the ytt
fundamentals at a depth that's required to accomplish what I'm trying to do.
However, as mentioned before, there's no reason that I should have higher layers
bleed their information into lower layers; the entire point is for a lower
layer to have no knowledge over what's getting layered on top of it.

I'm 99% sure that ytt is the right tool for this job, I'm just hitting a wall
trying to wrap my head around the "right" manner to do it. Is someone able
to point me in the right direction? TYIA.