vendir support for "dot" paths without conflict

[caphrim007]

Describe the problem/challenge you have

vendir's requirement that paths do not overlap makes it difficult, or impossible, to create a directory structure that custom ytt modules can be found in if there are also environment specific yamls which need to be included.

consider the following.

• a _ytt_lib directory needs to be included from a git location
• some number of blobs of yaml (which i'll just call "modules" for now) need to be included
• some number of environment specific yaml one-off files need to be included.

The final directory structure that we're looking to create wants to look like this

├── config
    ├── _ytt_lib
    │   └── ambassador-mappings
    │       └── deployment.yaml
    ├── modules
    │   ├── base
    │   │   ├── defaults.yaml
    │   │   ├── deployment.yaml
    │   │   ├── namespace.yaml
    │   │   ├── service-account.yaml
    │   │   └── service.yaml
    │   ├── cdm-subscription
    │   │   └── deployment.yaml
    │   └── ...
    ├── secret.enc.yaml
    ├── terraform.yaml
    └── values.yaml

The above directory layout does not seem to be possible due to a constraint in vendir; Expected to not manage overlapping paths: 'config/modules' and 'config'

The vendir.yml file we're making using of looks roughly like the following.

---
apiVersion: "vendir.k14s.io/v1alpha1"
kind: "Config"
directories:
  - path: "config"
    contents:
      - path: "."
        git:
          url: "https://gitlab.foo.bar/vss-cloud-accounts-service-infra.git"
          ref: "v1.28.1"
        includePaths:
          - "k8s/envs/stg-01/secret.enc.yaml"
        newRootPath: "k8s/envs/stg-01"
  - path: "config/service"
    contents:
      - path: "modules"
        git:
          url: "https://gitlab.foo.bar/vss-cloud-accounts-service-infra.git"
          ref: "v1.28.1"
        includePaths:
          - "k8s/modules/base/*"
          - "k8s/modules/cdm-subscription/*"
          - "k8s/modules/cloud-resources/*"
          - "k8s/modules/cloud-account-credentials/*"
          - "k8s/modules/csp/*"
          - "k8s/modules/data-reconciler/*"
          - "k8s/modules/flagger-canary/*"
          - "k8s/modules/inventory-new-accounts/*"
          - "k8s/modules/irsa-service-account-annotation/*"
          - "k8s/modules/k8s-collector/*"
          - "k8s/modules/kapp-config/*"
          - "k8s/modules/legacy-changes/*"
          - "k8s/modules/pod-disruption-budget/*"
          - "k8s/modules/service-annotations/*"
          - "k8s/modules/service-endpoints/*"
          - "k8s/modules/wavefront/*"
        newRootPath: "k8s/modules"
  - path: "config/_ytt_lib"
    contents:
      - path: "ambassador-mappings"
        git:
          url: "https://gitlab.foo.bar/vss-cloud-accounts-service-infra.git"
          ref: "v1.28.1"
        includePaths:
          - "k8s/modules/ambassador-mappings/*"
        newRootPath: "k8s/modules/ambassador-mappings"

the first blob is the blob at issue because it wants to add content to ".", but "." conflicts with the overlay constraint.

Describe the solution you'd like

I'd like vendir to allow this use case similar to how gilt allows it. For example, the following gilt configuration produces the desired result.

---
- git: https://gitlab.foo.bar/vss-cloud-accounts-service-infra.git
  version: v1.28.1
  files:
    - src: k8s/modules
      dst: config/modules
      post_commands:
        - rm -rf ambassador-mappings
    - src: k8s/modules/ambassador-mappings
      dst: config/_ytt_lib/ambassador-mappings
    - src: k8s/envs/stg-01/*
      dst: config/
      post_commands:
        - rm -rf vendir.yml

but gilt itself has its own constraints in terms of functionality. constraints that vendir doesnt have.

Anything else you would like to add:

part of the issue here, as far as I understand things, has to do with the ytt requirement that the _ytt_lib directory must be a sibling of any yaml files that use content in that directory.

If this ytt constraint did not exist, then there would be a clear path forward in how to use vendir. We could create any number of sub-directories and then ytt could either be told where it's _ytt_lib is, or, ytt could walk up the tree until it found a suitable _ytt_lib to use.

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[caphrim007]

context about this issue is in the kubernetes carvel thread here https://kubernetes.slack.com/archives/CH8KCCKA5/p1680541035054199

[Boojapho]

I had a similar use case where I wanted to add a helm chart to . and add a subchart to ./charts/xxx from a different source. I cannot currently do this in vendir.

I understand vendir doesn't know how to handle conflicts. If ignorePaths contains the path defined in other entries, there would be no conflict. That could be a requirement to avoid issues.

[nicholasmhughes-smax]

I had a similar use case where I wanted to add a helm chart to . and add a subchart to ./charts/xxx from a different source.

Ran into this exact use case today. Would love to see this issue worked. @renuy, if the team has an opinion on the best solution, I can potentially get one of my developers on it.