[Proposal] _dev folder

[ycombinator]

We are starting to implement test runners for various types of package tests in the elastic-package tool: elastic/elastic-package#15, elastic/elastic-package#16, etc. This is bringing up the need to keep files that are needed only at development time somewhere under the package root folder. Examples of such files could be test golden files, docker compose files, etc. As such files would be needed at package development time only, we would not want them to be bundled into the package when it is published to the package registry.

So I'm proposing one change to the package spec and one change to the spec format itself:

• Enhancing the package spec to allow for a new _dev folder that could exist under the package root folder or under a dataset's folder, so essentially:

  <packageRootFolder>/
    _dev/
    dataset/
      <dataset>/
        _dev/
  

  Conceptually, _dev is similar to the _meta folders we see in Beats. However, I went with _dev to make it more explicit that the files under this folder are intended for development-time only. That said, I'm not married to the name _dev; happy to consider other names or other ideas that achieve the goal of creating a space for development-time-only files.

• Adding an optional field in the package spec format called visibility. This field is applicable to file or folder elements in the specification format. It can have values such as private or public, defaulting to public. It indicates whether the file or folder being defined in the spec should exist in the public version of the package, i.e. the one available from the package registry (visibility: public) or not (visibility: private).

  Accordingly, the aforementioned _dev folder specification would be set to visibility: private in the spec.

[ycombinator]

@ruflin @mtojek @jonathan-buttner Thoughts on this proposal?

[mtojek]

I like the approach of keeping a side _dev folder with extra stuff. My original ideas was to keep extra files and directory as other resources (e.g. test directory next to manifest files) and use sth like .packageignore file with resources to skip (test, Dockerfile, etc, or just skip known names while building with elastic-package build).

Regarding the name, I don't discuss :) Both _dev and _meta looks good to me.

[ruflin]

++ on this proposal.

I wonder if you really need to add it to the package spec or could go with the convention that anything starting with _ is assumed to be private?

Inside the _dev directory there will also be quite a bit of conventions on how to name test files, structure etc. Will this be part of the package spec or not? I would probably exclude it for now to still be able to iterate on it quickly

[mtojek]

Inside the _dev directory there will also be quite a bit of conventions on how to name test files, structure etc. Will this be part of the package spec or not? I would probably exclude it for now to still be able to iterate on it quickly

I think it would be fine to "version" it once first packages with test samples are pushed to the repository.

[ycombinator]

I wonder if you really need to add it to the package spec or could go with the convention that anything starting with _ is assumed to be private?

It's a fair question. I think the only time it makes sense to add something to the spec — as opposed to relying on conventions — is if there is some code somewhere depending on a certain structure for that something. In the case of the _dev folder, the specific use case that led to it was the elastic-package test command's implementation. The implementation will look for a certain folder structure under the _dev/test/system folder in a dataset folder (the complete hierarchy is shown here). So in this case I thought it would be good to add the _dev folder and certain expected sub-folders under it to the spec.

Inside the _dev directory there will also be quite a bit of conventions on how to name test files, structure etc. Will this be part of the package spec or not? I would probably exclude it for now to still be able to iterate on it quickly.

I actually think it's better to define it in the spec as we see it now and then modify it as needed. Better that IMO than having each package follow slightly different naming conventions and then try to corral them all into a single spec later when we have several packages.

Also, as mentioned in the previous paragraph, the elastic-package code will be relying on a certain structure anyway so packages won't be able to deviate too much from this structure and still work with the elastic-package tool anyway. So we might as well make it part of the spec IMO.

The nice thing is that the spec version is 1.0.0-alpha1 right now so we have the flexibility to change it as we wish for some time longer. Until things settle down we can keep it at this version. So with this "protection" in place, I would rather define more in the spec than less.

[ruflin]

@ycombinator SGTM. What I want to prevent, is that the spec is slowing us down on the implementation side and that we worry making breaking changes also after the package spec itself went to 1.0.0. We had quite a few iterations on the package spec and I think we are getting in the direction of 1.0.0. But on the _dev side, everything is still fresh and will change.

[mtojek]

I'm going with this part.

[mtojek]

The _dev folder will evolve with package development, so I believe we can close this one.