Supporting Puppet 6

[alexjfisher]

We want to start marking our modules as puppet 6 compatible as soon as possible. We should probably not change all modules en masse, but as soon as someone has at least done some light testing or if the module has acceptance tests, contributors need to be aware of what their PRs should and should not include. The puppet 6 release notes and blog don't make this very clear. Hopefully these notes will help.

metadata.json

version_requirement

Update the version_requirement. We will likely start dropping puppet 4 support in the new year, but right now, the majority of modules will have a requirements section like this.

"requirements": [
    {
      "name": "puppet",
      "version_requirement": ">= 4.10.0 < 7.0.0"
    }
  ]

dependencies

Do not add any of the new puppetlabs/X_core "Supported types in modules" to the dependencies section. These modules are already included with the puppet 6 agent packages. Importantly, these modules "are available during compilation, but won't be pluginsync'ed.". This is fortunate as they are all marked as supporting puppet 6 only, and at least one module, (yumrepo_core), definitely breaks puppet 5 as it uses the sensitive DSL which isn't available before puppet 6.

Testing

fixtures.yml

core modules used need to be added to fixtures.yml as (unlike the agent packages), the puppet gem doesn't vendor the modules. On the assumption that puppetlabs/puppetlabs_spec_helper#258 will be merged and a new release made soon, mark any core modules as only being needed on puppet 6 as follows.

---
fixtures:
  repositories:
    stdlib: 'https://github.com/puppetlabs/puppetlabs-stdlib.git'
    yumrepo_core:
      repo: 'https://github.com/puppetlabs/puppetlabs-yumrepo_core.git'
      puppet_version: ">= 6.0.0"

beaker

TODO

Questions we'd like to ask Puppet and/or things we're not sure about

• Why are all the new puppetlabs/X_core modules marked as puppet 6 only? To use the Types not repackaged into Puppet agent or Deprecated types these have to be installed onto the modulepath. At this point they will be pluginsynced to all agents, including puppet 5 agents that aren't EOL any time soon. See https://tickets.puppetlabs.com/plugins/servlet/mobile#issue/PUP-9052
• Could pluginsync become more intelligent and somehow not sync modules that aren't compatible??

[alexjfisher]

@bastelfreak @dhollinger
This is what I've come up with so far. Once we nail it down, we could move this to https://voxpupuli.org/

[dhollinger]

The beaker section probably doesn't need a lot of work since you can use one of the following patterns:

• *_core are built into the puppet6 agent meaning we don't need to really do anything different for beaker tests
• Force remove puppet6-only "dependencies" when the BEAKER_PUPPET_COLLECTION is set to anything other than puppet6
• For non-AIO puppet tests, write code in the spec_helper to install the *_core modules as part of startup/RSpec.configure block

[bastelfreak]

Some thoughts (I don't have enough time at the moment to think properly about all this, sorry):

• At the point where a module requires a newer version of a *_core module than what is vendored, it needs to be added to the dependency section
• Therefor I think it would be more consistent to include those modules always as dependency. That would also make it easier for people running non-AIO versions, that might not vendor the modules
• That would require it to work properly in an env where a *_core module is on the puppetmaster and a puppet 5 agent is in the same env

[dhollinger]

@bastelfreak The major problem right now with adding *_core modules as a dependency is that they will break testing for anyone using the AIO package, which is a majority of our users. I think that we need to focus on supporting the users first via the official Puppet way and then second for users (like Arch users) that either don't or can't use the AIO packages.

[alexjfisher]

Even for distributions not bundling Puppet's ruby/openssl etc, I'd still expect them to include the same _core modules as Puppet Inc vendor in their own packages.

[dhollinger]

@alexjfisher True, but that is ultimately up to the distro package maintainers

[joshcooper]

Thanks for writing this up @alexjfisher. I had a few comments:

version_requirement

Supporting 4.10.x and up, but less than 7 makes sense.

dependencies

I tend to agree with @bastelfreak that as soon as someone installs the latest version of a _core module, then it will be pluginsync'ed etc. So I'm 👍 to keeping the dependencies section, even though it means using the latest module version, and not the version from the AIO package. As we start to make more changes/improvements to the core modules, I can see folks wanting to use the latest version of the module (to pick up bug fixes/improvements) before the module is bumped in puppet-agent and released.

fixtures.yml

https://tickets.puppetlabs.com/browse/MODULES-7856 was merged and released in puppetlabs_spec_helper 2.11.0, so optional dependencies are now supported.

X_core modules marked as puppet 6 only

AFAIK, only yumrepo_core is truly not compatible with puppet5, and we decided to make puppet6 the minimum lower bound across the board for consistency. However, given all of the grief the yumrepo sensitive issue has called, I'm inclined to update yumrepo so that it only tries to use the new sensitive DSL method when running on Puppet 6, otherwise have it fallback to the older way of marking the parameter as sensitive. This affects the puppetlabs-puppet_agent module as well, since it relies on yumrepo. And since it's used to upgrade puppet agents, the lack of backwards compatibility is a problem.

Filed as https://tickets.puppetlabs.com/browse/MODULES-8114

Pluginsync

Unfortunately pluginsync is rather dumb, copying everything everywhere, flattening modules so custom facts, types, etc from different modules can collide. We have thoughts about making this better, but there's nothing tangible that would help this issue.

[binford2k]

Copied a comment from @Dan33l on MODULES-7856

What is it planed about metadata.json ?
For instance, how to manage types not repackaged into Puppet agent like k5login. The doc says "type contained in modules that are maintained, but are not repackaged into Puppet agent. If you need to use them, you must install the modules separately."
The module puppetlabs/k5login_core does not support puppet < 6.
And so does it exist a special syntax in metadata.json to be able to install this module only on puppet6 server ?
Or, as the only available solution, is it needed to explain this big mess in the README.md of the module ?

[ekohl]

I think by now we can consider this fixed.