Puppetserver mixes up modules of different environments

[gettalong]

We have been using Puppet for 10+ years now without much troubles on the server sider. However, since updating to Puppet 7 (currently on puppetserver version 7.17.2 on Ubuntu 22.04) we are having troubles with our environments. It seems that the puppetserver uses code from one environment for other environments.

We have two main environments: jammy and noble (for the respective Ubuntu versions). The jammy version worked fine until we introduced the noble environment and used updated modules there. It will probably apply to any module but to illustrate the problem I will use the https://forge.puppet.com/modules/puppetlabs/firewall module.

The jammy environment uses version 4.0.1 of the firewall module which still uses "provider" instead of "protocol". The noble environment uses version 8.0.2 of the firewall module which has transitioned to using "protocol" over "provider".

So when I run systemctl reload puppetserver, wait for it to finish and then run puppet agent -t on a jammy node, I get the following output:

$ puppet agent -t
Info: Using environment 'jammy'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Could not autoload puppet/type/package: Could not autoload puppet/provider/package/snap: no such file to load -- puppet_x/snap/api (file: /etc/puppetlabs/code/environments/jammy/modules/cleanstuff/manifests/init.pp, line: 6, column: 3) on node JAMMY-HOST
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
$ puppet agent -t
Info: Using environment 'jammy'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for JAMMY-HOST
Info: Applying configuration version '1724255357'
Notice: Applied catalog in 8.87 seconds

So I'm not sure while there is the error on the first run but the second and further runs work okay.

If I then run puppet on a noble host, I get this:

$ puppet agent -t
Info: Using environment 'noble'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Notice: Requesting catalog from PUPPETSERVER:8140 (PUPPETIP)
Notice: Catalog compiled by PUPPETSERVER
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'protocol' (file: /etc/puppetlabs/code/environments/noble/modules/profile/manifests/nfs/server.pp, line: 26) on Firewall[202 allow access to rpc.rquotad UDP IPv6] (file: /etc/puppetlabs/code/environments/noble/modules/profile/manifests/nfs/server.pp, line: 26) on node NOBLE-HOST
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

It complains about the invalid parameter "protocol" because the puppetserver somehow uses the firewall module from jammy which still uses "provider". Note that the /opt/puppetlabs/puppet/cache/lib/puppet/type/firewall.rb on the client hosts are the correct ones.

When doing it the other way around, i.e. starting with the noble host, running the puppet agent on a jammy host leads to a failure because now the firewall module from noble is used.

We have looked through the documentation and issues to find a solution for this but were unsuccessful so far. As seen above it looks like the separation of the code of the environments doesn't work but we don't know why. I thought that the configuration option "environment_timeout" might be at fault but it is not changed from its default value of 0.

Has anybody experienced this or something similar?

[elfranne]

How do you deploy code and manage modules versions to the Puppet server?
If using r10k+puppetfile i would manually run r10k puppetfile install (from /etc/puppetlabs/code/environments/jammy and /etc/puppetlabs/code/environments/noble) and investigate the if the deployed versions are correct.
https://www.puppet.com/docs/puppet/7/puppetfile_usage.html

[gettalong]

Thanks! We are using r10k+puppetfile but that doesn't seem to be the problem.

I now started to re-create our environment from scratch, adding one module at a time and running on a test client to see what happens. The first few modules were okay.

Then I installed puppetlabs-concat version 7.4.0. This version defines a parameter named "create_empty_file". In our jammy environment we also have puppetlabs-concat but version 7.3.0 which doesn't have that parameter.

When I now run the agent on the client:

$ puppet agent -t
Info: Using environment 'noble'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Notice: Requesting catalog from PUPPETSERVER:8140 (IP)
Notice: Catalog compiled by PUPPETSERVER
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'create_empty_file' (file: /etc/puppetlabs/code/environments/noble/ext-modules/concat/manifests/init.pp, line: 126) on Concat_file[/etc/default/grub.d/50-custom.cfg] (file: /etc/puppetlabs/code/environments/noble/ext-modules/concat/manifests/init.pp, line: 126) on node NOBLEHOST
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

This shows that the problem doesn't occur because of problems with a module itself but with different versions of modules in different environments.

Searching the internet I found this https://binford2k.com/2015/08/31/environment-leakage/ which suggests that this is common behaviour. However, in the follow-up post https://binford2k.com/2016/12/13/environments-no-longer-leak/ it is described that this is fixed with puppet generate types. I tried this originally but it didn't work. With the now minimal environment and the above error it does work, i.e. after generating the types the error is gone.

So I will try to build up the environment module by module and see how it goes.

[gettalong]

Okay, using the generated types worked now. Not sure, though, why it didn't work originally... However, good enough for me 😄