Support Forge API v3

[johndlong]

Newer revisions of Puppet Enterprise appear to utilize a different API model that break the current proxy implementation.

# puppet master --version
3.4.3 (Puppet Enterprise 3.2.3)

# puppet module search apache
Notice: Searching http://10.103.77.65:8888 ...
Error: Could not execute operation for 'apache'
  The server being queried was /v3/modules?query=apache
  The HTTP response we received was '404 Not Found'
    Check the author and module names are correct.
Error: Try 'puppet help module search' for usage

If I change the repository back to the default, it now goes to forgeapi.puppetlabs.com

# puppet module search apache
Notice: Searching https://forgeapi.puppetlabs.com ...

[drrb]

Thanks for the report. The same is true of Puppet 3.6, I think. This one's been at the top of the todo list for a while, and it's a bit daunting. I'm hoping to support both APIs, but initial attempts have been tough because of Puppet Library's proxy support. These are the options I'm thinking of:

• Support API versions 1 and 3 from the front end and back end. This will keep existing installations working as they currently are, and add support for Puppet 3.6+ and Puppet Enterprise 3.2+. For v3 API calls on forges proxying v1 forges, we would need to either:
  • cut down the v3 responses to exclude information not available from v1
  • fill the v3 responses with fake data
  • download the actual modules for each query using v1, and inspect them to get the missing data for the v3 API, which would be unusably slow
• Support API v1 and v3 front the front end, but only v3 on the back end. This will break the setup of anyone proxying a Pulp forge or Puppet Library forge, but people proxying the Puppet Forge could switch to using forgeapi.puppetlabs.com

I guess the deciding factor should be how prevalent proxies of v1 forges are. If few people are using that feature, I think it's reasonable to increment the major version and remove support for proxying v1 forges. If the feature is widely used, I'd like to give the first option another try to see if it's feasible.

If you urgently need this to work, a workaround you could try is installing Puppet open source on the box with Puppet Enterprise (using the system Ruby), and then use the open source one to run puppet module search/install against your Puppet Library forge. We ran into this problem trying to use librarian-puppet with Puppet Enterprise, and that was the workaround we used. Admittedly, it was annoying to have two puppet's installed on the one machine, but it might tide you over until Puppet Library supports v3.

[s7726]

Has there been any progress on this issue? Is there a workaround to force agents to work with v1?

I think the ultimate goal would be a hybrid approach that could serve v3 info to a v1 client and vise versa. Assumptions would have to be made for missing data in the ladder case.

I would be immediately interested in a v3 client interface in order to support v3 agents. Currently I have both local modules and would like a caching proxy for the traditional forge.puppetlabs. I may start hacking this in to at least get basic functionality. Currently the v3.6.2 puppet agent will not work with puppet library due to it requesting via v3.

[drrb]

I haven't had time to work on this yet, but I'm very aware that more and more people are wanting to switch to 3.6. Feel free to open a pull request if you get something working, otherwise I'll keep you posted when I get some time to work on it.

[s7726]

If you can call it a start I have "started" on an attempt at the beginnings of this. s7726/puppet-library@13e800ef310cea13878f105a0a7f6d483f076bcd

I'm learning Ruby as I go so let me know If I'm totally doing something wrong. Aside from the fact that it currently returns v1 data to the v3 url.

[i11]

Hello!

Just to let you guys know, I've started another project at https://github.com/unibet/puppet-forge-server with an aim to support both v1 and v3 APIs. The main reason for forking it was the difficulty to add v3 support to the current code base. It felt like a lot of code had to be re-written anyway. If you feel like projects could be merge I would gladly pursuit it.

In the mean time I took the liberty of using some of your code and approaches. Please let me know if you feel like there is a lack of credit to puppet-library in any class or file.

Cheers!

[drrb]

Hi i11,

Congratulations on your new project. It looks good! Regarding attribution, please make sure that you add me to the copyright header of any files containing code from puppet-library. Also, if your project is based on puppet-library, please note that it must be released under the GPL (because puppet-library itself is released under the GPL). Please let me know if you have questions about it.

Good luck with the project :)

[jskarpe]

Any progress on this issue?

[walidshaari]

Yuav, there is https://github.com/jbronn/django-forge, and pulp 2.5 both claim to support the v1 and v3 api