Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RubyGems.org v1 dependency API deprecation #8

Closed
headius opened this issue May 2, 2023 · 4 comments
Closed

RubyGems.org v1 dependency API deprecation #8

headius opened this issue May 2, 2023 · 4 comments
Milestone
2.0.0

Comments

@headius
Copy link
Member

headius commented May 2, 2023

We learned recently that the v1 version of the dependency-lookup API on RubyGems.org was deprecated on February 23, slated for complete removal on (at time of writing) May 10.

This API is used by the mavengem plugin to fetch a list of dependencies for gem artifacts, and is critical to this plugin. When the API goes away–if we are unable to update, test, and release a new version of mavengems–JRuby's build and any downstream builds that fetch Ruby gems as Maven dependencies will cease to function.

I have joined an issue here, basically begging for more time to make this transition: rubygems/rubygems.org#3698 (comment)

This came to our attention, finally, due to the intermittent "brown outs" that RubyGems.org has deployed in the last month. That led to a revival of this jruby-maven-plugins issue: jruby/jruby-maven-plugins#124 (comment)

Had we noticed the original deprecation blog post two months ago we might have had time to make this transition. Now, with only 8 days left and a major Ruby conference coming up (RubyKaigi on the 11th), I don't see how we can fix this before the API shuts down altogether.

I am begging the RubyGems.org maintainers for additional time to make this transition. Meanwhile we need to move quickly to try to fix our uses of the deprecated API (and make sure we're not using any others).
@headius headius mentioned this issue May 3, 2023
Closed
@Axel-1
Copy link

Axel-1 commented May 15, 2023

Hello,
Do you have any updates about the migration to API V2?
We use mavengem for our software and we can no longer build our project.

Thank!

@headius
Copy link
Member Author

headius commented Jul 13, 2023

@Axel-1 Sorry for the late reply. The RubyGems servers have whitelisted our use of this API from around mid-May until mid-August. We still need to update the plugin for the new API and get a release out (ASAP).

@headius headius mentioned this issue Jul 13, 2023
Merged
headius added a commit to headius/mavengem that referenced this issue Jul 27, 2023
@headius
Hook up compact and v2 to get most things working
3359dcf 
This is the big drop to get all the major features working
properly atop the v2 and compact index APIs rather than the now-
defunct v1 dependencies API.

I was guessing my way through this codebase a bit but the new APIs
are hooked up as follows:

* The v1 dependencies API (dependencies?gems=foo,bar) has been
  mostly replaced with the compact index API. This API provides
  all versions of one specified gem. Any features that depend on
  the dependencies API to get multiple gems at once are still
  broken and will need to iterate over all given gems using the
  compact index.
* The creation of gemspecs and poms from the old gzipped spec
  format has been moved to the v2 gem info API.

All tests are passing in mavengem-protocol, which only depends on
features like poms and gems, all of which have been moved to the
new APIs.

This is close to addressing jruby#8.
headius added a commit to headius/mavengem that referenced this issue Jul 28, 2023
@headius
Update dependencies to build a snapshot
e74ea43 
Group ID is changed to "com.headius" because I do not have push
privileges for "org.torquebox.mojo".

Inter-library dependencies switched to use "com.headius" for
snapshot deploy, but build still uses the previous release of
mavengem extension (did not seem to like me using snapshot
extensions from the same project, or at least refused to resolve
them.

maven-tools updated to latest to fix a bug in gem logic that tried
to gsub! on an empty string that now appears to be coming in
frozen:

jruby/maven-tools@1188743#diff-e4394efc5b28617f42311627dd8af1193c01d1a39b939959085c6836f3e9e5b7R149

With these changes, I have successfully been able to:

* Deploy 1.0.4-SNAPSHOT versions of all three artifacts
* Show JRuby build failing with a patched 1.0.3 that uses a
  non-"Java" User-Agent when contacting rubygems.org (causing the
  v1 deps API to reject requests and gems fail to resolve.
* Show JRuby build working with the same User-Agent patch but the
  new logic to use v2 and compact APIs.

Nearly complete solution for jruby#8 but not all features in
rubygems-tools are updated and not all v1 deps API and gz data
file uses have been purged yet.
@headius
Copy link
Member Author

headius commented Jul 31, 2023

An update on this:

I have in #9 managed to wire up the v2/rubygems API and part of the compact index API used by bundler to replace most of the uses of the v1/dependencies API.

The changes are nearly all in rubygems-tools, which is where the bulk of the integration code lives. All tests in mavengem-protocol and mavengem-wagon pass with the new code. Most tests pass in rubygems-tools; the known failures are due to the "bundler API" features and other API endpoints still dependent on getting multiple gems' worth of data using the form dependencies?gems=rails,sequel.

I confirmed that the new mavengem works with the disabled v1/dependencies API by doing the following:

  • Run with an unmodified build of the updated and cleaned-up mavengem 1.0.4-SNAPSHOT. Gems were retrieved successfully.
  • Run with a modified build that removes "Java" from the request User-Agent, preventing our requests from getting whitelisted. Gems failed to be retrieved with the "not found" errors users reported this spring during the API brownouts.
  • Run with a build of the PR that both removes "Java" from the User-Agent and includes the new code for v2 and compact API. Gems were retrieved successfully.

So in a pinch, we could do a release with the known issue that the "bundler API" feature is temporarily broken. That would ensure JRuby and other mavengem users can still resolve dependencies past the August 8 shut-down of the v1/dependencies API.

Note we have moved this library from the "org.torquebox.mojo" group ID to the "org.jruby.maven" group ID. Torquebox is defunct now and we do not have push rights to that group anyway. Going forward, mavengem maintenance is done by the JRuby team.

@headius headius added this to the 2.0 milestone Aug 2, 2023
headius added a commit to headius/jruby-maven-plugins that referenced this issue Aug 3, 2023
@headius
Migrate to JRuby group ID and get things passing
f0cb867 
This makes the following changes:

* de.saumya.mojo maven group ID changes to org.jruby.maven in all
  published artifacts
* mavengem is used for all gem retrieval instead of the defunct
  TorqueBox proxy server
* mavengem and related dependencies are now sourced from the
  org.jruby.maven 2.0.0 versions
* minor additional changes to get tests passing with the new
  artifacts

This passes all unit and integration tests and deploys
successfully as org.jruby.maven:jruby-maven-plugins:3.0.0-SNAPSHOT
and should now be immune to the v1/dependencies rubygems.org API
shutdown (jruby/mavengem#8).
@headius headius mentioned this issue Aug 3, 2023
Merged
headius added a commit to headius/jruby-maven-plugins that referenced this issue Aug 3, 2023
@headius
Migrate to JRuby group ID and get things passing
ef3a632 
This makes the following changes:

* de.saumya.mojo maven group ID changes to org.jruby.maven in all
  published artifacts
* mavengem is used for all gem retrieval instead of the defunct
  TorqueBox proxy server
* mavengem and related dependencies are now sourced from the
  org.jruby.maven 2.0.0 versions
* minor additional changes to get tests passing with the new
  artifacts

This passes all unit and integration tests and deploys
successfully as org.jruby.maven:jruby-maven-plugins:3.0.0-SNAPSHOT
and should now be immune to the v1/dependencies rubygems.org API
shutdown (jruby/mavengem#8).
headius added a commit to headius/jruby-maven-plugins that referenced this issue Aug 3, 2023
@headius
Migrate to JRuby group ID and get things passing
5c43659 
This makes the following changes:

* de.saumya.mojo maven group ID changes to org.jruby.maven in all
  published artifacts
* mavengem is used for all gem retrieval instead of the defunct
  TorqueBox proxy server
* mavengem and related dependencies are now sourced from the
  org.jruby.maven 2.0.0 versions
* minor additional changes to get tests passing with the new
  artifacts

This passes all unit and integration tests and deploys
successfully as org.jruby.maven:jruby-maven-plugins:3.0.0-SNAPSHOT
and should now be immune to the v1/dependencies rubygems.org API
shutdown (jruby/mavengem#8).
@headius
Copy link
Member Author

headius commented Aug 8, 2023

Resolved by #9.

@headius headius closed this as completed Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.0.0
Development

No branches or pull requests
2 participants
@headius @Axel-1