Solution to rubygems api deprecation for direct api use

[neilharris123]

We are experiencing issues relating to the rubygems api deprecation which has caught us by surprise.

I've read the blog here, but I've not seen a solution for users who are currently using the api directly. Only solutions for users making use of products such as artifactory.

We are building an image which installs ruby, then bundler, then uses bundler to install various other gems.

The step we are having issues is when we attempt to install bundler like so:

gem install bundler -v '~>1.14'

At which point we receive:

 > [5/8] RUN gem install --no-ri --no-rdoc bundler -v '~>1.14':
#10 8.144 ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
#10 8.144     bad response Not Found 404 (https://api.rubygems.org/api/v1/dependencies?gems=bundler)

I have seen the suggestion in the article about to try switching to 'Gem Version API', but there are no links to this in the article, and haven't had any luck googling this. Would it be possible to provide a more detailed explanation, either here, or in the article or both?

[saraid]

The deprecation article doesn't discuss it for some reason, but newer versions of gem and bundler (JFrog's Artifactory identifies the desired versions to be "Gem 3.2.3 or higher; Bundler 1.12 or higher") should be requesting the Versions API directly. This may mean that you'll want to upgrade Ruby, too.

I'm not really sure why the RubyGems.org maintainers felt that deprecating the fallback API would mean people would need to take no action, but it isn't true; you do have to make sure that you're using a sufficiently modern version of the clients, which not everyone is.

[simi]

The deprecation article doesn't discuss it for some reason, but newer versions of gem and bundler (JFrog's Artifactory identifies the desired versions to be "Gem 3.2.3 or higher; Bundler 1.12 or higher") should be requesting the Versions API directly. This may mean that you'll want to upgrade Ruby, too.

@saraid Usually there is no need to upgrade Ruby. Just install newer bundler using gem install bundler. All currently supported Rubies (2.7+) includes bundler 2.1.4 or newer. Why would you need Ruby upgrade?

[saraid]

@saraid Usually there is no need to upgrade Ruby. Just install newer bundler using gem install bundler. All currently supported Rubies (2.7+) includes bundler 2.1.4 or newer. Why would you need Ruby upgrade?

@simi It is possible that people are running unsupported Rubies. When the OP is discussing Bundler 1.14, that does not suggest they are running Ruby 2.7.

[simi]

@saraid Usually there is no need to upgrade Ruby. Just install newer bundler using gem install bundler. All currently supported Rubies (2.7+) includes bundler 2.1.4 or newer. Why would you need Ruby upgrade?

@simi It is possible that people are running unsupported Rubies. When the OP is discussing Bundler 1.14, that does not suggest they are running Ruby 2.7.

Yup, it seems some old Ruby is involved. @neilharris123 which Ruby and RubyGems are related to your report? You can find out by ruby -v and gem -v.

[neilharris123]

@saraid Usually there is no need to upgrade Ruby. Just install newer bundler using gem install bundler. All currently supported Rubies (2.7+) includes bundler 2.1.4 or newer. Why would you need Ruby upgrade?

@simi It is possible that people are running unsupported Rubies. When the OP is discussing Bundler 1.14, that does not suggest they are running Ruby 2.7.

Yup, it seems some old Ruby is involved. @neilharris123 which Ruby and RubyGems are related to your report? You can find out by ruby -v and gem -v.

We're using the following:

ruby -v
ruby 2.1.5p273 (2014-11-13) [x86_64-linux-gnu]

root@0ff4883d2b77:/# gem -v
2.2.2

It's not possible for us to upgrade to the newer versions suggested, as the machines where we're encountering this problem run Debian8, so this puts constraints on the version of ruby we can install. In the medium term we plan to upgrade these machines, but It's not something that we can do in the short term. They are responsible for configuration management of a large part of our infra, and upgrading the operating system and other services would require a fairly substantial re-working of our current configuration management manifests.

[simi]

@neilharris123 all you need to do now is to upgrade to bundler 1.12 or newer. Latest bundler 1.12+ should support your Ruby version - https://rubygems.org/gems/bundler/versions/1.12.6.

Can you try that by doing gem install bundler:~>1.12.6? Alternativelly you can try to download gem file at https://rubygems.org/downloads/bundler-1.12.6.gem and install from local file.

For your Ruby, I recommend to upgrade as soon as possible since you're using version not supported for 6 years now. But I understand it is not easy sometimes.

[headius]

We are also facing a serious breakage due to the abrupt RubyGems.org v1 API removal. What's serious? The entirety of JRuby's build process for installing standard library gems will break until we can update the Maven plugins responsible for it to use the v2 API. These same plugins are used by thousands of JRuby users and dozens of downstream projects to pull gems and repackage them for JVM usage.

We are unlikely to have the manpower before May 10 to update our plugin to the newer API. Even if we did manage to make the update, those downstream projects will need time to upgrade the plugin.

So this is me pleading for more time to make this transition. 🥺🙏

[headius]

See jruby/jruby-maven-plugins#124 for the issue that finally brought this deprecation to our attention.

I want to make clear... we missed the memo on this, and will play catch-up to make the transition. But that will be very difficult if all JRuby builds start to fail when the API goes away on the 10th.

[headius]

I think it's also worth pointing out that this removal date is just days before RubyKaigi, so anyone like me preparing talks for that conference will be unable to build our own projects after next Wednesday.

Please, extend this deadline by at least a month, if not until the end of the summer (6 month rampdown of the API).

[headius]

A follow-up here... We worked out with the rubygems.org maintainers to whitelist JRuby's uses of the dependencies API until August (8th, I believe) so that we'd have time to adjust our build tools to use the new API. That is in process at jruby/mavengem#9 now.

[headius]

The modifications to mavengem in jruby/mavengem#9 are largely complete, and I've confirmed that JRuby's build works properly with mavengem even when altering the user-agent to something other than Java.

That does not really affect this issue but it does mean JRuby will not be affected when the API finally goes dark for Java users next week.

[coldnebo]

can anyone explain what is going on with rubygems? Our performance is really bad because the latest rubygems and bundler keep falling back to the legacy api. I thought the latest versions were supposed to use V2?

We are following this chain of bug reports:

• https://jfrog.com/help/r/jfrog-artifactory-documentation/rubygems-repositories
• https://blog.rubygems.org/2023/02/22/dependency-api-deprecation.html
  "To learn more, read on, or see the original RFC about this deprecation.
• RFC is a broken link.

We have no guidance or support for this issue. Please, can someone help us? Are these not the latest gems?

$ ruby --version
ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
$ gem --version
3.4.21
$ bundler --version
Bundler version 2.4.21

$ time bundle install --verbose
Running `bundle install --verbose` with bundler 2.4.21
Found changes from the lockfile, re-resolving dependencies because the dependencies in your gemfile changed
HTTP GET http://local-rubygems/artifactory/api/gems/gems-repos/versions
HTTP 404 Not Found http://local-rubygems/artifactory/api/gems/gems-repos/versions
Bundler::Fetcher::FallbackError: Net::HTTPNotFound: http://local-rubygems/artifactory/api/gems/gems-repos/versions
/DONOTDELETE/shared/rvm/gems/ruby-3.1.4/gems/bundler-2.4.21/lib/bundler/fetcher/downloader.rb:47:in `fetch'
  /DONOTDELETE/shared/rvm/gems/ruby-3.1.4/gems/bundler-2.4.21/lib/bundler/fetcher/compact_index.rb:120:in `call'
  ...
HTTP GET http://local-rubygems/artifactory/api/gems/gems-repos/api/v1/dependencies
HTTP 200 OK http://local-rubygems/artifactory/api/gems/gems-repos/api/v1/dependencies
Fetching gem metadata from http://local-rubygems/artifactory/api/gems/gems-repos/
Query List: ["actioncable", "actionmailbox", "actionmailer", "actionpack", "actiontext", "actionview", "activejob", "activemodel", "activerecord", ... "websocket-extensions", "zeitwerk"]
Query Gemcutter Dependency Endpoint API: actioncable,actionmailbox,actionmailer,actionpack,actiontext,actionview,activejob,activemodel,activerecord,activerecord-nulldb-adapter,activerecord-sqlserver-adapter,activestorage,activesupport,addressable,akami,binding_of_caller,...,websocket-driver,websocket-extensions,zeitwerk
HTTP GET http://local-rubygems/artifactory/api/gems/gems-repos/api/v1/dependencies?gems=actioncable%2Cactionmailbox%2Cactionmailer%2Cactionpack%2Cactiontext%2Cactionview%2Cactivejob%2Cactivemodel%2Cactiverecord%2Cactiverecord-nulldb-adapter%2C...
HTTP 404 Not Found http://local-rubygems/artifactory/api/gems/gems-repos/api/v1/dependencies?gems=actioncable%2Cactionmailbox%2Cactionmailer%2Cactionpack%2Cactiontext%2Cactionview%2Cactivejob%2Cactivemodel%2Cactiverecord%2Cactiverecord-nulldb-adapter%2C...
could not fetch from the dependency API, trying the full index
...

[simi]

@coldnebo RFC is available at https://github.com/rubygems/rfcs/blob/master/text/0009-deprecate-dependency-api.md.

There are some known problems with JFrog, feel free to follow rubygems/rubygems#6632 for more details. Are you able to try bundler from https://github.com/rubygems/rubygems/compare/opaque-etag-http-digest to see if that fixes your problems? If I remember well, the only missing parts are specs for that branch to get merged.