Unable to install google-protobuf gem on JRuby

[blowmage]

The glcoud-ruby project has an issue opened by a user (googleapis/google-cloud-ruby#712) that is unable to install the latest version due to the dependencies on the grpc and google-protobuf gems not installing on jruby. I am not able to get the google-protobuf gem to install using versions 9.1.1.0 or 9.0.0.0.

Here is the install on mac:

$ ruby -v
jruby 9.1.1.0 (2.3.0) 2016-05-19 fe84e89 Java HotSpot(TM) 64-Bit Server VM 25.91-b14 on 1.8.0_91-b14 +jit [darwin-x86_64]

$ gem install --ignore-dependencies google-protobuf -v 3.0.0.alpha.5.0.5.1
Building native extensions.  This could take a while...
ERROR:  Error installing google-protobuf:
    ERROR: Failed to build gem native extension.

    current directory: /Users/blowmage/.gem/repos/jruby-9.1.1.0-grpc/gems/google-protobuf-3.0.0.alpha.5.0.5.1/ext/google/protobuf_c
/Users/blowmage/.rubies/jruby-9.1.1.0/bin/jruby -r ./siteconf20160524-43044-1fuqmwb.rb extconf.rb
NotImplementedError: C extensions are not supported
    <top> at /Users/blowmage/.rubies/jruby-9.1.1.0/lib/ruby/stdlib/mkmf.rb:1
  require at org/jruby/RubyKernel.java:944
   (root) at /Users/blowmage/.rubies/jruby-9.1.1.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
    <top> at extconf.rb:3

extconf failed, exit code 1

Gem files will remain installed in /Users/blowmage/.gem/repos/jruby-9.1.1.0-grpc/gems/google-protobuf-3.0.0.alpha.5.0.5.1 for inspection.
Results logged to /Users/blowmage/.gem/repos/jruby-9.1.1.0-grpc/extensions/universal-java-1.8/2.3.0/google-protobuf-3.0.0.alpha.5.0.5.1/gem_make.out

Here is the install on linux:

$ ./bin/jruby -v
jruby 9.1.1.0 (2.3.0) 2016-05-19 fe84e89 OpenJDK 64-Bit Server VM 24.95-b01 on 1.7.0_95-b00 +jit [linux-x86_64]

$ ./bin/jruby -S gem install --ignore-dependencies google-protobuf -v 3.0.0.alpha.5.0.5.1
Building native extensions.  This could take a while...
ERROR:  Error installing google-protobuf:
    ERROR: Failed to build gem native extension.

    current directory: /home/vagrant/.rvm/gems/ruby-2.3.0/gems/google-protobuf-3.0.0.alpha.5.0.5.1/ext/google/protobuf_c
/home/vagrant/jruby-9.1.1.0/bin/jruby -r ./siteconf20160524-3528-jeg3gw.rb extconf.rb
NotImplementedError: C extensions are not supported
    <top> at /home/vagrant/jruby-9.1.1.0/lib/ruby/stdlib/mkmf.rb:1
  require at org/jruby/RubyKernel.java:944
   (root) at /home/vagrant/jruby-9.1.1.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
    <top> at extconf.rb:3

extconf failed, exit code 1

Gem files will remain installed in /home/vagrant/.rvm/gems/ruby-2.3.0/gems/google-protobuf-3.0.0.alpha.5.0.5.1 for inspection.
Results logged to /home/vagrant/.rvm/gems/ruby-2.3.0/extensions/universal-java-1.7/2.3.0/google-protobuf-3.0.0.alpha.5.0.5.1/gem_make.out

[markluopp]

@blowmage, is any EAT or temporary solution for this bug fix? I am stuck on this bug for week.

[blowmage]

@genghuiluo I think @xfxyjwf might be better to able to answer than me.

[blowmage]

The 3.0.0 release notes say:

We have added proto3 support for Ruby via a native C/JRuby extension.

Will this be reflected in the ruby gem as well?

[haberman]

JRuby is supported when running from source, but I am not 100% the gem is built properly to allow this. Can someone who knows more about JRuby look at our Rakefile and let us know what we need to change about it to support JRuby?

https://github.com/google/protobuf/blob/master/ruby/Rakefile

[simonrussell]

I have a small amount of experience making gems, but no experience making JRuby binary gems; it's kind of unnecessary in a way -- it is possible to just package the JAR in the normal gem, then use it if you're running inside JRuby. Quite a few gems work this way.

All that said, I'm using JRuby, and I'd like to use Protocol Buffers, so I'm eager to see this resolved. So I'm willing to help where I can, but I guess I need to know more about the build process you're using to generate all the binary gems you're distributing.

[JasonLunn]

+1

$ ruby -v
jruby 9.1.5.0 (2.3.1) 2016-09-07 036ce39 Java HotSpot(TM) 64-Bit Server VM 24.45-b08 on 1.7.0_45-b18 +jit [darwin-x86_64]

$ gem install google-protobuf
Fetching: google-protobuf-3.0.2.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing google-protobuf:
    ERROR: Failed to build gem native extension.

    current directory: /usr/local/rvm/gems/jruby-9.1.5.0@player/gems/google-protobuf-3.0.2/ext/google/protobuf_c
/usr/local/rvm/rubies/jruby-9.1.5.0/bin/jruby -r ./siteconf20160915-70672-ue0ydp.rb extconf.rb
creating Makefile

current directory: /usr/local/rvm/gems/jruby-9.1.5.0@player/gems/google-protobuf-3.0.2/ext/google/protobuf_c
make "DESTDIR=" clean

current directory: /usr/local/rvm/gems/jruby-9.1.5.0@player/gems/google-protobuf-3.0.2/ext/google/protobuf_c
make "DESTDIR="
make: *** No rule to make target `/usr/local/rvm/rubies/jruby-9.1.5.0/lib/ruby/include/ruby/ruby.h', needed by `protobuf.o'.  Stop.

make failed, exit code 2

Gem files will remain installed in /usr/local/rvm/gems/jruby-9.1.5.0@player/gems/google-protobuf-3.0.2 for inspection.
Results logged to /usr/local/rvm/gems/jruby-9.1.5.0@player/extensions/universal-java-1.7/2.3.0/google-protobuf-3.0.2/gem_make.out

$ cat /usr/local/rvm/gems/jruby-9.1.5.0@player/extensions/universal-java-1.7/2.3.0/google-protobuf-3.0.2/gem_make.out
current directory: /usr/local/rvm/gems/jruby-9.1.5.0@player/gems/google-protobuf-3.0.2/ext/google/protobuf_c
/usr/local/rvm/rubies/jruby-9.1.5.0/bin/jruby -r ./siteconf20160915-70672-ue0ydp.rb extconf.rb
creating Makefile

current directory: /usr/local/rvm/gems/jruby-9.1.5.0@player/gems/google-protobuf-3.0.2/ext/google/protobuf_c
make "DESTDIR=" clean

current directory: /usr/local/rvm/gems/jruby-9.1.5.0@player/gems/google-protobuf-3.0.2/ext/google/protobuf_c
make "DESTDIR="
make: *** No rule to make target `/usr/local/rvm/rubies/jruby-9.1.5.0/lib/ruby/include/ruby/ruby.h', needed by `protobuf.o'.  Stop.

make failed, exit code 2

[pchaganti]

+1

Same error installing protobuf gem on OSX

[apolcyn]

Could the problem here just be that a protobuf with the "java" platform suffix hasn't been uploaded to rubygems? (e.g., google-protobuf--java.gem)

When building the package, the rake clobber_package gem suggestion builds the google-protobuf gem, but it seems to be missing the "java" platform suffix, which, to my knowledge, the other platform specific gems with pre-compiled code use, in order for gem to bring down the proper package on install.

I'm not sure exactly how to change the Rakefile to do this, (I think that normally the java extension task for rake does this sort of thing automatically) but my guess is that building only with maven is preventing the proper naming from happening?

[JasonLunn]

@apolcyn - simple patch in #2199 locally creates the .gem file with both the correct suffix and the correct metadata to identify itself as having platform java. Are other changes required to facilitate the publishing process from within Google?

[JasonLunn]

With both #2199 and #2254 having merged, and thus Travis and Jenkins are both happy at the same time, is there anything else blocking the release of v3.1.0 to RubyGems?

[JasonLunn]

CC: @TeBoring, @xfxyjwf

[TeBoring]

@haberman

[JasonLunn]

❓ @haberman ❓

[JasonLunn]

@apolcyn

[skull-squadron]

Please release google-protobuf-3.1.0-java.gem, the currently released gem (3.1.0) is completely useless on JRuby. The PR merged in master is correct.

(Here's a copy of the java gem built from master.)

[xfxyjwf]

I'm not familiar with ruby or jruby. Do we need to do a jruby specific release, or just another normal ruby gem release will fix the issue?

3.1.0 is already out. The next release will likely be 3.2.0 this quarter.

[skull-squadron]

Yes, just a JRuby 3.1.0 gem. It's possible to release just a 3.1.0 interterm JRuby -java gem, each platform's gem is separate, this will fix it without requiring a full release.

On the box of whomever has rubygem write credentials:

curl -LO https://gist.github.com/steakknife/0773a917ffc52631d8f42e7ad330a6d5/raw/30c43722610c50a88581fe6f105b65e7deed2027/google-protobuf-3.1.0-java.gem
# or build it yourself: cd ruby; rake build && gem build *.gemspec
gem push google-protobuf-3.1.0-java.gem

[JasonLunn]

I don't know who has the credentials - @haberman perhaps?
Less awesome workarounds include using the git option in your Gemfile and / or vendoring the gem you compile yourself.

[skull-squadron]

@JasonLunn That won't scale because the git repo doesn't include the jar, bundler can't build it from source without all files present. It would be better for everyone if this were released correctly.

[skull-squadron]

@haberman JRuby Rakefile #2320

[acozzette]

I went ahead and pushed a JRuby gem as version 3.1.0.0.pre. Frankly I know little about Ruby so I am not sure if the gem is sane or not; if the JRuby users on this thread could test it out and let me know if it works I would really appreciate it. If it looks OK then I'll push it as 3.1.0.

[JasonLunn]

Works for me, in that I can get the gem from RubyGems and it appears well formed after installation. I can't yet install the google-cloud-data gem, but that appears to now be gprc's issue (probably of the same flavor) rather than google-protobuf.

[acozzette]

Thanks, Jason. I think I'll just wait a few more days to see if anyone else has feedback and then push the gem as 3.1.0. I'll keep an eye on the gRPC issue, too (grpc/grpc#6705).

[ghost]

I have managed to build jruby version of gem using Rakefile after some tweaks.
I guess bigger blocker is the outdated Java wrapper code which fail tests:

https://github.com/protocolbuffers/protobuf/tree/master/ruby/src/main/java/com/google/protobuf/jruby

rake test

protobuf/ruby/tests/basic.rb:87: warning: Ambiguous first argument; make sure.
NoMethodError: undefined method `add_file' for #Google::Protobuf::Builder:0x7770f470

at /mnt/repos/protobuf/ruby/tests/google/protobuf/wrappers_pb.rb:7
build at com/google/protobuf/jruby/RubyDescriptorPool.java:81

[headius]

I'm a JRuby team member offering any assistance to get this gem updated and released for JRuby users. Let me know what I can do to help resolve this.

JRuby is actively maintained and in use by hundreds of organizations around the world. Feel free to stop by our Matrix channel (#jruby) or IRC (#jruby on Freenode) or Gitter (jruby/jruby) if you have questions or need assistance.

[luke-hill]

@headius the latest version of this gem (Or at least the 3.8.x series), have proper versioned support (So work correctly for everything aside from JRuby that is advertised). If you start there and pull down the project and try fixing it up, it would be a good starting place.

[JasonLunn]

I have changed jobs and so I am no longer in a JRuby-shop full time. Despite that, I'm interested and willing to help solve this again, but I don't think it is unreasonable to ask for more investment from the maintainers. If the community provides a second fix for this issue, there should be a commitment from Google to support JRuby on this gem going forward. Ideally, this would come in the form of:

1. Making execution of unit tests under JRuby a requirement for future branches to get merged
2. Integrating JRuby releases into the standard release process so that new JRuby builds are built every release without fail

[Legogris]

Is anyone aware of any version >= 3.8 that has a published version working with JRuby?

$ jruby --version
jruby 9.2.8.0 (2.5.3) 2019-08-12 a1ac7ff OpenJDK 64-Bit Server VM 25.222-bga on 1.8.0_222-ga +jit [linux-x86_64]

3.2.0.2 works fine, but 3.8 and 3.11 both fail to install:

$ jgem install google-protobuf -v '3.11.0'
Building native extensions. This could take a while...
ERROR:  Error installing google-protobuf:
        ERROR: Failed to build gem native extension.

    current directory: .nix-gems/gems/google-protobuf-3.11.0/ext/google/protobuf_c
/nix/store/mn24a8n3bfwf9m0znkisaxn76nkrpmd8-jruby-9.2.8.0/bin/jruby -r ./siteconf20191128-6268-68irf2.rb extconf.rb
creating Makefile

current directory: .nix-gems/gems/google-protobuf-3.11.0/ext/google/protobuf_c
make "DESTDIR=" clean

current directory: .nix-gems/gems/google-protobuf-3.11.0/ext/google/protobuf_c
make "DESTDIR="
make: *** No rule to make target '/nix/store/mn24a8n3bfwf9m0znkisaxn76nkrpmd8-jruby-9.2.8.0/lib/ruby/include/ruby/ruby.h', needed by 'protobuf.o'.  Stop.

make failed, exit code 2

Gem files will remain installed in .nix-gems/gems/google-protobuf-3.11.0 for inspection.
Results logged to .nix-gems/extensions/universal-java-1.8/2.5.0/google-protobuf-3.11.0/gem_make.out

[luke-hill]

At cucumber we use 3.2.0.2 which works "partially" with JRuby. But other than that youre very much on your own sorry :(

We've removed JRuby from our accepted build matrix also - https://github.com/cucumber/cucumber-ruby/blob/master/.circleci/config.yml - So we just don't bother with it any more.

[mvz]

We've removed JRuby from our accepted build matrix also - https://github.com/cucumber/cucumber-ruby/blob/master/.circleci/config.yml - So we just don't bother with it any more.

That's really unfortunate.

[rwojsznis]

Hello all; I wanted to try jruby with protobuf and stumbled upon this issue.

I have managed to build jruby version of gem using Rakefile after some tweaks.
I guess bigger blocker is the outdated Java wrapper code which fail tests:

Just for anyone interested - after jumping through hoops I managed to install locally compiled gem, but instantly got the same result just to see that the wrapper was kinda abandoned 🙈

Additionally it seems release for jruby wasn't pushed to rubygems since version 3.5.0pre? that is why recent version will fail while trying to build native c extension, but as said it's not the biggest problem here

so if you're here for a quick solution regarding using proto3 with jruby. Tl;dr: there is no quick solution 😅

[headius]

Perhaps someone could try to restore the old JRuby implementation and we can go from there? I would like to help but we're swamped with JRuby release 9.2.10 work right now, and I know nothing about how this gem is structured.

[deivid-rodriguez]

If I understood the issues I'm watching correctly, cucumber switched to the pure ruby implementation https://github.com/ruby-protobuf/protobuf, so this should no longer be a problem for them. Of course, that doesn't change the status of this issue, but perhaps it can provide a way out to others facing the same issues.

[haberman]

This should be fixed by #7923, but I'm leaving this open to verify that the packages get updated correctly.

[luke-hill]

@haberman if you have time could you pop some thoughts onto the cucumber repo. I'm happy to work / pair on anything you need help with.

[schanjr]

@haberman msged you on the other pull request #7923 , I had issue installing
gem "opentelemetry-exporter-otlp", "~> 0.16.0" as google-protobuf -v '3.15.8' is a dependency.

[JasonLunn]

@blowmage -

With the release of 3.19.1 last week, https://rubygems.org/gems/google-protobuf/versions/3.19.1-java is available. Does this fully address the originally reported issue?

CC: @schanjr @headius

[schanjr]

@JasonLunn I was able to use install opentelemetry-exporter-otlp (0.19.0) which had dependency on the protobuf gem. Have yet to use the gem, but the original error is gone now.

[brauliobo]

Still googleapis-common-protos and gapic-common (on which google-cloud-translate-v3 depend on) rely on grpc gem

[headius]

@JasonLunn It sounds like you had made some progress on a JRuby extension for the grpc gem. Anything I can do to help get that finished and into a grpc release?

[JasonLunn]

gRPC is the next level of abstraction up. I haven't tackled it beyond a short-lived effort a few years ago (see grpc/proposal#13). The work done here was strictly at the protocol buffers level, on top of which gRPC is built.

[headius]

So based on comments there, the wrapper you made is probably too old to be appropriate, @rdubya was hoping to have some time early last year to help, and other than that I don't know if any progress has been made.

Seems like there's still demand for this.