Explore precompiled binaries for Darwin (OSX) #2063
Comments
|
@larskanis I'm hoping you can help clarify some things for me. There seems to be a few different platform name values on OSX and I'm not sure how to configure the cross-ruby tasks to do a proof-of-concept native darwin build. TL;DR: On Linux, all three of these values are the same:
On OSX, running the system Ruby, all three of these values are different:
Edit addition: On OSX, running the homebrew-installed Ruby OR an rvm-installed Ruby, all three of these values are different in different ways:
Longer explanation First, the native tasks are defined in rake-compiler's This matches what
Finally, though, if I add def platform
@platform ||= case host
when /\Ax86_64.*mingw32/
"x64-mingw32"
when /\Ai[3-6]86.*mingw32/
"x86-mingw32"
when /\Ax86_64.*linux/
"x86_64-linux"
when /\Ai[3-6]86.*linux/
"x86-linux"
when /\Auniversal.x86_64-darwin19/
"universal.x86_64-darwin19"
else
raise "unsupported host: #{host}"
end
endthen the error I see is: which implies the existence of a THIRD platform name which inserts a space between "darwin" and "19" -- and I discovered that this is the spec I can't quite put all the pieces together and make this work, but it feels like this might be pointing towards an inconsistency in how these names are handled in rake-compiler? |
|
@larskanis I'm also happy to send a PR to rake-compiler if that's what's needed to sort this out -- I'm only asking for your advice on what solution to pursue! I'll keep playing around. |
|
OK, I managed to build a working native darwin gem by applying the following patch! diff --git a/.cross_rubies b/.cross_rubies
index 57c8fb0..e3f5f1e 100644
--- a/.cross_rubies
+++ b/.cross_rubies
@@ -2,6 +2,7 @@
2.7.0:x86_64-w64-mingw32
2.7.0:i686-linux-gnu
2.7.0:x86_64-linux-gnu
+2.7.0:universal-darwin-19
2.6.0:i686-w64-mingw32
2.6.0:x86_64-w64-mingw32
2.6.0:i686-linux-gnu
diff --git a/Rakefile b/Rakefile
index f5f6f47..16be862 100644
--- a/Rakefile
+++ b/Rakefile
@@ -118,6 +118,7 @@ else
end
Rake::ExtensionTask.new("nokogiri", HOE.spec) do |ext|
+ ext.platform = "universal-darwin-19"
ext.lib_dir = File.join(*['lib', 'nokogiri', ENV['FAT_DIR']].compact)
ext.config_options << ENV['EXTOPTS']
ext.cross_compile = true
diff --git a/tasks/cross-ruby.rb b/tasks/cross-ruby.rb
index 35c098f..d4e84bd 100644
--- a/tasks/cross-ruby.rb
+++ b/tasks/cross-ruby.rb
@@ -26,6 +26,8 @@ CrossRuby = Struct.new(:version, :host) do
"x86_64-linux"
when /\Ai[3-6]86.*linux/
"x86-linux"
+ when /\Auniversal-darwin-19/
+ "universal-darwin-19"
else
raise "unsupported host: #{host}"
endand then running $ time gem install pkg/nokogiri-1.11.0.rc2-universal-darwin-19.gem
Successfully installed nokogiri-1.11.0.rc2-universal-darwin-19
1 gem installed
real 0m0.795s
user 0m0.350s
sys 0m0.185s
$ ruby -S nokogiri -v
# Nokogiri (1.11.0.rc2)
---
warnings: []
nokogiri: 1.11.0.rc2
ruby:
version: 2.7.1
platform: x86_64-darwin19
description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
engine: ruby
libxml:
source: packaged
patches:
- 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
- 0002-Remove-script-macro-support.patch
- 0003-Update-entities-to-remove-handling-of-ssi.patch
- 0004-libxml2.la-is-in-top_builddir.patch
- 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
compiled: 2.9.10
loaded: 2.9.10
libxslt:
source: packaged
patches: []
compiled: 1.1.34
loaded: 1.1.34 |
|
Ah, this is actually better and addresses the root problem: that we're trying to invoke the wrongly-named rake task: diff --git a/.cross_rubies b/.cross_rubies
index 57c8fb0..e884b16 100644
--- a/.cross_rubies
+++ b/.cross_rubies
@@ -2,6 +2,7 @@
2.7.0:x86_64-w64-mingw32
2.7.0:i686-linux-gnu
2.7.0:x86_64-linux-gnu
+2.7.0:x86_64-darwin19
2.6.0:i686-w64-mingw32
2.6.0:x86_64-w64-mingw32
2.6.0:i686-linux-gnu
diff --git a/tasks/cross-ruby.rb b/tasks/cross-ruby.rb
index 35c098f..1d5c2fd 100644
--- a/tasks/cross-ruby.rb
+++ b/tasks/cross-ruby.rb
@@ -26,6 +26,8 @@ CrossRuby = Struct.new(:version, :host) do
"x86_64-linux"
when /\Ai[3-6]86.*linux/
"x86-linux"
+ when /\Ax86_64-darwin19/
+ "x86_64-darwin19"
else
raise "unsupported host: #{host}"
end
@@ -179,7 +181,7 @@ namespace "gem" do
task "guest" do
# use Task#invoke because the pkg/*gem task is defined at runtime
Rake::Task["native:#{plat}"].invoke
- Rake::Task["pkg/#{HOE.spec.full_name}-#{plat}.gem"].invoke
+ Rake::Task["pkg/#{HOE.spec.full_name}-#{Gem::Platform.new(plat).to_s}.gem"].invoke
end
end
endAnd the output file is now |
|
Voila: huygens (master *)
nokogiri $ time gem install pkg/nokogiri-1.11.0.rc2-x86_64-darwin-19.gem
Successfully installed nokogiri-1.11.0.rc2-x86_64-darwin-19
1 gem installed
real 0m0.764s
user 0m0.355s
sys 0m0.187s
huygens (master *)
nokogiri $ ruby -S nokogiri -v
# Nokogiri (1.11.0.rc2)
---
warnings: []
nokogiri: 1.11.0.rc2
ruby:
version: 2.7.1
platform: x86_64-darwin19
description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
engine: ruby
libxml:
source: packaged
patches:
- 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
- 0002-Remove-script-macro-support.patch
- 0003-Update-entities-to-remove-handling-of-ssi.patch
- 0004-libxml2.la-is-in-top_builddir.patch
- 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
compiled: 2.9.10
loaded: 2.9.10
libxslt:
source: packaged
patches: []
compiled: 1.1.34
loaded: 1.1.34
|
|
Maybe worth noting: if I build the binary gem using homebrew-installed ruby 2.7, but install it into an rvm-installed ruby 2.7, it installs but I get this error at runtime: But then when I re-build the gem with rvm-installed ruby, and then install the gem, it works fine: The problem is that both of those rubies (homebrew-installed and rvm-installed) have the same RUBY_PLATFORM and |
|
On linux, dynamic libraries are names searched for in a runtime path (LD_LIBRARY_PATH): On OSX this is what I see: |
|
This blog post has some good info on linker paths on OSX, along with links to deeper reference material: https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172 |
|
@flavorjones Congrats for building OSX binary gems! In rake-compiler-dock we use a nasty trick to avoid library name mismatches to libruby: We don't link the shared object of the gem to libruby at all, but assume that libruby is already loaded and symbols can be resolved that way. See here and here. So the shared object on Linux has the following dependencies: $ ldd pkg/nokogiri-1.11.0.rc2-x86_64-linux/lib/nokogiri/2.7/nokogiri.so
linux-vdso.so.1 (0x00007fff413ca000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5b23f10000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5b23d1e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5b244d8000) |
|
@larskanis I noticed that last night (something I had not seen before)! It looks like when ruby is configured without |
|
Confirmed, setting Ruby build with Ruby built with |
|
Update:
|
|
Closing. Will cut an rc3 with darwin native gems, and will point people to #2075 to provide feedback. |
This is a placeholder for a future conversation (with myself?) about precompiling Darwin libraries, similarly to how we've shipped precompiled libraries for Windows since ~v1.4 and how we will hopefully ship for Linux in v1.11.0.
cc @larskanis @tenderlove tell me if you think I'm crazy for thinking we can do this.
Here's what I've learned so far
(I'll keep this section up to date as I go along)
Architecture names relevant to us
Needed changes to Nokogiri to support OSX
Invoke the properly-named gem task:
Compatibility
If Ruby is configured with
--enable-shared, the shared object libraries generated will contain references tolibruby(and possibly other libraries likelibgmp,libcrypt, andlibrt).This means that a shared object library built by a ruby that was configured with
--enabled-sharedcan't be run by a ruby that was configured with--disabled-shared. However, a shared object library built by a ruby that was configured with--disable-sharedseems to run fine in a ruby configured with--enable-shared.It's possible to hack the shared object files to get around this. See here for how rake-compiler-dock gets around this:
And it's possible on a Mac to modify the .bundle file path to contain
@rpathand then make sure thatDYLD_LIBRARY_PATHis set appropriately. Some resources on this topic:But these seem like unnatural workarounds, and so I'm going to attempt to use a Ruby built with
--disable-sharedto generate the precompiled native gem for darwin.Universal architecture
The OSX system ruby has an architecture of "universal.x86_64" which ... I'm not sure what that is. I started googling and found:
which implies that Universal is a binary that would work for both x86_64 and arm 64.
The text was updated successfully, but these errors were encountered: