Ruby gem cannot build on ruby 2.7.2 on Mac M1 #8372

NielsKSchjoedt · 2021-03-04T09:41:42Z

What version of protobuf and what language are you using?
Version: 3.15.4
Language: Ruby

What operating system (Linux, Windows, ...) and version?
Mac OS 11.2.2 on an M1

What runtime / compiler are you using (e.g., python version or gcc version)
Ruby: 2.7.2 (using RVM)

GCC:
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: arm64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

What did you do?

Cloned the code: git clone [email protected]:protocolbuffers/protobuf.git
cd protobuf/ruby
bundle config set force_ruby_platform true
rake
rake clobber_package gem
gem install ls pkg/google-protobuf-*.gem
rake test

What did you expect to see
A successful build

What did you see instead?

../../../../ext/google/protobuf_c/ruby-upb.c:6552:11: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
  int i = *iter;
      ~   ^~~~~
20 warnings generated.
compiling ../../../../ext/google/protobuf_c/wrap_memcpy.c
linking shared-object google/protobuf_c.bundle
cd -
** Invoke tmp/arm64-darwin20/stage/lib/google (first_time)
** Execute tmp/arm64-darwin20/stage/lib/google
mkdir -p tmp/arm64-darwin20/stage/lib/google
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/convert.c (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (first_time)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c
mkdir -p tmp/arm64-darwin20/stage/ext/google/protobuf_c
** Invoke ext/google/protobuf_c/convert.c (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/convert.c
cp ext/google/protobuf_c/convert.c tmp/arm64-darwin20/stage/ext/google/protobuf_c/convert.c
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/convert.h (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/convert.h (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/convert.h
cp ext/google/protobuf_c/convert.h tmp/arm64-darwin20/stage/ext/google/protobuf_c/convert.h
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/defs.c (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/defs.c (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/defs.c
cp ext/google/protobuf_c/defs.c tmp/arm64-darwin20/stage/ext/google/protobuf_c/defs.c
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/defs.h (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/defs.h (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/defs.h
cp ext/google/protobuf_c/defs.h tmp/arm64-darwin20/stage/ext/google/protobuf_c/defs.h
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/extconf.rb (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/extconf.rb (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/extconf.rb
cp ext/google/protobuf_c/extconf.rb tmp/arm64-darwin20/stage/ext/google/protobuf_c/extconf.rb
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/map.c (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/map.c (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/map.c
cp ext/google/protobuf_c/map.c tmp/arm64-darwin20/stage/ext/google/protobuf_c/map.c
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/map.h (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/map.h (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/map.h
cp ext/google/protobuf_c/map.h tmp/arm64-darwin20/stage/ext/google/protobuf_c/map.h
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/message.c (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/message.c (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/message.c
cp ext/google/protobuf_c/message.c tmp/arm64-darwin20/stage/ext/google/protobuf_c/message.c
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/message.h (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/message.h (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/message.h
cp ext/google/protobuf_c/message.h tmp/arm64-darwin20/stage/ext/google/protobuf_c/message.h
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/protobuf.c (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/protobuf.c (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/protobuf.c
cp ext/google/protobuf_c/protobuf.c tmp/arm64-darwin20/stage/ext/google/protobuf_c/protobuf.c
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/protobuf.h (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/protobuf.h (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/protobuf.h
cp ext/google/protobuf_c/protobuf.h tmp/arm64-darwin20/stage/ext/google/protobuf_c/protobuf.h
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/repeated_field.c (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/repeated_field.c (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/repeated_field.c
cp ext/google/protobuf_c/repeated_field.c tmp/arm64-darwin20/stage/ext/google/protobuf_c/repeated_field.c
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/repeated_field.h (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/repeated_field.h (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/repeated_field.h
cp ext/google/protobuf_c/repeated_field.h tmp/arm64-darwin20/stage/ext/google/protobuf_c/repeated_field.h
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/ruby-upb.c (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/ruby-upb.c (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/ruby-upb.c
cp ext/google/protobuf_c/ruby-upb.c tmp/arm64-darwin20/stage/ext/google/protobuf_c/ruby-upb.c
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/ruby-upb.h (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/ruby-upb.h (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/ruby-upb.h
cp ext/google/protobuf_c/ruby-upb.h tmp/arm64-darwin20/stage/ext/google/protobuf_c/ruby-upb.h
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/third_party/wyhash/wyhash.h (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/third_party/wyhash (first_time)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/third_party/wyhash
mkdir -p tmp/arm64-darwin20/stage/ext/google/protobuf_c/third_party/wyhash
** Invoke ext/google/protobuf_c/third_party/wyhash/wyhash.h (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/third_party/wyhash/wyhash.h
cp ext/google/protobuf_c/third_party/wyhash/wyhash.h tmp/arm64-darwin20/stage/ext/google/protobuf_c/third_party/wyhash/wyhash.h
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c/wrap_memcpy.c (first_time)
** Invoke tmp/arm64-darwin20/stage/ext/google/protobuf_c (not_needed)
** Invoke ext/google/protobuf_c/wrap_memcpy.c (not_needed)
** Execute tmp/arm64-darwin20/stage/ext/google/protobuf_c/wrap_memcpy.c
cp ext/google/protobuf_c/wrap_memcpy.c tmp/arm64-darwin20/stage/ext/google/protobuf_c/wrap_memcpy.c
** Invoke tmp/arm64-darwin20/stage/lib/google/protobuf.rb (first_time)
** Invoke tmp/arm64-darwin20/stage/lib/google (not_needed)
** Invoke lib/google/protobuf.rb (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/lib/google/protobuf.rb
cp lib/google/protobuf.rb tmp/arm64-darwin20/stage/lib/google/protobuf.rb
** Invoke tmp/arm64-darwin20/stage/lib/google/protobuf/message_exts.rb (first_time)
** Invoke tmp/arm64-darwin20/stage/lib/google/protobuf (first_time)
** Execute tmp/arm64-darwin20/stage/lib/google/protobuf
mkdir -p tmp/arm64-darwin20/stage/lib/google/protobuf
** Invoke lib/google/protobuf/message_exts.rb (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/lib/google/protobuf/message_exts.rb
cp lib/google/protobuf/message_exts.rb tmp/arm64-darwin20/stage/lib/google/protobuf/message_exts.rb
** Invoke tmp/arm64-darwin20/stage/lib/google/protobuf/repeated_field.rb (first_time)
** Invoke tmp/arm64-darwin20/stage/lib/google/protobuf (not_needed)
** Invoke lib/google/protobuf/repeated_field.rb (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/lib/google/protobuf/repeated_field.rb
cp lib/google/protobuf/repeated_field.rb tmp/arm64-darwin20/stage/lib/google/protobuf/repeated_field.rb
** Invoke tmp/arm64-darwin20/stage/lib/google/protobuf/well_known_types.rb (first_time)
** Invoke tmp/arm64-darwin20/stage/lib/google/protobuf (not_needed)
** Invoke lib/google/protobuf/well_known_types.rb (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/lib/google/protobuf/well_known_types.rb
cp lib/google/protobuf/well_known_types.rb tmp/arm64-darwin20/stage/lib/google/protobuf/well_known_types.rb
** Invoke tmp/arm64-darwin20/stage/tests/basic.rb (first_time)
** Invoke tmp/arm64-darwin20/stage/tests (first_time)
** Execute tmp/arm64-darwin20/stage/tests
mkdir -p tmp/arm64-darwin20/stage/tests
** Invoke tests/basic.rb (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/tests/basic.rb
cp tests/basic.rb tmp/arm64-darwin20/stage/tests/basic.rb
** Invoke tmp/arm64-darwin20/stage/tests/generated_code_test.rb (first_time)
** Invoke tmp/arm64-darwin20/stage/tests (not_needed)
** Invoke tests/generated_code_test.rb (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/tests/generated_code_test.rb
cp tests/generated_code_test.rb tmp/arm64-darwin20/stage/tests/generated_code_test.rb
** Invoke tmp/arm64-darwin20/stage/tests/stress.rb (first_time)
** Invoke tmp/arm64-darwin20/stage/tests (not_needed)
** Invoke tests/stress.rb (first_time, not_needed)
** Execute tmp/arm64-darwin20/stage/tests/stress.rb
cp tests/stress.rb tmp/arm64-darwin20/stage/tests/stress.rb
** Execute copy:protobuf_c:arm64-darwin20:2.7.2
install -c tmp/arm64-darwin20/protobuf_c/2.7.2/protobuf_c.bundle lib/google/protobuf_c.bundle
cp tmp/arm64-darwin20/protobuf_c/2.7.2/protobuf_c.bundle tmp/arm64-darwin20/stage/lib/google/protobuf_c.bundle
** Execute compile:protobuf_c:arm64-darwin20
** Execute compile:arm64-darwin20
** Execute compile
** Invoke genproto (first_time)
** Invoke lib/google/protobuf/any_pb.rb (first_time)
** Invoke ../src/google/protobuf/any.proto (first_time, not_needed)
** Execute lib/google/protobuf/any_pb.rb
../src/protoc -I../src --ruby_out=lib ../src/google/protobuf/any.proto
rake aborted!
Command failed with status (127): [../src/protoc -I../src --ruby_out=lib ../s...]
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/file_utils.rb:67:in `block in create_shell_runner'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/file_utils.rb:57:in `sh'
/Users/niels-kristian/Programming/autouncle/scout/protobuf/ruby/Rakefile:37:in `block (2 levels) in <top (required)>'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:281:in `block in execute'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:281:in `each'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:281:in `execute'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:241:in `each'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:241:in `invoke_prerequisites'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:241:in `each'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:241:in `invoke_prerequisites'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:241:in `each'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:241:in `invoke_prerequisites'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:116:in `each'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:83:in `block in run'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/lib/rake/application.rb:80:in `run'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/bin/rake:23:in `load'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/bin/rake:23:in `<main>'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/bin/ruby_executable_hooks:22:in `eval'
/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/bin/ruby_executable_hooks:22:in `<main>'

The text was updated successfully, but these errors were encountered:

bbuchalter · 2021-03-12T21:17:41Z

I get the error message you report, but on step 4, when issuing the rake command. Do you only get this when running tests?

bbuchalter · 2021-03-12T21:25:32Z

Command failed with status (127): [../src/protoc -I../src --ruby_out=lib ../s...]

127 means that ../src/protoc isn't a command that can be found.... where'd it go? 🤔

bbuchalter · 2021-03-12T21:36:14Z

./build-protoc.sh seems relevant, but I can't get it running.

bbuchalter · 2021-03-12T21:58:23Z

I already have a protoc in my PATH, so modifying the Rakefile like this:

-      sh "../src/protoc -I../src --ruby_out=lib #{input_file}"
+      sh "protoc -I../src --ruby_out=lib #{input_file}"

Allows the steps you suggested to work.

bbuchalter · 2021-03-12T22:10:53Z

Switching to gem 'google-protobuf', github: 'protocolbuffers/protobuf', submodules: true in my application's Gemfile and then running bundle install yielded this:

Using google-protobuf 3.15.5 (was 3.15.6) from https://github.com/protocolbuffers/protobuf.git (at master@f154323)
/Users/brian.buchalter/.rubies/ruby-2.7.2/lib/ruby/2.7.0/rubygems/ext/builder.rb:167: warning: conflicting chdir during another chdir block
/Users/brian.buchalter/.rubies/ruby-2.7.2/lib/ruby/2.7.0/rubygems/ext/builder.rb:175: warning: conflicting chdir during another chdir block
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/brian.buchalter/.gem/ruby/2.7.2/bundler/gems/protobuf-f154323fa682/ruby/ext/google/protobuf_c
/Users/brian.buchalter/.rubies/ruby-2.7.2/bin/ruby -I /Users/brian.buchalter/.rubies/ruby-2.7.2/lib/ruby/2.7.0 -r ./siteconf20210312-62659-158a0dt.rb extconf.rb
creating Makefile

current directory: /Users/brian.buchalter/.gem/ruby/2.7.2/bundler/gems/protobuf-f154323fa682/ruby/ext/google/protobuf_c
make "DESTDIR=" clean

current directory: /Users/brian.buchalter/.gem/ruby/2.7.2/bundler/gems/protobuf-f154323fa682/ruby/ext/google/protobuf_c
make "DESTDIR="
compiling protobuf.c
In file included from protobuf.c:31:
In file included from ./protobuf.h:38:
./ruby-upb.h:555:38: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'uint32_t' (aka 'unsigned int') [-Wshorten-64-to-32]
    return ((uint64_t)_upb_be_swap32(val) << 32) | _upb_be_swap32(val >> 32);
                      ~~~~~~~~~~~~~~ ^~~
./ruby-upb.h:1784:22: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
  int overrun = *ptr - d->end;
      ~~~~~~~   ~~~~~^~~~~~~~
protobuf.c:63:30: warning: redefinition of typedef 'StringBuilder' is a C11 feature [-Wtypedef-redefinition]
typedef struct StringBuilder StringBuilder;
                             ^
./protobuf.h:88:30: note: previous definition is here
typedef struct StringBuilder StringBuilder;
                             ^
3 warnings generated.
compiling convert.c
In file included from convert.c:40:
In file included from ./convert.h:36:
In file included from ./protobuf.h:38:
./ruby-upb.h:555:38: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'uint32_t' (aka 'unsigned int') [-Wshorten-64-to-32]
    return ((uint64_t)_upb_be_swap32(val) << 32) | _upb_be_swap32(val >> 32);
                      ~~~~~~~~~~~~~~ ^~~
./ruby-upb.h:1784:22: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
  int overrun = *ptr - d->end;
      ~~~~~~~   ~~~~~^~~~~~~~
convert.c:44:10: fatal error: 'third_party/wyhash/wyhash.h' file not found
#include "third_party/wyhash/wyhash.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings and 1 error generated.
make: *** [convert.o] Error 1

make failed, exit code 2

convert.c:44:10: fatal error: 'third_party/wyhash/wyhash.h' file not found now seems to be the interesting bit

wyhash was not retrieved as one of the submodules during bundle install:

Fetching https://github.com/protocolbuffers/protobuf.git
Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
Cloning into '/Users/brian.buchalter/.gem/ruby/2.7.2/bundler/gems/protobuf-f154323fa682/third_party/benchmark'...
Cloning into '/Users/brian.buchalter/.gem/ruby/2.7.2/bundler/gems/protobuf-f154323fa682/third_party/googletest'...

However, when I modified the Rakefile for the gem and ran rake from source, ext/google/protobuf_c/third_party/wyhash/wyhash.h appeared. 🤔

bbuchalter · 2021-03-12T22:28:50Z

#8000 (comment) appears related

bbuchalter · 2021-03-12T22:34:34Z

Switching to gem 'google-protobuf', github: 'bbuchalter/protobuf', branch: 'bb.wyhash_extconf', submodules: true allowed my application to bundle successfully, but when I start it, I receive this error:

$ bin/rails c
/Users/brian.buchalter/.gem/ruby/2.7.2/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require': cannot load such file -- google/protobuf/any_pb (LoadError)
	from /Users/brian.buchalter/.gem/ruby/2.7.2/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
	from /Users/brian.buchalter/.gem/ruby/2.7.2/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:89:in `register'
	from /Users/brian.buchalter/.gem/ruby/2.7.2/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
	from /Users/brian.buchalter/.gem/ruby/2.7.2/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:44:in `require'
	from /Users/brian.buchalter/.gem/ruby/2.7.2/gems/activesupport-6.0.3.5/lib/active_support/dependencies.rb:324:in `block in require'
	from /Users/brian.buchalter/.gem/ruby/2.7.2/gems/activesupport-6.0.3.5/lib/active_support/dependencies.rb:291:in `load_dependency'
	from /Users/brian.buchalter/.gem/ruby/2.7.2/gems/activesupport-6.0.3.5/lib/active_support/dependencies.rb:324:in `require'
	from /Users/brian.buchalter/.gem/ruby/2.7.2/bundler/gems/protobuf-4039ed2efa44/ruby/lib/google/protobuf/well_known_types.rb:32:in `<main>'
<snip>

bouk · 2021-03-18T10:12:16Z

@bbuchalter it seems the rakefile also builds a bunch of well_known proto files: https://github.com/protocolbuffers/protobuf/blob/master/ruby/Rakefile

So that's also needed

bouk · 2021-03-18T21:22:00Z

I have a working branch here: https://github.com/bouk/protobuf/tree/ruby-build

You can use it with:

gem 'google-protobuf', github: 'bouk/protobuf', branch: 'ruby-build', submodules: true

jamesgecko · 2021-04-02T21:28:22Z

This bug isn't M1 specific. I can reproduce on a 2013 MacBook Air running macOS 11.2.3 and Ruby 2.7.2 building the master branch (currently 97cb3a8). Step 4, running rake fails. The ../src/protoc command doesn't exist.

haberman · 2021-04-03T22:26:02Z

I agree with @jamesgecko, none of the reports on this bug are M1-specific. I think they are seen on M1 due to #8199

I think the best fix is to ship M1 binary gems, which should fix #8199.

However, I see two other independent issues raised in this bug:

rake build depends on ../src/protoc having previously been built, because it wants to rebuild the generated code for the well-known types.
our extconf.rb depends on rake build having previous run, because it depends on wyhash.h having been copied from ../../../../third_party/wyhash.

If anybody knows of proper fixes for these issues I am happy to merge them.

Regarding bouk@e7edc92, will this work properly when building a gem from source? Since the source gem will only contain the contents of the ext directory, I am worried that the source gem will not actually contain wyhash.h unless we copy it into ext.

bouk · 2021-04-04T07:19:15Z

It does work, because of the find_header invocation. That makes sure the gem builder can find the correct header.

haberman · 2021-04-04T17:01:13Z

@bouk I know it works when building from a git client. But the Ruby source gem only contains these files:

$ gem unpack google-protobuf
Fetching google-protobuf-3.15.7.gem
Unpacked gem: '/tmp/google-protobuf-3.15.7'
√ desktop /tmp$ find google-protobuf-3.15.7/
google-protobuf-3.15.7/
google-protobuf-3.15.7/ext
google-protobuf-3.15.7/ext/google
google-protobuf-3.15.7/ext/google/protobuf_c
google-protobuf-3.15.7/ext/google/protobuf_c/map.c
google-protobuf-3.15.7/ext/google/protobuf_c/message.c
google-protobuf-3.15.7/ext/google/protobuf_c/ruby-upb.c
google-protobuf-3.15.7/ext/google/protobuf_c/third_party
google-protobuf-3.15.7/ext/google/protobuf_c/third_party/wyhash
google-protobuf-3.15.7/ext/google/protobuf_c/third_party/wyhash/wyhash.h
google-protobuf-3.15.7/ext/google/protobuf_c/convert.c
google-protobuf-3.15.7/ext/google/protobuf_c/defs.h
google-protobuf-3.15.7/ext/google/protobuf_c/convert.h
google-protobuf-3.15.7/ext/google/protobuf_c/message.h
google-protobuf-3.15.7/ext/google/protobuf_c/repeated_field.c
google-protobuf-3.15.7/ext/google/protobuf_c/defs.c
google-protobuf-3.15.7/ext/google/protobuf_c/map.h
google-protobuf-3.15.7/ext/google/protobuf_c/wrap_memcpy.c
google-protobuf-3.15.7/ext/google/protobuf_c/ruby-upb.h
google-protobuf-3.15.7/ext/google/protobuf_c/extconf.rb
google-protobuf-3.15.7/ext/google/protobuf_c/protobuf.c
google-protobuf-3.15.7/ext/google/protobuf_c/protobuf.h
google-protobuf-3.15.7/ext/google/protobuf_c/repeated_field.h
google-protobuf-3.15.7/tests
google-protobuf-3.15.7/tests/generated_code_test.rb
google-protobuf-3.15.7/tests/stress.rb
google-protobuf-3.15.7/tests/basic.rb
google-protobuf-3.15.7/lib
google-protobuf-3.15.7/lib/google
google-protobuf-3.15.7/lib/google/protobuf
google-protobuf-3.15.7/lib/google/protobuf/field_mask_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/wrappers_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/type_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/any_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/struct_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/message_exts.rb
google-protobuf-3.15.7/lib/google/protobuf/source_context_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/timestamp_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/empty_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/well_known_types.rb
google-protobuf-3.15.7/lib/google/protobuf/repeated_field.rb
google-protobuf-3.15.7/lib/google/protobuf/duration_pb.rb
google-protobuf-3.15.7/lib/google/protobuf/api_pb.rb
google-protobuf-3.15.7/lib/google/protobuf.rb

I was worried that find_header() would break this case, because there is no ../../../../third_party/wyhash/wyhash.h here to find.

I tried adding find_header('third_party/wyhash/wyhash.h', '../../../..') to extconf.rb here, and surprisingly it did work, even though the file does not exist in that directory. It looks like mkmf always looks in the current directory first, which is convenient for our use case.

$ ruby ext/google/protobuf_c/extconf.rb 
checking for third_party/wyhash/wyhash.h in ../../../..... yes
creating Makefile

In that case, adding this find_header() into extconf.rb sounds like a good idea and a safe change. It can help make extconf.rb work independently of rake when building in Git, and for the source gem it doesn't hurt anything.

jamesgecko · 2021-04-05T19:49:13Z

First, a clarification:
In step 3 in the OP, the correct command to make bundler compile from source is bundle config force_ruby_platform true. 🙂

Next, I've got a branch here with the wyhash fix. I've also made it use a version of protoc found in the PATH if ../src/protoc doesn't exist.

disregard

But I think I've found another issue. The build process works as expected when I run bundle exec rake in the ruby directory of the protobuf repo. The build appears to behave differently when I build the gem in an actual project. Here's a minimal Gemfile:

source 'https://rubygems.org'
gem 'google-protobuf', github: 'jamesgecko/protobuf', branch: 'jd/fix-ruby-compile', submodules: true

When I run bundle exec ruby -r 'google/protobuf/timestamp_pb' -e "puts 'test'" in this directory, I get this error:

Traceback (most recent call last):
        1: from /Users/james/.rbenv/versions/2.7.2/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
/Users/james/.rbenv/versions/2.7.2/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- google/protobuf/timestamp_pb (LoadError)

Visiting the location indicated by bundle info google-protobuf | grep Path and navigating to the lib/google/protobuf directory shows that the following generated protobuf files are missing when the gem is built through bundler:

any_pb.rb
api_pb.rb
duration_pb.rb
empty_pb.rb
field_mask_pb.rb
source_context_pb.rb
struct_pb.rb
timestamp_pb.rb
type_pb.rb
wrappers_pb.rb

I'm not very familiar with how building gems works, but it looks like it's skipping some steps in the Rakefile when bundler builds the gemspec?

Edit: Ah, apparently the reason I'm still having trouble building my branch with bundler after these changes is because I've been sourcing the gem directly from git. Bundler wants the gemspec file to be in the root directory of the repo. Because bundler and gem make a lot of assumptions about directory structure, the google-protobuf gem would probably have to be moved into a submodule for compilation to work when the gem is sourced directly from a git repo.

NielsKSchjoedt changed the title ~~Ruby gem cannot build on ruby 2.7 on Mac M1~~ Ruby gem cannot build on ruby 2.7.2 on Mac M1 Mar 4, 2021

This was referenced Mar 4, 2021

Update Ruby gem with arm64 binaries (Apple M1 support) #8199

Closed

macOS Apple M1/Arm Support #8062

Closed

bouk referenced this issue in bouk/protobuf Mar 21, 2021

Commit generated ruby protobufs

e7edc92

ghost mentioned this issue Apr 1, 2021

google-protobuf dependency doesn't work on Apple Silicon pganalyze/pg_query#204

Closed

jamesgecko mentioned this issue Apr 9, 2021

Fix source gem compilation #8471

Merged

acozzette closed this as completed in #8471 Apr 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ruby gem cannot build on ruby 2.7.2 on Mac M1 #8372

Ruby gem cannot build on ruby 2.7.2 on Mac M1 #8372

NielsKSchjoedt commented Mar 4, 2021 •

edited

Loading

bbuchalter commented Mar 12, 2021

bbuchalter commented Mar 12, 2021

bbuchalter commented Mar 12, 2021 •

edited

Loading

bbuchalter commented Mar 12, 2021

bbuchalter commented Mar 12, 2021 •

edited

Loading

bbuchalter commented Mar 12, 2021

bbuchalter commented Mar 12, 2021

bouk commented Mar 18, 2021 •

edited

Loading

bouk commented Mar 18, 2021 •

edited

Loading

jamesgecko commented Apr 2, 2021

haberman commented Apr 3, 2021

bouk commented Apr 4, 2021

haberman commented Apr 4, 2021

jamesgecko commented Apr 5, 2021 •

edited

Loading

Ruby gem cannot build on ruby 2.7.2 on Mac M1 #8372

Ruby gem cannot build on ruby 2.7.2 on Mac M1 #8372

Comments

NielsKSchjoedt commented Mar 4, 2021 • edited Loading

bbuchalter commented Mar 12, 2021

bbuchalter commented Mar 12, 2021

bbuchalter commented Mar 12, 2021 • edited Loading

bbuchalter commented Mar 12, 2021

bbuchalter commented Mar 12, 2021 • edited Loading

bbuchalter commented Mar 12, 2021

bbuchalter commented Mar 12, 2021

bouk commented Mar 18, 2021 • edited Loading

bouk commented Mar 18, 2021 • edited Loading

jamesgecko commented Apr 2, 2021

haberman commented Apr 3, 2021

bouk commented Apr 4, 2021

haberman commented Apr 4, 2021

jamesgecko commented Apr 5, 2021 • edited Loading

NielsKSchjoedt commented Mar 4, 2021 •

edited

Loading

bbuchalter commented Mar 12, 2021 •

edited

Loading

bbuchalter commented Mar 12, 2021 •

edited

Loading

bouk commented Mar 18, 2021 •

edited

Loading

bouk commented Mar 18, 2021 •

edited

Loading

jamesgecko commented Apr 5, 2021 •

edited

Loading