Skip to content

Commit

Permalink
Make libprotobuf symbols local on OSX
Browse files Browse the repository at this point in the history
@gnossen gave a great overview in grpc/grpc#24992 of the overall problem.

If a python process using both protobuf _and_ another native library linking in libprotobuf
frequently can cause crashes.  This seems to frequently affect tensorflow as well:

tensorflow/tensorflow#8394,
tensorflow/tensorflow#9525 (comment)
tensorflow/tensorflow#24976,
tensorflow/tensorflow#35573,
https://github.com/tensorflow/tensorflow/blob/v2.0.0/tensorflow/contrib/makefile/rename_protobuf.sh,
tensorflow/tensorflow#16104

Testing locally this fixes both crashes when linking in multiple versions of protobuf
and fixes `DescriptorPool` clashes as well (e.g. Python and Native code import different versions of the same message).
  • Loading branch information
Roy Williams committed Feb 25, 2021
1 parent a6c28fb commit 8f26959
Showing 1 changed file with 14 additions and 0 deletions.
14 changes: 14 additions & 0 deletions python/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,18 @@ def get_option_from_sys_argv(option_str):

extra_compile_args = []

message_extra_link_args = None
api_implementation_link_args = None
if "darwin" in sys.platform:
if sys.version_info[0] == 2:
message_init_symbol = 'init_message'
api_implementation_init_symbol = 'init_api_implementation'
else:
message_init_symbol = 'PyInit__message'
api_implementation_init_symbol = 'PyInit__api_implementation'
message_extra_link_args = ['-Wl,-exported_symbol,_%s' % message_init_symbol]
api_implementation_link_args = ['-Wl,-exported_symbol,_%s' % api_implementation_init_symbol]

if sys.platform != 'win32':
extra_compile_args.append('-Wno-write-strings')
extra_compile_args.append('-Wno-invalid-offsetof')
Expand Down Expand Up @@ -242,13 +254,15 @@ def get_option_from_sys_argv(option_str):
include_dirs=[".", "../src"],
libraries=libraries,
extra_objects=extra_objects,
extra_link_args=message_extra_link_args,
library_dirs=['../src/.libs'],
extra_compile_args=extra_compile_args,
),
Extension(
"google.protobuf.internal._api_implementation",
glob.glob('google/protobuf/internal/api_implementation.cc'),
extra_compile_args=extra_compile_args + ['-DPYTHON_PROTO2_CPP_IMPL_V2'],
extra_link_args=api_implementation_link_args,
),
])
os.environ['PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION'] = 'cpp'
Expand Down

0 comments on commit 8f26959

Please sign in to comment.