BLD: for C extension builds on mac, target macOS 10.9 where possible

[robbuckley]

• closes build failure with Xcode 10 - libstdc++ not supported anymore #23424
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

For extension builds, distutils targets the same macOS version as python was built for, even if the host system is newer. This can cause problems when building on a 10.9+ system, with a python built for <10.9. Targetting macOS <10.9 sets the c++ standard library to stdlibc++, which was deprecated in 10.9 in favour of libc++. Starting with Xcode 10, stdlibc++ is no longer installed by default, so isn't guaranteed to be available. This can be an issue with the current 64/32b universal builds of python, which target macOS 10.7.

Another reason to do this is to move over to 10.9 / libc++ where possible, as the current 64-bit builds of python have done.

[pep8speaks]

Hello @robbuckley! Thanks for submitting the PR.

• There are no PEP8 issues in the file setup.py !

[codecov]

Codecov Report

Merging #24274 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #24274   +/-   ##
=======================================
  Coverage   92.29%   92.29%           
=======================================
  Files         162      162           
  Lines       51841    51841           
=======================================
  Hits        47847    47847           
  Misses       3994     3994

Flag	Coverage Δ
#multiple	90.7% <ø> (ø)	⬆️
#single	42.98% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d134ec...f8c89c2. Read the comment docs.

[WillAyd]

Thanks for the research on this PR. I have some general code comments though I'm curious if you have thoughts nohow to manage such a change long term, i.e. how will we know if / when such a directive is no longer necessary.

[TomAugspurger]

Thanks for looking into this.

Do you know if this affects libraries like NumPy as well?

[robbuckley]

@TomAugspurger I'd expect anything that uses the c++ standard lib to be affected. I cloned numpy and it builds fine for me without any workaround for this issue. I guess they don't use the c++ standard lib. I tried to check this by running otool -L on all the .a and .so files to look for linkages to libc++ and there are none. The same check on my pandas branch gives 6 objects that link to libc++.1.dylib:

./build/lib.macosx-10.7-x86_64-3.7/pandas/io/msgpack/_packer.cpython-37m-darwin.so:
./build/lib.macosx-10.7-x86_64-3.7/pandas/io/msgpack/_unpacker.cpython-37m-darwin.so:
./build/lib.macosx-10.7-x86_64-3.7/pandas/_libs/window.cpython-37m-darwin.so:
./pandas/io/msgpack/_unpacker.cpython-36m-darwin.so:
./pandas/io/msgpack/_packer.cpython-36m-darwin.so:
./pandas/_libs/window.cpython-36m-darwin.so:

If there are other libs you'd like to me to check I'm happy to take a look

[robbuckley]

@WillAyd on long term maintenance: once python targetting macOS 10.9+ becomes the norm, my change will stop having any effect. At some point after that, you could decide to drop support for sub-10.9 systems. 10.9 is >4 years old now, so maybe this isnt so far away? For example Chrome requires 10.10, Firefox 10.9

[robbuckley]

BTW, for building wheels for distribution, I suppose you want to keep compatibility with older systems, at least for now. You may need to set MACOSX_DEPLOYMENT_TARGET in the script that builds the wheels

[TomAugspurger]

Thanks for the explanation.

Anyone have concerns with this? (@jreback @chris-b1)?

[jreback]

let me look

[jreback]

so there is this section lowerdown

try:
    from wheel.bdist_wheel import bdist_wheel

    class BdistWheel(bdist_wheel):
        def get_tag(self):
            tag = bdist_wheel.get_tag(self)
            repl = 'macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64'
            if tag[2] == 'macosx_10_6_intel':
                tag = (tag[0], tag[1], repl)
            return tag
    cmdclass['bdist_wheel'] = BdistWheel
except ImportError:
    pass

I would say this is 'causing' this issue so likely should just fix this

[robbuckley]

hi, I dont think the bdist_wheel stuff is directly related to the issue this PR is trying to address (#23424 ) , which happens with any kind of build -- for example python setup.py build_ext not just when building wheels.

[jreback]

so I would ideally like to test this with our wheel builder: https://travis-ci.org/MacPython/pandas-wheels. @robbuckley might be non-trivial to do this, basically you would fork this repo then update the hash to build from this PR. Alternatively we could merge this and make sure the wheels build, reverting if there is a problem.

[robbuckley]

I’ll give the wheel build a try on the PR branch.

[jreback]

@robbuckley pls also add a note in the whatsnew (Other section is fine)

[robbuckley]

@jreback ive pushed a whatsnew change, is it OK? I've forked the wheel building repo and will try to get this tested in the next day or 2.

[jreback]

@robbuckley pls indicate if the wheel building is successful.

[robbuckley]

hi @jreback I forked https://github.com/MacPython/pandas-wheels, made a branch and updated the pandas submodule to point to the head of my PR branch. 4 out of 11 builds failed. I reran the fails, same result. See https://github.com/robbuckley/pandas-wheels/runs/43815545.

Build #2 ( macosx ) failed early, in the the multibuild scripts. It looks like the same failure is on the head of https://github.com/MacPython/pandas-wheels

The other fails aren't on the branch I forked from. They dont seem obviously related to my change: since they are all in linux builds, the code in my change shouldn't be active when building those targets.

My thoughts on how i would investigate: branch pandas-wheels from the last passing commit, rebase my changes onto the corresponding pandas commit.

Details of the other 3 fails:

Builds 3 and 4 (linux) failed later, with ERROR collecting tests/indexes/datetimes/test_tools.py

Build 5 failed with the following report.
E hypothesis.errors.Flaky: Hypothesis test_tick_add_sub(cls=Second, n=-492, m=492) produces unreliable results: Falsified on the first call but did not on a subsequent one /venv/lib/python3.5/site-packages/hypothesis/core.py:797: Flaky ---------------------------------- Hypothesis ---------------------------------- Falsifying example: test_tick_add_sub(cls=Second, n=-492, m=492) Unreliable test timings! On an initial run, this test took 1240.61ms, which exceeded the deadline of 500.00ms, but on a subsequent run it took 0.13 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.

[jreback]

should have mentioned
run on the daily branch

the natale branch only gets updated on release

[robbuckley]

Ok will try on the daily instead

On Wed, 26 Dec 2018 at 14:25, Jeff Reback ***@***.***> wrote: should have mentioned run on the daily branch the natale branch only gets updated on release — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24274 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATkI0N5HNl15AMGNPKFKN6eacs08jR4Oks5u82kLgaJpZM4ZSYCU> .

-- Sent from Gmail Mobile

[robbuckley]

hi, wheel building passed.

I rebased my changes onto 0.24.0.dev0+1370.g08c920eab which passed on the daily wheel builder yesterday (https://travis-ci.org/MacPython/pandas-wheels/builds/472537433). My rebased changes passed on my fork of the wheel builder (https://travis-ci.com/robbuckley/pandas-wheels/builds/95843192)

[jreback]

thanks for putting in all of this effort @robbuckley !