[openssl] on *-osx-dynamic triplets don't have install_name set properly

[russelltg]

Describe the bug

On macOS, when building OpenSSL in a dynamic configuration, the output .dylib's have an absolute install name

Environment

• OS: macOS Arm64
• Compiler: XCode 14.3

To Reproduce
Steps to reproduce the behavior:

1. ./vcpkg install 'openssl[tools]:arm64-osx-dynamic'
2. otool -D ./installed/arm64-osx-dynamic/lib/libcrypto.3.dylib

   ./installed/arm64-osx-dynamic/lib/libcrypto.3.dylib:
   /Users/russellgreene/vcpkg/installed/arm64-osx-dynamic/lib/libcrypto.3.dylib
   

Expected behavior

It should be

./installed/arm64-osx-dynamic/lib/libcrypto.3.dylib:
@rpath/libcrypto.3.dylib

Failure logs
-(please attached failure logs)

Additional context
What i've done to get around the issue temporarily is to add

find_program(
    INSTALL_NAME_TOOL
    install_name_tool
    HINTS /usr/bin /Library/Developer/CommandLineTools/usr/bin/
    DOC "Absolute path of install_name_tool"
    REQUIRED
)

message(STATUS "setting rpath prefix for macOS dynamic libraries")

foreach(LIB_NAME IN ITEMS libcrypto.3 libssl.3)
    vcpkg_execute_build_process(
        COMMAND "${INSTALL_NAME_TOOL}" -id "@rpath/${LIB_NAME}.dylib" "${LIB_NAME}.dylib"
        WORKING_DIRECTORY "${CURRENT_PACKAGES_DIR}/lib"
        LOGNAME "make-build-fix-rpath"
    )
endforeach()

after the vcpkg_fixup_pkgconfig(), but this is not a proper solution as it doesn't apply to the tools:

otool -L ./installed/arm64-osx-dynamic/tools/openssl/openssl
./installed/arm64-osx-dynamic/tools/openssl/openssl:
        /Users/russellgreene/vcpkg/installed/arm64-osx-dynamic/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
        /Users/russellgreene/vcpkg/installed/arm64-osx-dynamic/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

[russelltg]

I've written a vcpkg-macos-fixup-install-name port that goes through all the binaries and rewrites the install names as well as dependencies, so the tools also work. Would this be an interesting contribution?

[dg0yt]

Thanks @russelltg.
Isn't this similar to rpath fixup for ELF, scripts/cmake/z_vcpkg_fixup_rpath.cmake? Should these things be integrated?
I see that the openssl port used to have special hooks for osx install_name at the time when it was sort-of an amalgamation of special hooks, https://github.com/microsoft/vcpkg/pull/14785/files. Do we know other ports which would benefit immediately?

[russelltg]

Ah I see, it seems like it's regressed then--those install_name_tool commands look correct.

Yes, it is quite similar, except it seems the situation is more complex on macOS--there are 3 things that need fixing up:

• The install name, which indicates how the library will be found at runtime by other libraries/executables that link to it
• The "paths" to dependent libs (which can be wrong if they were linked to the dependent lib with an incorrect install name)
• The rpath of the binary itself

Of the ~200 ports that we use for our project, the affected ones are openssl, fribidi, and libgeotiff

[JonLiu1993]

This issue hasn’t been updated in 3 month, if it is still an issue, please reopen this issue.

[derekcyruschow-catapult]

Note #14785 fixed this in the past but somewhere down the line those changes got reverted? Anyway I've tried to make a more generic fix for various osx-dynamic builds by extending #39313 with #39889