How do I handle build tools artifacts?

[wrobelda]

There's absolutely no trace of any documentation on how are binary artifacts supposed to be handled by the maintainers and to what extent vcpkg does that automatically.

For example, I am working on a cmake-less port for GPG libraries that use autoconf. The library in speech, libgpg-error builds a bunch of build tools that are in turn required by libassuan. Now:

1. I can see that the binaries are being automatically copied to vcpkg_installed/x64-osx/tools/libgpg-error/bin. But since they're being copied automatically, what's the purpose of vcpkg_copy_tools() and vcpkg_copy_tool_dependencies()?
2. I see that some projects patch cmake to use find_program, eg. Qt in find_program(QMAKE_COMMAND NAMES qmake PATHS ${VCPKG_QT_HOST_TOOLS_ROOT_DIR}/bin ${_triplet_hostbindir} NO_DEFAULT_PATH)
3. Adding configure switch --with-libgpg-error-prefix=${CURRENT_INSTALLED_DIR} makes autoconf correctly pick up vcpkg_installed root, but it still fails because it expects to find the tool at bin/gpg-error-config, while it's in tools\bin/gpg-error-config
4. Is there some function that can use to obtain a path to a tool, which I can then pass to --with-libgpg-error-prefix=${MY_TOOL_LOCATION)?
5. I see that some projects use --with-prefix=${CURRENT_INSTALLED_DIR}/tools option, which would imply that executables are meant to be copied to tools\bin. So why does vcpkg copy them automatically to tools\portname\bin?
6. Is autoconf expected to find the library in $PATH (via CMAKE_PROGRAM_PATH)?
7. Should there be any differentiation between build tools and regular tools? If so, am I supposed to explicitly copy the binaries to bin/ folder? But what about static builds, since these don't allow bin/ at all?
8. Not to mention any additional differences between platforms...

It's been previously requested that some additional documentation was added.

It is extremely frustrating that one has to spend 2-4 hours only grepping through the port files, commits history and other issues and still no be able to understand something that is fundamentally at core of this project and should be a part of the guidelines.

EDIT: also touched/discussed/requested here, here, and here, and likely many others. Some of them are 2 y/o, some are getting closed without resolution.

[Neumann-A]

makes autoconf correctly pick up

is the autoconf script using pkg-config? If so the pkg-config file needs correction. Otherwise patch the *.ac file

[wrobelda]

@Neumann-A I traced the libassuan's ./configure script and it insists on finding the libgpg-error's execs in PATH, which doesn't include the /tools/*/bin. It does lookup the gpg-error.pc and finds it, but does not check for its exec_prefix=${prefix} bindir at all, so patching it won't help.

Yes, I can patch the .ac file, but there's several libraries that GnuPG provides, each depend on libgpg-error and have same detection logic, and it simply just doesn't seem right that I'd be patching the sourcecode of several tools only to accommodate vcpkg enforcing the binaries to be kept in /tools folder but also not adding them to PATH? I mean these are build tools, it's completely common that other software/libraries expect them to be in PATH or in /bin, so why force them into tools? I can see how this makes sense for any additional non-build artifact tools, but... Or am I still missing something here?

EDIT: I think I ended up fixing it via right combination of ./configure switches, but still think it would be preferential if build tools went into /bin or /tools/*/bin had been added to PATH (even optionally).

[JackBoosY]

Was this issue resolved?

[wrobelda]

Well, for the most part I did workaround my issues.

However, the current solution where vcpkg_copy_tools() copies to tools\${PORT}\bin is IMHO needlessly complex. It requires passing a --with-libX-tool-prefix={CURRENT_INSTALLED_DIR}\tools\libX\bin\tool_executable_name for each tool that port libY needs from port libX. And that only works if the libY's configure scripts had the foresight to offer switches for the required tooling in the first place. Otherwise you need to patch the .ac files.

Why is there a need for ${PORT} in tools\${PORT}\bin, perhaps except for avoiding some potential naming conflicts? Can't it just be tools\bin? The latter would simply enable adding it to PATH by default, thus not requiring --with-libX-tool-prefix switches at all.

[JackBoosY]

@wrobelda As you might guess, we use tools/port/bin to avoid binary conflicts.

[wrobelda]

@JackBoosY I still think it creates more problems than it solves. You already have guidelines for maintainers on handling any \lib naming conflicts. Why not use the same approach to \bin tooling? Most ports don't come with tooling at all anyway, so it's less likely for tool executables to clash. Why then make this odd exception, which, as explained, introduces its own significant share of issues? Why not K.I.S.S. and consistent?

Moreover, would any tools in the ports currently in the repo even have any conflicts? It wouldn't be hard to test it.

EDIT: just to add to my argument on conflicts: a typical Linux distribution comes with orders of magnitude more packages and still successfully allows to install them all into \usr\bin (\sbin and \bin aside). I really don't see why vcpkg with its very few tooling binaries implements such systemic workaround to avoid conflicts.

[JackBoosY]

cc @ras0219 @ras0219-msft

[JackBoosY]

Replied by @ras0219-msft in #17607 (comment):

In general, vcpkg's policies around tools are informed by the idea that vcpkg is first and foremost a C/C++ library dependency manager. We are strongly against growing into a general purpose package manager containing things like vim, apache, or toolchains. Unfortunately, there is no 100% clean line of separation between these things -- we can always imagine adding just a little bit of scope creep to get more functionality at any point on the spectrum, eventually resulting in apt et al.

When to use tools and when to use another prefix?

Our general intent right now is that tools/${PORT} is generally the correct place for tools, unless the port has a separate reason it might want to put them into subfolders or siblings (tools/${PORT}/tool1 or tools/${PORT}-tool1).

We do not have a wholistic solution for curl-config, save that we have a strong preference for consumers inside and outside vcpkg to use pkg-config or CMake instead.

For all declared dependencies, the following locations are automatically added

Shouldn't the modification of the PATH environment be restricted to explicitly declared (host) dependencies instead of globbing tools/*?

Yes, with host dependencies this needs attention (though we must maintain backwards compatibility).

Where to place debug variants, if needed?

Generally, a need for debug variants violates the design model. If pressed, they would go in debug/tools/, but I would strongly avoid that if at all possible.

Where to place tools created for the host during a cross-build, vs. tools created for the target?

vcpkg's build model is that only one toolchain is available at a time, so it's "not possible" for a single build to produce both. If, for some reason, a build did produce both, the package would be responsible for only deploying the requested subset (probably both if native compiling, but only the target tools if cross-compiling).