VCPkg integration with SCons build system

[jediry]

Hello,

I am investigating adding SCons support to VCPkg. I see that there is a scripts/buildsystems directory containing integrations for msbuild and CMake, so this seems like the natural place to add a SCons "tool". I imagine the syntax within an SConstruct file using this tool looking something like this:

packages += env.VCPkg('boost-smart-ptr', STATIC=true)
packages += env.VCPkg('fftw3')
...
env.Program('my_program', LIBS=packages.libs())

...possibly with some (currently unspecified) integration with SCons's Configure mechanism for detecting the presence of headers/libs, so that we skip the VCPkg download/build if the desired packages are already otherwise available.

Questions:

1. Is this a change that is likely to be accepted?
2. Is anyone already working on this, or have they in the past?
3. Is there a way to query for the list of files that would be built without actually building a package? SCons wants to know build outputs before they're built in order to do dependency analysis. Thus, it would be extremely useful to be able to generate the list of files that would be built, given a package name and triplet. I could not find such a command in the command-line help of vcpkg.exe...does it exist? If not, how hard would it be to add this?

[jediry]

UPDATE: since posting this, I have mostly implemented this, in the SCons source, without needing any changes to vcpkg. However, my last question is still pertinent: without being able to get a list of built files that a package build produces, the integration is...awkward. I basically have to front-load the building of a package into the wrong phase of SCons setup, so that I can get the list of built files early enough for SCons to compute build dependencies. I would like to fix this if possible.

I can think of two basic solutions:

1. Add a flag to "vcpkg install" that simulates a build without building anything, but just spits out the list of files that would be built. At first glance, this seems difficult...from inspecting the source, it appears that vcpkg doesn't know what files will be built, but instead just invokes CMake to build the package and then discovers what it built. But perhaps there's some way to get CMake to report what it would build?
2. Maintain a cache of built file .lists that can be queried. Two variations here come to mind:
   a. Maintain a web service from which vcpkg.exe can download the built file .list file for a given package + triplet + feature list. These files could be generated by actually building each project on a continuous integration server somewhere, which would also serve the purpose of ensuring that each package builds correctly.
   b. Have a continuous integration server that builds each project, and then checks in the resulting .list files into the vcpkg Git repo.

Option (1.) seems more desirable, since it avoids the need for setting up additional web service and/or any automated processes on a CI server. However, if (1.) is intractable, then (2.a.) or (2.b.) could provide a viable way to predict build artifacts.

[ras0219-msft]

Unfortunately, there is no way to determine this for vcpkg and we have nothing close to production-ready that could be used for this purpose (determining the files that will be produced before building the package).

While we do produce some file lists as part of our CI, the produced lists of files can and does changed based on tons of possible information:

1. The versions of dependencies
2. The features applied. This can be even more complicated than simple "feature X adds files Y", including unions, intersections, or even potentially removal in strange circumstances.

It might be possible to make some heuristics available, but we are nowhere close to having exhaustive, iron-clad "you can use this for correctness" file lists without having the full and complete built packages available.

[axsaucedo]

Is this a planned feature or still in discussion? The integration in cmake works quite well as it doesn't force users to add any custom modifications besides passing the -DCMAKE_TOOLCHAIN_FILE= parameter. A similar solution for scons would also be ideal

[axsaucedo]

@jediry for the meantime it woudl actually be quite useful if you have managed to identify a recommended way to implement this, if you have an example that would be quite useful as well! Thank you

[fbrier]

@jediry Did you ever create a github project for your enhancements to SCons? A PR? I need your integration. Perhaps I can complete it? I am a SCons neophyte, but am willing to give it a go. Thank you.

[jediry]

@fbrier I do have a GitHub fork of SCons for this, but I have not yet pushed a PR for it. I got hung up on figuring out how to write unit tests for this functionality, and then I got busy with "life". I should probably just push the PR without tests, rather than letting it languish forever.

I'll try to get this done over the weekend.

[fbrier]

@jediry "Life" does put a crimp in our side projects. Is the repo with your changes the fork from Scons? I haven't ever written unit tests for Python, but please let me know if I can help and what you would like done. Thank you.

[jediry]

@fbrier I've dusted off my vcpkg integration, and fixed a few bugs that I found. I think it's not quite ready for a PR back to the mainline SCons repo, but getting close.

I just pushed my changes up to my fork of SCons ([email protected]:jediry/scons.git), in a branch called 'vcpkg'. If you'd like to give it a try, I would welcome any feedback.

[fbrier]

@jediry I will definitely give it a try this weekend. Thank you!

[jediry]

@fbrier FYI I've fixed several bugs over the last 48 hours, so please pull most recent changes before you try this out.

[fbrier]

@jediry I will. I started looking at the branch this weekend, but you have a lot of functions in vcpkg.py. Perhaps rather than try to understand it, I will just try to use it. I had installed vcpkg using the gitmodule approach, so I will see whether that works. Thank you again.