Manifest mode does not support overlay ports

[ras0219]

Overlay ports are extremely useful for project-specific fixes or library updates; manifest mode does not currently offer a mechanism for projects to continue using this feature.

Proposed fix:

In the vcpkg.json, a field overlay-ports would be a list of relative paths from the manifest file that functions like the existing --overlay-ports=...

[Neumann-A]

see #12226

actually if ports from a overlay are installed vcpkg should store the value of --overlay-ports=... somwhere in the triplet. Alternativly give me some environment variable like VCPKG_PORT_OVERLAYS (related #10119)

[strega-nil]

I think adding VCPKG_PORT_OVERLAYS is likely the best thing to do, since we want to add federation and that will replace most uses of overlay ports; thus, I don't want to add a flag for it.

[ras0219]

An environment variable is much less ergonomic for my use case: I have an application project that needs a tweaked version of some library. I'd like to submodule upstream vcpkg (making it very easy for me to update in the future), but use this tweaked version[1] just for this project.

The need for the tweaked version is not coming from my environment, it's coming from the needs of my project. Therefore, it's at-odds to track it through an environment variable.

Even if temporary, I think that this is a common enough need that we should introduce a stop gap measure. An experimental field "x-overlay-ports": [] would be sufficient.

It would also be acceptable to put overlay-ports values into the triplet if I could then override triplets.

[1] Specifically, I need to pin fltk forward to a beta commit because they've added High DPI support :)

[strega-nil]

Hrmm... okay, fair enough.

[sylveon]

I similarly have project-specific fixes which haven't been merged into the original project yet, as well as the portfile for a library I'm writing in parallel to my main project, but I do not feel is ready for mainlining in vcpkg yet. This is a blocker to adopting manifest mode for me.

[Neumann-A]

@sylveon: if you use cmake you can set the variables VCPKG_OVERLAY_PORTS and/or VCPKG_OVERLAY_TRIPLETS

[sylveon]

I don't use cmake.

[strega-nil]

@sylveon are you using the msbuild support?

[sylveon]

Yes

[strega-nil]

Hm, yeah, we haven't stabilized that support yet. Definitely needs work.

[aggieNick02]

I'd also like to use manifests but use overlay triplets and overlay ports. My use case is similar to @ras0219 . I submodule (actually subrepo for me but same end result) vcpkg in my repo, and have everything defined in my repo - any overlay triplets and overlay ports, and currently a homemade script and vcpkg dependency list to build what my repo depends on.

I'd much prefer to use manifests and have each project declare what it depends on, gaining all the power of manifests and being able to throw away my little build script. But without a way to point to the overlay ports folder that sits at a relative path to the manifest, I'd have to resort to an environment variable that I change depending on the repo I'm working on, which isn't good. So for now I think I'll unfortunately have to stick with a homemade script.

@ras0219 , with your setup, do you use per-project integration so that your different repos can use different commits of the vcpkg subrepo for your different repos?

[hajokirchhoff]

Hi all, what is the status here (other than "other")? Any new plans or development?

My problem is with boost-serialization. I have a bug fix I need for my project and cannot wait until it is integrated in an official release (esp. since boost-serialization seems not to be actively maintained anymore).

I followed https://vcpkg.io/en/docs/examples/patching.html, created a patch and modified ./ports/boost-serialization. This works fine in classic mode when I host vcpkg on my own, which we do. I simply require boost-serialization 1.77.0#1 and there it is, checked out and build from my own portfile/vcpkg files in my own git repository.

In versioning mode however the portfile.cmake/vcpkg.json files in the vcpkg/ports directory seem to be ignored, so my patch is not picked up. b-/boost-serialization lets me specify a "git-tree" SHA-1 for the portfile/vcpkg files, but I don't know what repository this SHA-1 refers to. The directory buildtrees/versioning contains the portfiles for the specific versions, but it is not under source control. It is somehow "build".

I guess am not really using an overlay-port, but host vcpkg on my own git server instead and would like to use my own versions of portfile/vcpkg with vcpkg.json versioning. (Or is that what's called an overlay-port?).

Any help would be much appreciated.

Specifiy question: Where does the versioning get the specific portfile/vcpkg version from? What repository is it contained in?

[aggieNick02]

@hajokirchhoff Are you using per-project or sitewide vcpkg integration? Knowing that would be helpful.

It's been a few months since I dealt with most of this in my linked issue - #20690 . But the gist of what I learned is that basically the issue you are commenting on should probably be closed.

What you are trying to do is (as far as I know/understand, I'm not an expert ;-)) is called "overlay ports", when you want to use a portfile that is not the one in the git repo. Once we know if you are doing site-wide or per-project integration I can try to help you get things functioning.

I'm pretty sure what you want to do is possible - I have a few custom-specified ports similar to yours that "just work" once everything is set up right. Most of getting it to work is figuring out the terminology that is used for what you are trying to do, and dealing with the fact that some of the newest functionality isn't always super-documented.

To your specific question, the portfiles are generally grabbed from the gitrepo you've locally cloned and have activated for site-wide integration, or the gitrepo you are pointing to via per-project integration.

When you say classic vs versioning mode, what do you mean?

[hajokirchhoff]

I am using the "manifests" feature with per-project integration. Sorry, I confused "versions" with "manifests". My question is: How can I make the vcpkg "manifest" mode pick up my own portfile. My problem is this: The boost-serialization lib has a bug, I have a patch. 18 months ago I followed the (probably outdated) vcpkg description on how to add my own patch to the vcpkg port and modified `ports/boost-serialization/portfile.cmake` and vcpkg.json. I pushed these changes to my own vcpkg git repository. This works fine in system-wide integration mode, without manifests. When I run vcpkg install boost-serialization, my patch is picked up and the patched library is installed and visible for every project. But my projects set "vcpkg manifest" to true and use a specific vcpkg.json file. This file specifies the baseline, and that is where my problems start. My patch is not visible, even though I set the correct SHA-1 for the baseline. I set these environment variables: VCPKG_DEFAULT_BINARY_CACHE=C:\nobackup\vcpkg_binary_cache VCPKG_DEFAULT_TRIPLET=x64-windows VCPKG_FEATURE_FLAGS=manifests,binarycaching and run "vcpkg integrate install", but do not install any packages. These are installed in manifest mode by MSBuild. Here is what I understand so far: - without manifests, vcpkg picks up the portfile.cmake files from the vcpkg repository from the path ./ports/boost-serialization. I changed these and pushed them to my own vcpkg repository. That is why it works. - but with manifests, vcpkg looks up ./versions/b-/boost-serialization.json (in the vcpkg repository), searches for the required version, sees a (poorly documented) "git-tree" there and obtains the portfile.cmake/vcpkg.json from somewhere else, not from my own repository. Question: Where do the `portfile.cmake/vcpkg.json` files come from, when vcpkg is run in manifest mode?

[strega-nil-ms]

@hajokirchhoff when using builtin-baseline, we do:

1. get the baseline version from versions/baseline.json
2. get the port files from the git tree specified for the baseline version, in versions/p-/port.json (with p- and port replaced by the first letter of and the name of the port specifically)

The best way to solve this problem in the future is using overlays; add an overlay-ports directory to the base of your project, and add set(VCPKG_OVERLAY_PORTS "${CMAKE_CURRENT_LIST_DIR}/overlay-ports") to your CMakeLists.txt.

[hajokirchhoff]

1. get the baseline version from `versions/baseline.json`

Yep, did that.

2. get the port files from the git tree specified for the baseline version, in `versions/p-/port.json` (with `p-` and `port` replaced by the first letter of and the name of the port specifically)

Question: what git repository do these files come from? "git-tree" for boost-serialization specifies an SHA-1 that is not part of the vcpkg git repository.

The best way to solve this problem in the future is using overlays; add an overlay-ports directory to the base of your project, and add set(VCPKG_OVERLAY_PORTS "${CMAKE_CURRENT_LIST_DIR}/overlay-ports") to your CMakeLists.txt.

I am using MSBuild. Any idea how to do this with MSBuild?

Hm, I just found this: After modifiying the port files you have to run this command

vcpkg x-add-version

(https://vcpkg.io/en/docs/users/versioning.html#updating-the-version-files)

[aggieNick02]

@hajokirchhoff - my apologies, I wrote up a nice long explanation of what is going on yesterday. Either I didn't hit "Comment" or something went wrong when I did... sigh. I'll try to reconstruct the most important bits. Please excuse any typos/poor formatting.

The reason you are seeing different behavior between classsic vs manifest mode is that classic vcpkg mode (non-manifest) does not support versioning at all. It sees your portfile (that you've checked into your fork of the vcpkg repo) and uses that.

Manifest mode does support versioing, and versioning is complicated. That SHA1 you mentioned is actually the git object ID of the port's directory, which lets vcpkg get the portfile at a previous revision as necessary - see https://vcpkg.readthedocs.io/en/latest/specifications/registries-2/

To properly version, you need to make the port like you did, and then after it is committed, update versions/p-/port.json with your new port and its git object ID (obtained by running git rev-parse on the port folder), and update versions/baseline.json to make it the default version.

Rather than deal with all of that, things are much easier if you just use overlay ports. Why? Because overlay ports don't deal with versioning. If a port is present in the overlay ports directory, it always wins. So you need not deal with versioning at all. The port you already made, if moved to an overlay port, will be used both in manifest and classic mode.

@strega-nil-ms mentioned how to do this if you are building with CMakeLists.txt. If you are using visual studio, you do this by specifying additional arguments for vcpkg like so (be sure to change configuration and platform to "all").

A couple additional thoughts/things I've learned.

You probably don't want to directly make modifications to a port in a fork of vcpkg. Eventually you will want to merge upstream changes, and may hit conflicts. Instead, using an overlay port will allow you to keep working, and separately you can work with the vcpkg folks to get your port fixes integrated into the official vcpkg repo. Once those changes are present, you can move to the latest vcpkg, and eliminate your overlay port.

If there's a chance you might be working on two different things that both use vcpkg on the same machine, it can work out really well to make vcpkg a subrepo or submodule of each repo. Then each repo can point to its own instance of vcpkg, and those instances can be at different commits as needed.

Hope this helps. Please ask any questions if anything's not clear. Quite frustrated at myself for somehow losing the nice post I had from yesterday. ;-)

[hajokirchhoff]

First off all, thanks for your informative reply and the time you took typing your answer again :)

Manifest mode does support versioing, and versioning is complicated. That SHA1 you mentioned is actually the git object ID of the port's directory, which lets vcpkg get the portfile at a previous revision as necessary - see https://vcpkg.readthedocs.io/en/latest/specifications/registries-2/

Yes, that's the mistake I have been making. I thought the SHA-1 of the repository was the same as "git rev-parse". It is not. And that is what had been causing the cyclic dependencies in my case.

Rather than deal with all of that, things are /much/ easier if you just use overlay ports. Why? Because overlay ports don't deal with versioning. If a port is present in the overlay ports directory, it always wins. So you need not deal with versioning at all. The port you already made, if moved to an overlay port, will be used both in manifest and classic mode.

The problem I have with port overlay is that they are specific to either a single project or a single machine. My understanding today is this: Port Overlay - useful, if I have a specific port I want to include in exactly one or two project - or if I work alone and have a general port overlay directory - requires manual work for each and every project that should use this overlay But in my case I want to patch boost-serialization and make it available for my entire team for all current and future projects. And I want to make sure that my project and all dependent libraries actually use this patch. So I could either 1. share the port overlay files and instructions with everyone, who would then have to include it in their projects 2. fork the vcpkg and modify the 1.78.0 boost-serialization port for classic and manifest mode, so everyone else can get a copy from our forked vcpkg repository 3. use registries

You probably don't want to directly make modifications to a port in a fork of vcpkg. Eventually you will want to merge upstream changes, and may hit conflicts.

Ah, I don't really mind conflicts here, as my port is pretty straightforward. And if I want to merge upstream changes, I will get conflicts anyway, even if I use a port overlay, because if I want the upstream changes, I need to integrate them into my overlayed port. Might as well do that within a git repository. What do you think?

Instead, using an overlay port will allow you to keep working, and separately you can work with the vcpkg folks to get your port fixes integrated into the official vcpkg repo. Once those changes are present, you can move to the latest vcpkg, and eliminate your overlay port.

Sounds like my idea #1.

If there's a chance you might be working on two different things that both use vcpkg on the same machine, it can work out really well to make vcpkg a subrepo or submodule of each repo. Then each repo can point to its own instance of vcpkg, and those instances can be at different commits as needed.

This seems like my idea #2. I've tried registries, which really seems to be a lot like #2 with more options - which make them more complicated. I haven't tried including the port overlay directory in my project (#1), but I am still leaning towards #2 now, since the port overlay method would require manual work for every new project. Whereas forking vcpkg would automatically give the fixed boost-serialization lib to all my collegues. #3 would mean to have a specific registry (list of versions and ports) for the company. Not difficult (I tried it yesterday), but with registries, again, some additional work for every new project would be needed. It seems to be a lot like having a fork of vcpkg in addition to the official vcpkg - and being able to choose on a per-lib basis which vcpkg to use: the internal forked version or the official version.

Hope this helps. Please ask any questions if anything's not clear. Quite frustrated at myself for somehow losing the nice post I had from yesterday. ;-)

Yes, it helped. Thanks again. BTW, I'd like to improve the vcpkg documentation and perhaps create a comparison between the different ways to achieve what we discussed here. Any suggestion on how I might do that? Clone vcpkg repo in github, add another *.md page and issue a PR request to the vcpkg team? I am starting to like vcpkg a lot, but the documentation is the worst part of it. Which is such a shame.

— Reply to this email directly, view it on GitHub <#12289 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD5IJCWESNPHXL4MK7NJQ5DUXCRZ3ANCNFSM4OR3IIMQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Hajo Kirchhoff Kirchhoff IT-Consulting Fraunhoferstr. 10 91058 Erlangen Germany http://www.litwindow.com Phone: ++49 9131 8277107

[aggieNick02]

Yes, that's the mistake I have been making. I thought the SHA-1 of the
repository was the same as "git rev-parse". It is not. And that is what
had been causing the cyclic dependencies in my case.

Right. Even having learned this once before, I also again thought it was the commit, and it took me a bit of digging when I saw your question before I found the docs explaining what it is. It is part of the "magic" that lets vcpkg reference an older version of the port, but definitely not obvious or easy to find in the docs!

I think your analysis of the pros and cons of the approaches is well thought out. You are right that even if you use port-overlay, you may need to consider what to do when merging the latest vcpkg. I think the merge will be a bit more complex though, just because you'll have to fix up all of the versioning-related files. I think the SHA1 object id for your modified port will change and need to be updated along with the entry in baseline.json .

I'm probably letting other parts of my general experiences with forks and subrepos bias my recommendations here. (The workflow for making a change to a subrepo and getting it merged to the parent fork isn't great.) Having colleagues not need to worry about specifying overlay ports is nice. But I do believe they will still have to enable vcpkg for any new projects.

One thing to point out is that I'm pretty certain that with overlay ports, your project's dependencies will also be using the overlay ports. Basically the way vcpkg works, the entire tree is built with the same set of environment variables.

In my setup, I control all of the projects, so adding overlay ports per project is not a big deal. I also have overlay triplets, which admittedly could also be done by adding new triplets to a fork.

If your goal is to have a repo that your entire team uses and have your port of boost-serialization used by everything automatically, I think doing what you've done and dealing with versioning seems like a good approach. Your analysis of the pros and cons of overlay ports vs forking seems pretty good to me. There is also the registries option, which I've not even looked into, so you know more about that one than me. :-)

If there's a chance you might be working on two different things that
both use vcpkg on the same machine, it can work out really well to
make vcpkg a subrepo or submodule of each repo. Then each repo can
point to its own instance of vcpkg, and those instances can be at
different commits as needed.

This seems like my idea #2.

The point I was trying to make with that comment is that you may have two different repos (or even different revisions of the same repo) on the same machine that need to use different vcpkg "instances" . For this to work, you need vcpkg to itself be part of each repo as a subrepo or submodule, so that the right vcpkg is pointed to in each case, if that makes sense.

BTW, I'd like to improve the vcpkg documentation and perhaps create a
comparison between the different ways to achieve what we discussed here.
Any suggestion on how I might do that? Clone vcpkg repo in github, add
another *.md page and issue a PR request to the vcpkg team?

I am starting to like vcpkg a lot, but the documentation is the worst
part of it. Which is such a shame.

I agree the doucmentation is sometimes not the best. I know there is good effort and even I believe dedicated documentation efforts/folks, and I think part of the problem is just how quickly vcpkg has been evolving and changing. It is really powerful and such a breath of fresh air for doing c++ dependencies on windows, and the features like versioining have happened quickly and the docs often don't reflect reality. I think half of the bugs I have filed have been closed by somebody responding that my issue is actually not an issue anymore, that I just need to do "foo", where "foo" hasn't really been documented. ;-)

I think the topic you've covered here is definitely relevant and worth documenting though. It's made me think critically about the options and what I do for my own projects. I think what you suggest with a PR and then discussion there would be a good place to start, and hopefully it will catch the attention of the right people. Feel free to tag me if you do make a PR and I'll try to contribute what I can.

[BillyONeal]

Fixed by microsoft/vcpkg-tool#743