Ports Overlay partial implementation (#6981)

* Ports Overlay feature spec * Ports Overlay implementation * [--overlay-ports] Refactor handling of additional paths * Code cleanup * [--overlay-ports] Add help * [depend-info] Support --overlay-ports * Add method to load all ports using PathsPortFileProvider * Make PortFileProvider::load_all_control_files() const * Remove unused code * [vcpkg] Avoid double-load of source control file between Build::perform_and_exit and Build::perform_and_exit_ex * [vcpkg] Clang format * [vcpkg] Fixup build failure introduced in b069ceb * Report errors from Paragraphs::try_load_port()
microsoft · Jun 22, 2019 · f3db66b · f3db66b
1 parent d1b4e88
commit f3db66b
Show file tree

Hide file tree

Showing 30 changed files with 598 additions and 123 deletions.
diff --git a/docs/specifications/ports-overlay.md b/docs/specifications/ports-overlay.md
@@ -0,0 +1,178 @@
+# Ports Overlay (Jun 19, 2019)
+
+
+## 1. Motivation
+
+### A. Allow users to override ports with alternate versions
+
+It's a common scenario for `vcpkg` users to keep specific versions of libraries to use in their own projects. The current recommendation for users is to fork `vcpkg`'s repository and create tags for commits containing the specific versions of the ports they want to use.
+
+This proposal adds an alternative to solve this problem. By allowing `vcpkg` users to specify additional locations in their file system containing ports for:
+
+  * older or newer versions of libraries,
+  * modified libraries, or
+  * libraries not available in `vcpkg`.
+
+These locations will be searched when resolving port names during package installation, and override ports in `<vcpkg-root>/ports`.
+
+### B. Allow users to keep unmodified upstream ports
+
+Users will be able to keep unmodified versions of the ports shipped with `vcpkg` and update them via `vcpkg update` and `vcpkg upgrade` without having to solve merge conflicts.
+
+
+## 2. Other design concerns
+
+* Allow a set of `vcpkg` commands to optionally accept additional paths to be used when searching for ports.
+* Additional paths must take precedence when resolving names of ports to install.
+* Allow users to specify multiple additional paths.
+* Provide a simple disambiguation mechanism to resolve ambiguous port names.
+* After resolving a port name, the installation process has to work the same as for ports shipped by `vcpkg`.
+* This **DOES NOT ENABLE MULTIPLE VERSIONS** of a same library to be **INSTALLED SIDE-BY-SIDE**.
+
+
+## 3. Proposed solution
+
+This document proposes allowing additional locations to search for ports during package installation that will override and complement the set of ports provided by `vcpkg` (ports under the `<vcpkg_root>/ports` directory).`
+
+A new option `--overlay-ports` will be added to the `vcpkg install`, `vcpkg update`, `vcpkg upgrade`, `vcpkg export`, and `vcpkg depend-info` commands to specify additional paths containing ports. 
+
+From a user experience perspective, a user expresses interest in adding additional lookup locations by passing the `--overlay-ports` option followed by a path to:
+
+* an individual port (directory containing a `CONTROL` file),
+  * `vcpkg install sqlite3 --overlay-ports="C:\custom-ports\sqlite3"`
+
+* a directory containing ports,
+  * `vcpkg install sqlite3 --overlay-ports=\\share\org\custom-ports`
+
+* a file listing paths to the former two.
+  * `vcpkg install sqlite3 --overlay-ports=..\port-repos.txt`
+
+    _port-repos.txt_
+
+    ```
+    .\experimental-ports\sqlite3
+    C:\custom-ports
+    \\share\team\custom-ports
+    \\share\org\custom-ports
+    ```
+    *Relative paths inside this file are resolved relatively to the file's location. In this case a `experimental-ports` directory should exist at the same level as the `port-repos.txt` file.*
+
+_NOTE: It is not the goal of this document to discuss library versioning or project dependency management solutions, which require the ability to install multiple versions of a same library side-by-side._ 
+
+### Multiple additional paths 
+
+Users can provide multiple additional paths by repeating the `--overlay-ports` option.
+
+```
+vcpkg install sqlite3 
+    --overlay-ports="..\experimental-ports\sqlite3" 
+    --overlay-ports="C:\custom-ports" 
+    --overlay-ports="\\share\team\custom-ports
+```
+
+### Overlaying ports
+
+Port name resolution follows the order in which additional paths are specified, with the first match being selected for installation, and falling back to `<vcpkg-root>/ports` if the port is not found in any of the additional paths.
+
+No effort is made to compare version numbers inside the `CONTROL` files, or to determine which port contains newer or older files.
+
+### Examples
+
+Given the following directory structure:
+
+  ```
+  team-ports/
+  |-- sqlite3/
+  |---- CONTROL
+  |-- rapidjson/
+  |---- CONTROL
+  |-- curl/
+  |---- CONTROL
+
+  my-ports/
+  |-- sqlite3/
+  |---- CONTROL
+  |-- rapidjson/
+  |---- CONTROL
+
+  vcpkg
+  |-- ports/
+  |---- <upstream ports>
+  |-- vcpkg.exe
+  |-- preferred-ports.txt
+  ```
+* #### Example #1:
+
+  Running:
+
+  ```
+  vcpkg/vcpkg.exe install sqlite3 --overlay-ports=my-ports --overlay-ports=team-ports
+  ```
+
+  Results in `my-ports/sqlite3` getting installed as that location appears first in the command line arguments.
+
+* #### Example #2:
+  
+  A specific version of a port can be given priority by adding its path first in the list of arguments:
+
+  ```
+  vcpkg/vcpkg.exe install sqlite3 rapidjson curl 
+      --overlay-ports=my-ports/rapidjson 
+      --overlay-ports=vcpkg/ports/curl
+      --overlay-ports=team-ports
+  ```
+
+  Installs:
+    * `sqlite3` from `team-ports/sqlite3`
+    * `rapidjson` from `my-ports/rapidjson`
+    * `curl` from `vcpkg/ports/curl`
+
+* #### Example #3:
+
+  Given the content of `preferred-ports.txt` as:
+
+  ```
+  ./ports/curl
+  /my-ports/rapidjson
+  /team-ports
+  ```
+
+  A location can be appended or prepended to those included in `preferred-ports.txt` via the command line, like this:
+
+  ```
+  vcpkg/vcpkg.exe install sqlite3 curl --overlay-ports=my-ports --overlay-ports=vcpkg/preferred-ports.txt
+  ```
+
+  Which results in `my-ports/sqlite3` and `vcpkg/ports/curl` getting installed.
+
+
+## 4. Proposed User experience
+
+### i. User wants to preserve an older version of a port
+
+Developer Alice and her team use `vcpkg` to acquire **OpenCV** and some other packages. She has even contributed many patches to add features to the **OpenCV 3** port in `vcpkg`. But, one day, she notices that a PR to update **OpenCV** to the next major version has been merged. 
+
+Alice wants to update some packages available in `vcpkg`. Unfortunately, updating her project to use the latest **OpenCV** is not immediately possible. 
+
+Alice creates a private GitHub repository and checks in the set of ports that she wants to preserve. Then provides her teammates with the link to clone her private ports repository.
+
+```
+mkdir vcpkg-custom-ports
+cd vcpkg-custom-ports
+git init 
+cp -r %VCPKG_ROOT%/ports/opencv .
+git add .
+git commit -m "[opencv] Add OpenCV 3 port"
+git remote add origin https://github.com/<Alice's GitHub username>/vcpkg-custom-ports.git
+git push -u origin master
+```
+
+Now her team is able to use: 
+
+```
+git clone https://github.com/<Alice's GitHub username>/vcpkg-custom-ports.git
+vcpkg update --overlay-ports=./vcpkg-custom-ports
+vcpkg upgrade --no-dry-run --overlay-ports=./vcpkg-custom-ports
+``` 
+
+to upgrade their packages and preserve the old version of **OpenCV** they require.
diff --git a/toolsrc/include/vcpkg/build.h b/toolsrc/include/vcpkg/build.h
@@ -20,7 +20,7 @@ namespace vcpkg::Build
     namespace Command
     {
         void perform_and_exit_ex(const FullPackageSpec& full_spec,
-                                 const fs::path& port_dir,
+                                 const SourceControlFileLocation& scfl,
                                  const ParsedArguments& options,
                                  const VcpkgPaths& paths);
 

diff --git a/toolsrc/include/vcpkg/dependencies.h b/toolsrc/include/vcpkg/dependencies.h
@@ -48,7 +48,7 @@ namespace vcpkg::Dependencies
                           const RequestType& request_type);
 
         InstallPlanAction(const PackageSpec& spec,
-                          const SourceControlFile& scf,
+                          const SourceControlFileLocation& scfl,
                           const std::set<std::string>& features,
                           const RequestType& request_type,
                           std::vector<PackageSpec>&& dependencies);
@@ -57,7 +57,7 @@ namespace vcpkg::Dependencies
 
         PackageSpec spec;
 
-        Optional<const SourceControlFile&> source_control_file;
+        Optional<const SourceControlFileLocation&> source_control_file_location;
         Optional<InstalledPackageView> installed_package;
 
         InstallPlanType plan_type;
@@ -129,26 +129,31 @@ namespace vcpkg::Dependencies
 
     struct PortFileProvider
     {
-        virtual Optional<const SourceControlFile&> get_control_file(const std::string& src_name) const = 0;
+        virtual Optional<const SourceControlFileLocation&> get_control_file(const std::string& src_name) const = 0;
+        virtual std::vector<const SourceControlFileLocation*> load_all_control_files() const = 0;
     };
 
     struct MapPortFileProvider : Util::ResourceBase, PortFileProvider
     {
-        explicit MapPortFileProvider(const std::unordered_map<std::string, SourceControlFile>& map);
-        Optional<const SourceControlFile&> get_control_file(const std::string& src_name) const override;
+        explicit MapPortFileProvider(const std::unordered_map<std::string, SourceControlFileLocation>& map);
+        Optional<const SourceControlFileLocation&> get_control_file(const std::string& src_name) const override;
+        std::vector<const SourceControlFileLocation*> load_all_control_files() const override;
 
     private:
-        const std::unordered_map<std::string, SourceControlFile>& ports;
+        const std::unordered_map<std::string, SourceControlFileLocation>& ports;
     };
 
     struct PathsPortFileProvider : Util::ResourceBase, PortFileProvider
     {
-        explicit PathsPortFileProvider(const VcpkgPaths& paths);
-        Optional<const SourceControlFile&> get_control_file(const std::string& src_name) const override;
+        explicit PathsPortFileProvider(const vcpkg::VcpkgPaths& paths, 
+                                       const std::vector<std::string>* ports_dirs_paths);
+        Optional<const SourceControlFileLocation&> get_control_file(const std::string& src_name) const override;
+        std::vector<const SourceControlFileLocation*> load_all_control_files() const override;
 
     private:
-        const VcpkgPaths& ports;
-        mutable std::unordered_map<std::string, SourceControlFile> cache;
+        Files::Filesystem& filesystem;
+        std::vector<fs::path> ports_dirs;
+        mutable std::unordered_map<std::string, SourceControlFileLocation> cache;
     };
 
     struct ClusterGraph;
@@ -181,7 +186,7 @@ namespace vcpkg::Dependencies
     std::vector<ExportPlanAction> create_export_plan(const std::vector<PackageSpec>& specs,
                                                      const StatusParagraphs& status_db);
 
-    std::vector<AnyAction> create_feature_install_plan(const std::unordered_map<std::string, SourceControlFile>& map,
+    std::vector<AnyAction> create_feature_install_plan(const std::unordered_map<std::string, SourceControlFileLocation>& map,
                                                        const std::vector<FeatureSpec>& specs,
                                                        const StatusParagraphs& status_db);
 

diff --git a/toolsrc/include/vcpkg/postbuildlint.h b/toolsrc/include/vcpkg/postbuildlint.h
@@ -9,5 +9,6 @@ namespace vcpkg::PostBuildLint
     size_t perform_all_checks(const PackageSpec& spec,
                               const VcpkgPaths& paths,
                               const Build::PreBuildInfo& pre_build_info,
-                              const Build::BuildInfo& build_info);
+                              const Build::BuildInfo& build_info,
+                              const fs::path& port_dir);
 }
diff --git a/toolsrc/include/vcpkg/sourceparagraph.h b/toolsrc/include/vcpkg/sourceparagraph.h
@@ -68,6 +68,15 @@ namespace vcpkg
         Optional<const FeatureParagraph&> find_feature(const std::string& featurename) const;
     };
 
+    /// <summary>
+    /// Full metadata of a package: core and other features. As well as the location the SourceControlFile was loaded from.
+    /// </summary>
+    struct SourceControlFileLocation
+    {
+        std::unique_ptr<SourceControlFile> source_control_file;
+        fs::path source_location;
+    };
+
     void print_error_message(Span<const std::unique_ptr<Parse::ParseControlErrorInfo>> error_info_list);
     inline void print_error_message(const std::unique_ptr<Parse::ParseControlErrorInfo>& error_info_list)
     {

diff --git a/toolsrc/include/vcpkg/vcpkgcmdarguments.h b/toolsrc/include/vcpkg/vcpkgcmdarguments.h
@@ -15,6 +15,7 @@ namespace vcpkg
     {
         std::unordered_set<std::string> switches;
         std::unordered_map<std::string, std::string> settings;
+        std::unordered_map<std::string, std::vector<std::string>> multisettings;
     };
 
     struct VcpkgPaths;
@@ -41,10 +42,22 @@ namespace vcpkg
         StringLiteral short_help_text;
     };
 
+    struct CommandMultiSetting
+    {
+        constexpr CommandMultiSetting(const StringLiteral& name, const StringLiteral& short_help_text)
+            : name(name), short_help_text(short_help_text)
+        {
+        }
+
+        StringLiteral name;
+        StringLiteral short_help_text;
+    };
+
     struct CommandOptionsStructure
     {
         Span<const CommandSwitch> switches;
         Span<const CommandSetting> settings;
+        Span<const CommandMultiSetting> multisettings;
     };
 
     struct CommandStructure
@@ -74,6 +87,7 @@ namespace vcpkg
 
         std::unique_ptr<std::string> vcpkg_root_dir;
         std::unique_ptr<std::string> triplet;
+        std::unique_ptr<std::vector<std::string>> overlay_ports;
         Optional<bool> debug = nullopt;
         Optional<bool> sendmetrics = nullopt;
         Optional<bool> printmetrics = nullopt;
@@ -88,6 +102,6 @@ namespace vcpkg
         ParsedArguments parse_arguments(const CommandStructure& command_structure) const;
 
     private:
-        std::unordered_map<std::string, Optional<std::string>> optional_command_arguments;
+        std::unordered_map<std::string, Optional<std::vector<std::string>>> optional_command_arguments;
     };
 }
diff --git a/toolsrc/include/vcpkg/vcpkgpaths.h b/toolsrc/include/vcpkg/vcpkgpaths.h
@@ -50,8 +50,6 @@ namespace vcpkg
         static Expected<VcpkgPaths> create(const fs::path& vcpkg_root_dir, const std::string& default_vs_path);
 
         fs::path package_dir(const PackageSpec& spec) const;
-        fs::path port_dir(const PackageSpec& spec) const;
-        fs::path port_dir(const std::string& name) const;
         fs::path build_info_file_path(const PackageSpec& spec) const;
         fs::path listfile_path(const BinaryParagraph& pgh) const;
 

diff --git a/toolsrc/src/tests.plan.cpp b/toolsrc/src/tests.plan.cpp
@@ -47,7 +47,8 @@ namespace UnitTest1
 
         Assert::AreEqual(plan.spec.triplet().to_string().c_str(), triplet.to_string().c_str());
 
-        Assert::AreEqual(pkg_name.c_str(), plan.source_control_file.get()->core_paragraph->name.c_str());
+        auto* scfl = plan.source_control_file_location.get();
+        Assert::AreEqual(pkg_name.c_str(), scfl->source_control_file->core_paragraph->name.c_str());
         Assert::AreEqual(size_t(vec.size()), feature_list.size());
 
         for (auto&& feature_name : vec)
@@ -79,7 +80,7 @@ namespace UnitTest1
     /// </summary>
     struct PackageSpecMap
     {
-        std::unordered_map<std::string, SourceControlFile> map;
+        std::unordered_map<std::string, SourceControlFileLocation> map;
         Triplet triplet;
         PackageSpecMap(const Triplet& t = Triplet::X86_WINDOWS) noexcept { triplet = t; }
 
@@ -94,7 +95,8 @@ namespace UnitTest1
         {
             auto spec = PackageSpec::from_name_and_triplet(scf.core_paragraph->name, triplet);
             Assert::IsTrue(spec.has_value());
-            map.emplace(scf.core_paragraph->name, std::move(scf));
+            map.emplace(scf.core_paragraph->name,
+                        SourceControlFileLocation{std::unique_ptr<SourceControlFile>(std::move(&scf)), ""});
             return PackageSpec{*spec.get()};
         }
     };