[EngSys] Get Vcpkg automatically

CONTRIBUTING.md

@@ -1,4 +1,4 @@
-## Azure SDK for C++ Contributing Guide
+# Azure SDK for C++ Contributing Guide
 
 Thank you for your interest in contributing to Azure SDK for C++.
 
@@ -39,9 +39,9 @@ Thank you for your interest in contributing to Azure SDK for C++.
   - Contributor is using an e-mail address other than the primary GitHub address and wants that preserved in the history. Contributor must be willing to squash
     the commits manually before acceptance.
 
-## Developer Guide
+# Developer Guide
 
-### Codespaces
+## Codespaces
 
 Codespaces is new technology that allows you to use a container as your development environment. This repo provides a Codespaces container which is supported by both GitHub Codespaces and VS Code Codespaces.
 
@@ -50,54 +50,54 @@ Codespaces is new technology that allows you to use a container as your developm
 1. From the Azure SDK GitHub repo, click on the "Code -> Open with Codespaces" button.
 1. Open a Terminal. The development environment will be ready for you. Continue to [Building and Testing](https://github.com/Azure/azure-sdk-for-cpp/blob/main/CONTRIBUTING.md#building-and-testing).
 
-#### VS Code Codespaces
+### VS Code Codespaces
 
 1. Install the [VS Code Remote Extension Pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack)
 1. When you open the Azure SDK for C++ repo in VS Code, it will prompt you to open the project in the Dev Container. If it does not prompt you, then hit CTRL+P, and select "Remote-Containers: Open Folder in Container..."
 1. Open a Terminal. The development environment will be ready for you. Continue to [Building and Testing](https://github.com/Azure/azure-sdk-for-cpp/blob/main/CONTRIBUTING.md#building-and-testing).
 
-### Full Local Setup
+## Full Local Setup
 
-#### Pre-requisites
+### Pre-requisites
 
-##### CMake
+#### CMake
 
 CMake version 3.13 or higher is required to build these libraries. Download and install CMake from the project's
 [website](https://cmake.org/download/).
 
-##### Third Party Dependencies
+### Third Party Dependencies
 
-- curl
-- libxml2
-- clang-format (min version 10)
+Azure SDK uses Vcpkg manifest mode to declare the [list of required 3rd party dependencies](https://github.com/Azure/azure-sdk-for-cpp/blob/main/vcpkg.json) for building the SDK service libraries. It will also get and set up Vcpkg automatically. **You can move on to [Building the project](#building-the-project)** and skip the next part if you are not interested in learning about alternatives for setting up dependencies.
 
-Vcpkg can be used to install the Azure SDK for CPP dependencies into a specific folder on the system instead of globally installing them.
-Follow [vcpkg install guide](https://github.com/microsoft/vcpkg#getting-started) to get vcpkg and install the following dependencies:
+#### Customize the Vcpkg dependency integration
 
-```sh
-./vcpkg install curl libxml2
-```
-
-When using vcpkg, you can set the `VCPKG_ROOT` environment variable to the vcpkg Git repository folder. This would automatically set the CMake variable `CMAKE_TOOLCHAIN_FILE` for you, enabling the project to use any library installed with vcpkg.
+If the CMake option _-DCMAKE_TOOLCHAIN_FILE=..._ is not defined to generate the project, the Azure SDK project will automatically get Vcpkg and link it to get its dependencies. You can use the next environment variables to change this behavior:
 
-The Azure SDK for C++ uses [this vcpkg release version](https://github.com/Azure/azure-sdk-for-cpp/blob/main/eng/vcpkg-commit.txt) for continuous integration (CI) building and testing. Make sure to checkout this version when following the next steps for building and running the Azure SDK for C++. Using a newer vcpkg version might still work, however, if it is tested.
-
-```sh
-# Checking out vcpkg release version before installing dependencies
+<center>
 
-git clone https://github.com/Microsoft/vcpkg.git
-cd vcpkg
-# Checkout the vcpkg commit from the vcpkg-commit.txt file (link above)
-git checkout <vcpkg commit>
+<table>
+<tr>
+<td>Environment Variable</td>
+<td>Description</td>
+</tr>
+<tr>
+<td>AZURE_SDK_DISABLE_AUTO_VCPKG</td>
+<td>When defined, Vcpkg won't be automatically cloned and linked. Use this setting, for example, if your dependencies are installed on the system and you don't need to get them.</td>
+</tr>
+<tr>
+<td>AZURE_SDK_VCPKG_COMMIT</td>
+<td>This variable can set the git commit id to be used when automatically cloning Vcpkg.</td>
+</tr>
+<tr>
+<td>VCPKG_ROOT</td>
+<td>Use this variable to set an existing Vcpkg folder from your system to be linked for building. Use this, for example, when working with Vcpkg classic mode, to switch between different Vcpkg folders.</td>
+</tr>
+</table>
 
-# build vcpkg (showing Linux command, see vcpkg getting started for Windows)
-./bootstrap-vcpkg.sh
-./vcpkg install curl libxml2
-```
+</center>
 
-### Building and Testing
 
-#### Building the project
+## Building the project
 
 Generate the CMake files and build as you would with any standard CMake project. From the
 repo root, run:

README.md

@@ -14,63 +14,146 @@ For API reference docs, tutorials, samples, quick starts, and other documentatio
 
 ### Download & Install the SDK
 
-The easiest way to acquire the C++ SDK is leveraging vcpkg package manager. You will need to install [Git](https://git-scm.com/downloads) before getting started.
+Here are some alternatives, from easiest to advanced, how you can get, build and integrate Azure SDK clients to your application.
 
-First clone and bootstrap vcpkg itself. You can install it anywhere on your machine, but **make note** of the directory where you clone the vcpkg repo.
+#### CMake Project + Vcpkg - manifest mode
 
-```cmd
-> git clone https://github.com/microsoft/vcpkg
+The easiest way to acquire the C++ SDK is leveraging [vcpkg](https://github.com/microsoft/vcpkg#getting-started) package manager. You will need to install [Git](https://git-scm.com/downloads) before getting started.
+
+##### 1. Create a [CMake](https://cmake.org/cmake/help/latest/) project
+
+CMake will take care of cross-operating system support.
+
+> Visual Studio installs CMake without adding it to the path. You need to [install CMake](https://cmake.org/download/) if you are not using Visual Studio or if you want to use a command line outside Visual Studio.
+
+Visual Studio:
+
+If you are using Visual Studio and you installed [support for CMake](https://docs.microsoft.com/cpp/build/cmake-projects-in-visual-studio?view=vs-2019), you can create a new CMake Project from Visual Studio, new project menu.
+
+-IMAGE HERE Visual Studio-
+
+Visual Studio Code:
+
+Install the VSCode extensions: [CMake](https://marketplace.visualstudio.com/items?itemName=twxs.cmake) and [CMake Tools](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools). Then, create folder for your project and open it with VSCode. Press `F1` and type _CMake: Quick Start_, follow the steps to give a name to your project, to select a compiler and any other initial configuration.
+
+-IMAGE HERE VSCode-
+
+> You can also manually create the root `CMakeLists.txt` with your own initial configuration and source.
+
+##### 2. Link the Vcpkg toolchain file to your CMake project
+
+Azure SDK provides a CMake module that you can use for your application. You only need to create a folder called _cmake-modules_ on the top level of your CMake project and copy [AzureVcpkg.cmake](https://github.com/Azure/azure-sdk-for-cpp/blob/main/cmake-modules/AzureVcpkg.cmake) to this folder.
+
+The AzureVcpkg module supports three scenarios:
+
+1. Getting and setting up Vcpkg automatically (default case). You can set the env var `AZURE_SDK_DISABLE_AUTO_VCPKG` to disable this bahavior.
+2. Automatically linking your application to an existing Vcpkg folder. Set the environment variable `VCPKG_ROOT` to the Vcpkg folder you want to link.
+3. Manually setting a toolchain file with cmake command option. `AzureVcpkg.cmake` module will respect the option.
+
+Add the next lines to your root `CMakeLists.txt` to use `AzureVcpkg.cmake` module:
+
+```cmake
+# Add this lines on the top, before the call to `project(name VERSION 0.0.0)
+list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake-modules")
+include(AzureVcpkg)
+az_vcpkg_integrate()
 ```
 
-On Windows:
+##### 3. Add Vcpkg manifest
+
+Add a new file called `vcpkg.json` on the root of your CMake project and add the Azure SDK clients you want to use in your application. For example, the next manifest defines Azure Identity and Blobs.
 
-```cmd
-> .\vcpkg\bootstrap-vcpkg.bat
+```json
+{
+    "name": "your-app-name",
+    "version-string": "<Your app version like 1.0.0>", 
+    "dependencies": [
+        "azure-identity-cpp",
+        "azure-storage-blobs-cpp"
+    ]
+}
 ```
 
-On Linux:
+##### 4. Link Azure SDK libraries to your application
+
+Add the next lines to your `CMakeLists.txt` file. It must be added after the cmake target name is defined.
 
-```sh
-> ./vcpkg/bootstrap-vcpkg.sh
+```cmake
+find_package(azure-identity-cpp CONFIG REQUIRED)
+find_package(azure-storage-blobs-cpp CONFIG REQUIRED)
+target_link_libraries(quick-sample PRIVATE Azure::azure-identity Azure::azure-storage-blobs)
 ```
 
-To install the libraries for your project, run the following, optionally specifying the triplet. For example, the following will install packages for the `x64-windows` triplet. On Windows, not specifying a triplet will default to `x86-windows`:
+> See the list of available SDK clients for C++ [here](https://azure.github.io/azure-sdk/releases/latest/cpp.html)
+
+
+##### 5. Generate project and compile
 
-```cmd
-> .\vcpkg\vcpkg install azure-storage-blobs-cpp:x64-windows
+At this point, you can press F7 on Visual Studio or VSCode to generate and build the project. Or you can also run the following commands from a command line:
+
+```bash
+# Create a build folder (if there's not one already there)
+mkdir build
+cd build
+cmake ..
+cmake --build .
 ```
 
-See the [list of packages](https://github.com/Azure/azure-sdk-for-cpp#vcpkg) available via vcpkg below. All Azure C++ SDK package names start with `azure-`. You can also search for the libraries you need with the `search` command. For example:
+> Using Vcpkg manifest makes easy to define multiple dependencies and delegate building them to Vcpkg.
+
+#### CMake Project + fetch content
+
+For this scenario, CMake will fetch the Azure SDK source code and make it part of your project. THe SDK client libraries will be compiled at the same time as your application.
+
+Follow the step 1 from above to create a CMake project first.
+
+###### 2. Define CMake fetch content
 
-```cmd
-> .\vcpkg\vcpkg search azure-
+Add the following code to your root `CMakeLists.txt` file:
+
+```cmake
+# Add this code before creating and linking your application
+
+include(FetchContent)
+FetchContent_Declare(
+    azuresdk
+    GIT_REPOSITORY      https://github.com/Azure/azure-sdk-for-cpp.git
+    GIT_TAG             <Release Tag or Git-Commit-Here>
+)
+FetchContent_GetProperties(azuresdk)
+if(NOT azuresdk_POPULATED)
+    FetchContent_Populate(azuresdk)
+    # Adding all Azure SDK libraries
+    add_subdirectory(${azuresdk_SOURCE_DIR} ${azuresdk_BINARY_DIR} EXCLUDE_FROM_ALL)
+    # Adding one Azure SDK Library only (Storage blobs)
+    # add_subdirectory(${azuresdk_SOURCE_DIR}/sdk/storage/azure-storage-blobs ${azuresdk_BINARY_DIR} EXCLUDE_FROM_ALL)
+endif()
 ```
 
-Once the library is installed, follow the instructions from the console output to include the library in your `CMake` application. For example, to include `azure-storage-blobs-cpp`, add the following to your `CMakeLists.txt` file:
+##### 3. Link Azure SDK libraries to your application
 
-```CMake
-find_package(azure-storage-blobs-cpp CONFIG REQUIRED)
-target_link_libraries(<your project name> PRIVATE Azure::azure-storage-blobs)
+The only difference from the previous scenario is that you don't need to call `find_package()`, since the cmake targets are integrated to your project. So you only need:
+
+```cmake
+# After creating the cmake target
+target_link_libraries(quick-sample PRIVATE Azure::azure-identity Azure::azure-storage-blobs)
 ```
 
-> NOTE: All the Azure client libraries take a dependency on `azure-core-cpp` which provides functionality commonly needed by all Azure clients. When you install any client library via vcpkg, it will bring in all the necessary dependencies as well. You don't need to install those individually to get started.
+> Note: You need to take care of getting the Azure SDK dependencies on your own. Either manually installing them or by integrating the source code to your project as well.
 
-You can reference this [vcpkg Quick Start](https://github.com/microsoft/vcpkg#quick-start-windows) for more details.
+Use step 5 from previous scenario to generate and build your project.
 
-#### Getting Beta Releases in Vcpkg
+> This scenario requires extra manual configuration to get dependencies, but it is useful as an alternative when Vcpkg is not available
 
-Official vcpkg registry may have beta versions of Azure SDK client libraries, up until a given library gets released as stable. After that, we don't publish post-first-stable beta releases of that library in the official registry.
+#### Other combinations
 
-If you are interested in both stable releases and post-first-stable beta releases, see [Azure SDK Beta Vcpkg Registry](https://github.com/Azure/azure-sdk-vcpkg-betas/).
+It should be possible to create your application without a CMake project. For example, manually cloning Azure SDK, building libraries and finally linking them to your application. However, this is considered an advanced scenario and it is not either described or maintained (The other scenarios described below are validated with CI pipelines).
 
-### Building your Application
+#### Getting Beta Releases in Vcpkg
 
-In order to use the SDK installed via vcpkg with CMake, you can use the toolchain file from vcpkg:
+Official vcpkg registry may have beta versions of Azure SDK client libraries, up until a given library gets released as stable. After that, we don't publish post-first-stable beta releases of that library in the official registry.
 
-```cmd
-> cmake -B [build directory] -S . -DCMAKE_TOOLCHAIN_FILE=[path to vcpkg repo]/scripts/buildsystems/vcpkg.cmake
-> cmake --build [build directory]
-```
+If you are interested in both stable releases and post-first-stable beta releases, see [Azure SDK Beta Vcpkg Registry](https://github.com/Azure/azure-sdk-vcpkg-betas/). You can update the `AzureVcpkg.cmake` module to use the beta registry.
 
 #### Using the SDK within your Application
 

cmake-modules/AzureVcpkg.cmake

@@ -5,14 +5,45 @@
 set(AZ_ROOT_DIR "${CMAKE_CURRENT_LIST_DIR}/..")
 
 macro(az_vcpkg_integrate)
-  # vcpkg Integration
-  if(DEFINED ENV{VCPKG_ROOT} AND NOT DEFINED CMAKE_TOOLCHAIN_FILE)
-    set(CMAKE_TOOLCHAIN_FILE "$ENV{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"
-        CACHE STRING "")
-  elseif(DEFINED ENV{VCPKG_INSTALLATION_ROOT} AND NOT DEFINED CMAKE_TOOLCHAIN_FILE)
-    set(CMAKE_TOOLCHAIN_FILE "$ENV{VCPKG_INSTALLATION_ROOT}/scripts/buildsystems/vcpkg.cmake"
-        CACHE STRING "")
+  # AUTO CMAKE_TOOLCHAIN_FILE:
+  #   User can call `cmake -DCMAKE_TOOLCHAIN_FILE="path_to_the_toolchain"` as the most specific scenario.
+  #   An env var VCPKG_ROOT or VCPKG_INSTALLATION_ROOT can be set to let Azure SDK to set the VCPKG toolchain automatically.
+  #   As the last alternative (default case), Azure SDK will automatically clone VCPKG folder and set toolchain from there.
+  if(NOT DEFINED CMAKE_TOOLCHAIN_FILE)
+    if(DEFINED ENV{VCPKG_ROOT})
+      set(CMAKE_TOOLCHAIN_FILE "$ENV{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"
+          CACHE STRING "")
+    elseif(DEFINED ENV{VCPKG_INSTALLATION_ROOT})
+      set(CMAKE_TOOLCHAIN_FILE "$ENV{VCPKG_INSTALLATION_ROOT}/scripts/buildsystems/vcpkg.cmake"
+          CACHE STRING "")
+    else()
+      # Set AZURE_SDK_DISABLE_AUTO_VCPKG env var to avoid Azure SDK from cloning and setting VCPKG automatically
+      # This option delegate package's dependencies installation to user.
+      if(NOT DEFINED ENV{AZURE_SDK_DISABLE_AUTO_VCPKG})
+        # GET VCPKG FROM SOURCE
+        #  User can set env var AZURE_SDK_VCPKG_COMMIT to pick the VCPKG commit to fetch
+        set(VCPKG_COMMIT_STRING f0aa678b7471497f1adedcc99f40e1599ad22f69) # default SDK tested commit
+        if(DEFINED ENV{AZURE_SDK_VCPKG_COMMIT})
+          set(VCPKG_COMMIT_STRING "$ENV{AZURE_SDK_VCPKG_COMMIT}") # default SDK tested commit
+        endif()
+        include(FetchContent)
+        FetchContent_Declare(
+            vcpkg
+            GIT_REPOSITORY      https://github.com/microsoft/vcpkg.git
+            GIT_TAG             ${VCPKG_COMMIT_STRING}
+            )
+        FetchContent_GetProperties(vcpkg)
+        # make sure to pull vcpkg only once.
+        if(NOT vcpkg_POPULATED) 
+            FetchContent_Populate(vcpkg)
+        endif()
+        # use the vcpkg source path 
+        set(CMAKE_TOOLCHAIN_FILE "${vcpkg_SOURCE_DIR}/scripts/buildsystems/vcpkg.cmake" CACHE STRING "")
+      endif()
+    endif()
   endif()
+
+  # enable triplet customization
   if(DEFINED ENV{VCPKG_DEFAULT_TRIPLET} AND NOT DEFINED VCPKG_TARGET_TRIPLET)
     set(VCPKG_TARGET_TRIPLET "$ENV{VCPKG_DEFAULT_TRIPLET}" CACHE STRING "")
   endif()

eng/pipelines/templates/jobs/archetype-sdk-client.yml

@@ -93,9 +93,6 @@ jobs:
           workingDirectory: $(Agent.TempDirectory)
           displayName: Download and Install Doxygen
 
-        # Install dependencies required for build from vcpkg
-        # TODO: We should be able to generate docs without having to install these
-        # prerequisites:
         - template: /eng/pipelines/templates/steps/vcpkg.yml
 
         - template: /eng/pipelines/templates/steps/cmake-build.yml

eng/pipelines/templates/steps/vcpkg.yml

@@ -1,16 +1,5 @@
 steps:
   - pwsh: |
-      $TargetPath = "$(Agent.TempDirectory)/vcpkg"
-      Remove-Item -Path $TargetPath -Recurse -Force -ErrorAction Ignore
-      New-Item -ItemType Directory -Path $TargetPath  -Force
-
-      $VcpkgCommit = $(Get-Content eng/vcpkg-commit.txt)
-      Write-Host "Target Path for vcpkg: $TargetPath"
-      Write-Host "Vcpkg SHA: $VcpkgCommit"
-
-      Write-Host "##vso[task.prependpath]$TargetPath"
-      Write-Host "##vso[task.setvariable variable=VCPKG_INSTALLATION_ROOT]$TargetPath"
-      Write-Host "##vso[task.setvariable variable=VcpkgCommit]$VcpkgCommit"
       Write-Host "##vso[task.setvariable variable=VCPKG_BINARY_SOURCES_SECRET;issecret=true;]clear;x-azblob,https://cppvcpkgcache.blob.core.windows.net/public-vcpkg-container,,read"
     displayName: Set Vcpkg Variables
 
@@ -22,13 +11,3 @@ steps:
       arguments: -StorageAccountKey '$(cpp-vcpkg-cache-storage-key)'
     displayName: Set Vcpkg Write-mode Cache
     condition: and(succeeded(), eq(variables['System.TeamProject'], 'internal'))
-
-  - task: PowerShell@2
-    inputs:
-      targetType: filePath
-      filePath: eng/scripts/vcpkg.ps1
-      arguments: >-
-        -Ref $(VcpkgCommit)
-        -VcpkgPath $(VCPKG_INSTALLATION_ROOT)
-      pwsh: true
-    displayName: Clone Vcpkg.