Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify asset caching output #1409

Merged
merged 28 commits into from
Jul 6, 2024

Conversation

JavierMatosD
Copy link
Contributor

@JavierMatosD JavierMatosD commented May 20, 2024

https://devdiv.visualstudio.com/DevDiv/_workitems/edit/2049386

The output is changed such that, without --debug enabled:

  1. Customers can observe a promise that Asset Caching is enabled or disabled.
  2. Customers can observe a promise that x-block-origin is enabled or disabled.
  3. Customers do not observe output that could be interpreted as downloading an external URL. (Unless it is downloading an external URL)

Scenario: asset cache hit!
Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Asset cache hit for cmake-3.29.2-macos-universal.tar.gz; downloaded from: file:///Users/javiermatos/dev/vcpkg-tool/work/testing/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533
Extracting cmake...

Scenario: asset cache miss (configured), x-block-origin disabled
Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Asset cache miss; downloading from https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz
Successfully stored cmake-3.29.2-macos-universal.tar.gz to  file:///Users/javiermatos/dev/vcpkg-tool/work/testing/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533. 
Extracting cmake...

Scenario: asset cache miss (configured), x-block-origin enabled

Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
error: Failed to download from mirror set

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
error: Asset cache miss for cmake-3.29.2-macos-universal.tar.gz and downloads are blocked by x-block-origin.

Scenario: asset cache miss (not configured), x-block-origin enabled

Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
error: Failed to download from mirror set

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
error: Missing cmake-3.29.2-macos-universal.tar.gz and downloads are blocked by x-block-origin.

Scenario: asset cache miss (not configured), x-block-origin disabled

Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Downloading https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...

** When --debug the following promise is printed:**

Asset caching not enabled, x-block-origin disabled

...
[DEBUG] External asset downloads are allowed (x-block-origin is disabled)...
[DEBUG] Asset cache is not configured.
...

Asset caching enabled, x-block-origin enabled

...
[DEBUG] External asset downloads are blocked (x-block-origin is enabled)...
[DEBUG] Asset cache is enabled.
...

@autoantwort
Copy link
Contributor

External asset downloads are allowed (x-block-origin is disabled).
Asset caching is enabled.

Seems to be very spammy to print this every time something is downloaded

Asset cache miss!
Attempting to download from: "https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz".

Asset cache hit!

I would prefer Asset cache hit/miss for <url>. Downloading...

include/vcpkg/base/message-data.inc.h Outdated Show resolved Hide resolved
src/vcpkg/base/downloads.cpp Outdated Show resolved Hide resolved
@BillyONeal
Copy link
Member

Could we make the output look something like this?

Asset Caching Enabled (that means x-azurl)
Scenario: x-azurl Asset Cache Hit!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-block-origin disabled, x-azurl Asset Cache Miss!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Asset cache miss, downloading original

Scenario: x-block-origin enabled, x-azurl Asset Cache Miss!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Asset cache miss and fetching the original is prohibited by x-block-origin

Scenario: x-block-origin enabled, x-azurl Asset Caching Disabled

Can't download https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz because no asset caches are configured and fetching the original is prohibited by x-block-origin

Scenario: x-block-origin disabled, no asset cache configured

Downloading https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-script Asset Cache Hit!

Using script to downloads https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-block-origin disabled, x-script Asset Cache Miss!

???????? Do we try to download https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz anyway??? I don't know what we do.

Scenario: x-block-origin enabled, x-script Asset Cache Miss!

Using script to downloads https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Asset cache miss and fetching the original is prohibited by x-block-origin

@JavierMatosD
Copy link
Contributor Author

Could we make the output look something like this?

Asset Caching Enabled (that means x-azurl) Scenario: x-azurl Asset Cache Hit!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-block-origin disabled, x-azurl Asset Cache Miss!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz Asset cache miss, downloading original

Scenario: x-block-origin enabled, x-azurl Asset Cache Miss!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz Asset cache miss and fetching the original is prohibited by x-block-origin

Scenario: x-block-origin enabled, x-azurl Asset Caching Disabled

Can't download https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz because no asset caches are configured and fetching the original is prohibited by x-block-origin

Scenario: x-block-origin disabled, no asset cache configured

Downloading https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-script Asset Cache Hit!

Using script to downloads https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-block-origin disabled, x-script Asset Cache Miss!

???????? Do we try to download https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz anyway??? I don't know what we do.

Scenario: x-block-origin enabled, x-script Asset Cache Miss!

Using script to downloads https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz Asset cache miss and fetching the original is prohibited by x-block-origin

I've updated the output to include some of the information you've suggested here. It's not one for one but I think it gets the point across.

Copy link
Member

@BillyONeal BillyONeal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't agree with the design change here to hide more things behind --debug

azure-pipelines/end-to-end-tests-dir/asset-caching.ps1 Outdated Show resolved Hide resolved
include/vcpkg/base/message-data.inc.h Outdated Show resolved Hide resolved
src/vcpkg/base/downloads.cpp Show resolved Hide resolved
src/vcpkg/base/downloads.cpp Outdated Show resolved Hide resolved
src/vcpkg/base/downloads.cpp Outdated Show resolved Hide resolved
@JavierMatosD
Copy link
Contributor Author

I don't agree with the design change here to hide more things behind --debug

No longer hidden behind --debug and urls are sanitized :)

@BillyONeal BillyONeal merged commit 9eb99f7 into microsoft:main Jul 6, 2024
5 checks passed
@JavierMatosD JavierMatosD deleted the clarify_asset_caching branch July 20, 2024 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants