Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a trait for remote ByteStore network implementation #19050

Merged
merged 6 commits into from
May 23, 2023

Conversation

huonw
Copy link
Contributor

@huonw huonw commented May 19, 2023

This separates the store::remote::ByteStore coordination code from the gRPC REAPI implementation code, by:

  • adding a store::remote::ByteStoreProvider trait
  • moving the REAPI implementation into store::remote::reapi and implementing that trait

This is, in theory, just a lift-and-shift, with no functionality change, other than maybe changing some logging and when exactly a work-unit is created.

This is partial preparation work for supporting more remote stores like GHA cache or S3, for #11149, and is specifically broken out of #17840. Additional work required to actually solve #11149:

  • generalising the action cache too
  • implementing other byte store providers
  • dynamically choosing the right provider for store::remote::ByteStore

The commits are individually reviewable:

  1. preparatory clean-up
  2. define the trait
  3. move the REAPI code and implement the trait, close to naively as possible:
  4. minor clean-up
  5. more major clean-up: inline the method calls into the trait definition, so that they're not just fn method(&self, ...) { self.real_method(...) }:
    • as with 3, this is just a move, without changing the details of the code...
    • ...except for some minor changes like converting ? into .map(|e| e.to_string())? on the get_capabilities call in store_bytes

@huonw huonw added the category:internal CI, fixes for not-yet-released features, etc. label May 19, 2023
@huonw huonw force-pushed the feature/byte-store-trait-2 branch from 91b6c30 to 1800f55 Compare May 19, 2023 07:34
@huonw huonw marked this pull request as ready for review May 19, 2023 07:37
@huonw huonw force-pushed the feature/byte-store-trait-2 branch from 1800f55 to 9bdf435 Compare May 19, 2023 11:11
src/rust/engine/fs/store/src/remote/reapi.rs Outdated Show resolved Hide resolved
src/rust/engine/fs/store/src/remote/reapi.rs Outdated Show resolved Hide resolved
@huonw huonw merged commit cc8be36 into pantsbuild:main May 23, 2023
@huonw huonw deleted the feature/byte-store-trait-2 branch May 23, 2023 00:55
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
@illicitonion illicitonion mentioned this pull request May 26, 2023
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit that referenced this pull request May 27, 2023
Internal changes:

* Fix TypedDict construction
([#19166](#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException
([#19149](#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking
([#19141](#19141))

* Set up remote caching in CI
([#19144](#19144))

* refactor(internal): adjust pants_venv venv_dir calculation
([#19140](#19140))

* Prepare `2.16.0rc4`.
([#19146](#19146))

* Warn about `files()` in PEXes in one place
([#19027](#19027))

* Drop our explicit dep on `syn`
([#19134](#19134))

* Fix auto-cherry-picker's merge-conflicts
([#19125](#19125))

* Use explicit depths on fetches when running cherry-picks in CI
([#19127](#19127))

* Make determine_python() pin to 3.9
([#19116](#19116))

* Upgrade to `actions/checkout@v3`
([#19112](#19112))

* Run the cherry pick script with `-x` for debugging
([#19095](#19095))

* Add more linux x86 test CI shards
([#19099](#19099))

* Use a trait for remote ByteStore network implementation
([#19050](#19050))

* Increase the logging of release version file publishing.
([#19092](#19092))

* Remove the Toolchain plugin
([#18978](#18978))

* Fix bad merge of #18829.
([#19080](#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex"
([#19071](#19071))

* Remove unnecessary interpreter constraint config.
([#19065](#19065))

* Use hand-written change detection for PR/push job trimming
([#19045](#19045))

* Do not build wheels on push, since that is now accomplished by the
release job. ([#19063](#19063))

* Additional release job publishing fixes
([#19058](#19058))

* Change CI trigger to includelist `main` and `2.*.x`
([#19059](#19059))

* Release doc and script fixes
([#19043](#19043))

* Update cherry-picker to use `pull_request_target`
([#19048](#19048))

* Prepare `2.16.0rc3`.
([#19044](#19044))

* Prepare `2.15.1`.
([#19042](#19042))

* Fixup welcome-newcomers.yaml
([#19039](#19039))
huonw added a commit that referenced this pull request Jun 1, 2023
This separates the `remote::remote_cache` coordination code from the
gRPC REAPI implementation by:

- adding a `remote::remote_cache::ActionCacheProvider` trait
- moving the REAPI implementation into `remote::remote_cache::reapi` and
implementing that trait

This is, in theory, just a lift-and-shift, with no functionality change.

This is preparation work for supporting more remote stores like GHA
cache or S3, for #11149, and
is specifically broken out of
#17840. It continues #19050.
Additional work required to actually solve
#11149:

- implementing other byte store and action cache providers
- dynamically choosing the right providers for
`store::remote::ByteStore` and `remote::remote_cache`

The commits are individually reviewable:
1. preparatory breaking out of gRPC code
2. defining the trait
3. move the REAPI code and implement the trait, close to naively as
possible:
  - https://gist.github.com/huonw/a60ad807b05ecea98387294c22de67cb has a
    white-space-ignoring diff between `remote_cache.rs` after 1, and the
    `reapi.rs` file in this commit (it's less useful than #19050, since most
    of the code is deleted, but, buried in there are chunks of completely
    unchanged code)
4. minor clean-up
stuhood pushed a commit that referenced this pull request Sep 7, 2023
This fixes #19732 by restoring the retries when storing hits retryable
server failures from the REAPI remote cache server, which were lost in
the #19050 refactoring.

This also explicitly tests for retries, refactoring `StubCAS` to
generalise `read_request_count` to expose the counts of more requests
than just reads, and also consistently return a `Status::internal(...)`
for the simulated errors.

I think #19050 fortunately landed just after 2.17 was cut, so this
regression only affects the 2.18 pre-releases.
WorkerPants pushed a commit that referenced this pull request Sep 7, 2023
This fixes #19732 by restoring the retries when storing hits retryable
server failures from the REAPI remote cache server, which were lost in
the #19050 refactoring.

This also explicitly tests for retries, refactoring `StubCAS` to
generalise `read_request_count` to expose the counts of more requests
than just reads, and also consistently return a `Status::internal(...)`
for the simulated errors.

I think #19050 fortunately landed just after 2.17 was cut, so this
regression only affects the 2.18 pre-releases.
huonw added a commit that referenced this pull request Sep 7, 2023
…#19737) (#19798)

This fixes #19732 by restoring the retries when storing hits retryable
server failures from the REAPI remote cache server, which were lost in
the #19050 refactoring.

This also explicitly tests for retries, refactoring `StubCAS` to
generalise `read_request_count` to expose the counts of more requests
than just reads, and also consistently return a `Status::internal(...)`
for the simulated errors.

I think #19050 fortunately landed just after 2.17 was cut, so this
regression only affects the 2.18 pre-releases.

Co-authored-by: Huon Wilson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:internal CI, fixes for not-yet-released features, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants