Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/dev/null in action cache, remote caches, etc ( /cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ) #13349

Closed
jerrymarino opened this issue Apr 13, 2021 · 2 comments
Assignees
Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: feature request

Comments

@jerrymarino
Copy link
Contributor

jerrymarino commented Apr 13, 2021

Description of the problem / feature request:

Hello! I noticed today about 100x more requests for a certian object in a remote http cache. /cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

git clone https://github.com/bazelbuild/rules_apple.git
cd rules_apple 
bazel build examples/ios/PrenotCalculator:PrenotCalculator --disk_cache=/tmp/disk-cache --appe_platform_type=ios
grep -r -a 2 e3b0c /tmp/disk_cache/

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

Happens on 3x and 4x releases, dev builds

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/bazelbuild/rules_apple.git

Have you found anything relevant by searching the web?

#11063
#6260

Any other information, logs, or outputs that you want to share?

$ grep -r -B 2 e3b0c /tmp/disk_cache/
/tmp/disk_cache//ac/ac/acc0ca28dbfd1fbba3851db7fab5db5e423dbc2e1e15a5d35a5fe8d8052439b7-Hbazel-out/apl-ios_x86_64-dbg/bin/tests/ios/app/_objs/App_objc/arc/main.dF
/tmp/disk_cache//ac/ac/acc0ca28dbfd1fbba3851db7fab5db5e423dbc2e1e15a5d35a5fe8d8052439b7-@2a034b09ad418213cad1f5da6c4903187ec852c7c76a0dd29f370269424495c6
                                                       2B
/tmp/disk_cache//ac/ac/acc0ca28dbfd1fbba3851db7fab5db5e423dbc2e1e15a5d35a5fe8d8052439b7:@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855BB
/tmp/disk_cache//ac/ac/acc0ca28dbfd1fbba3851db7fab5db5e423dbc2e1e15a5d35a5fe8d8052439b7:@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
--
/tmp/disk_cache//ac/f3/f3426f6e10e763deaf5de9d168186ba868e1f061cae95b5223a16d3e3dca0081-1bazel-out/host/bin/rules/hmap/_objs/lines/lines.dF
/tmp/disk_cache//ac/f3/f3426f6e10e763deaf5de9d168186ba868e1f061cae95b5223a16d3e3dca0081-@01554c275a4061479192d6fd5869ae3c34a78f6985ae847cdfbba51616970628Ї2B
/tmp/disk_cache//ac/f3/f3426f6e10e763deaf5de9d168186ba868e1f061cae95b5223a16d3e3dca0081:@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855BB
/tmp/disk_cache//ac/f3/f3426f6e10e763deaf5de9d168186ba868e1f061cae95b5223a16d3e3dca0081:@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
--
/tmp/disk_cache//ac/9a/9a433bc1302018f5d768e37d68507f701c3fb21b63c683e2803e6fc9bd529042-8bazel-out/host/bin/rules/hmap/_objs/hmaptool/hmapbuild.dF
/tmp/disk_cache//ac/9a/9a433bc1302018f5d768e37d68507f701c3fb21b63c683e2803e6fc9bd529042-@354fa08a651cb10d077192727c78dee725e9f1467ffaabe0fac3289c2419fd222B
/tmp/disk_cache//ac/9a/9a433bc1302018f5d768e37d68507f701c3fb21b63c683e2803e6fc9bd529042:@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855BB
/tmp/disk_cache//ac/9a/9a433bc1302018f5d768e37d68507f701c3fb21b63c683e2803e6fc9bd529042:@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
--
/tmp/disk_cache//ac/1b/1b769b94ca345c4aea2d785da6850e8bfc35108a0976e66815ca088173c2d2e7-/bazel-out/host/bin/rules/hmap/_objs/hmap/hmap.dF
/tmp/disk_cache//ac/1b/1b769b94ca345c4aea2d785da6850e8bfc35108a0976e66815ca088173c2d2e7-@8d56a4be7607d0a7855f7572aed83cde4bac913af59d6171e34e92b12f1ee3c42B
/tmp/disk_cache//ac/1b/1b769b94ca345c4aea2d785da6850e8bfc35108a0976e66815ca088173c2d2e7:@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855BB
/tmp/disk_cache//ac/1b/1b769b94ca345c4aea2d785da6850e8bfc35108a0976e66815ca088173c2d2e7:@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

It seems like this object is in AC entries and potentially related to runfiles / and or middlemen for thise runfiles

@jerrymarino
Copy link
Contributor Author

jerrymarino commented Apr 13, 2021

PS: It looks like many of the AC entries for a .d file for C++ have this.

For now, if anyone is looking for a mitigation without an extra proxy or file system, when adding a disk_cache infront of the remote cache the client will download it from disk_cache first. I imagine we can work around this with a custom file system or even with a route /cas/$null = { return 200 }

@jin jin added team-Remote-Exec Issues and PRs for the Execution (Remote) team untriaged labels Apr 14, 2021
@coeuvre
Copy link
Member

coeuvre commented Jun 22, 2021

#11063 mentioned that Bazel optimized away uploading zero-sized CAS blob (sha256 = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) to grpc cache but it's not true for http cache. I will work the fix to skip uploading zero-sized blob to other cache backends.

@coeuvre coeuvre added P1 I'll work on this now. (Assignee required) type: feature request and removed untriaged labels Jun 22, 2021
@coeuvre coeuvre self-assigned this Jun 22, 2021
@coeuvre coeuvre mentioned this issue Jul 13, 2021
9 tasks
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
An unintended side-effect of change cc2b3ec is that zero-sized blob won't be uploaded to gRPC cache. However, the behavior is existed for a long time and the REAPI spec is also updated accordingly. This change makes the behavior explicit and brings it to other remote cache backends.

Context bazelbuild#11063.

Fixes bazelbuild#13349.

Closes bazelbuild#13594.

PiperOrigin-RevId: 384457129
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
An unintended side-effect of change cc2b3ec is that zero-sized blob won't be uploaded to gRPC cache. However, the behavior is existed for a long time and the REAPI spec is also updated accordingly. This change makes the behavior explicit and brings it to other remote cache backends.

Context bazelbuild#11063.

Fixes bazelbuild#13349.

Closes bazelbuild#13594.

PiperOrigin-RevId: 384457129
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
An unintended side-effect of change cc2b3ec is that zero-sized blob won't be uploaded to gRPC cache. However, the behavior is existed for a long time and the REAPI spec is also updated accordingly. This change makes the behavior explicit and brings it to other remote cache backends.

Context bazelbuild#11063.

Fixes bazelbuild#13349.

Closes bazelbuild#13594.

PiperOrigin-RevId: 384457129
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
An unintended side-effect of change cc2b3ec is that zero-sized blob won't be uploaded to gRPC cache. However, the behavior is existed for a long time and the REAPI spec is also updated accordingly. This change makes the behavior explicit and brings it to other remote cache backends.

Context bazelbuild#11063.

Fixes bazelbuild#13349.

Closes bazelbuild#13594.

PiperOrigin-RevId: 384457129
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 16, 2021
An unintended side-effect of change cc2b3ec is that zero-sized blob won't be uploaded to gRPC cache. However, the behavior is existed for a long time and the REAPI spec is also updated accordingly. This change makes the behavior explicit and brings it to other remote cache backends.

Context bazelbuild#11063.

Fixes bazelbuild#13349.

Closes bazelbuild#13594.

PiperOrigin-RevId: 384457129
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    An unintended side-effect of change cc2b3ecb9283ebdb297fc3fb6407f4697c850ac2 is that zero-sized blob won't be uploaded to gRPC cache. However, the behavior is existed for a long time and the REAPI spec is also updated accordingly. This change makes the behavior explicit and brings it to other remote cache backends.

    Context bazelbuild/bazel#11063.

    Fixes bazelbuild/bazel#13349.

    Closes #13594.

    PiperOrigin-RevId: 384457129
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: feature request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants