BuildStream fails to cache a file w/ mode 0000

[AdrianVovk]

Basically, whenever a file exists w/ a mode of 0000 in a directory that BuildStream will try to cache, the job BuildStream is executing will fail.

I can reproduce this with the following versions. It didn't used to happen with older versions of bst2 (or bst1 for that matter)

• BuildStream 2.1.0
• /usr/local/lib64/python3.11/site-packages/buildstream/subprojects/buildbox/buildbox-casd 1.0.0-0-g824ee5f

Here's a trivial reproducer element:

kind: manual

build-depends:
- buildsystems/base.bst # Basic CLI environment

config:
  install-commands:
  - touch %{install-root}/foo
  - chmod 000 %{install-root}/foo

If you try to build this element, you'll get the following failure (TL;DR: it's a permission denied error trying to access /buildsteream-install/foo in the container)

2023-12-11T18:03:34.906-0500 [1542603:140399133368256] [buildboxcommon_runner.cpp:437] [ERROR] [actionDigest=b763e5057f3167fb391ded391c34753b52287a66021277b4fe025ce3d75947f6/178] Error executing command: std::runtime_error exception thrown at [buildboxcommon_localcasstageddirectory.cpp:124], errMsg = "Error capturing "buildstream/carbonOS/test.bst/../../../buildstream-install" in "/var/home/adrian/.cache/buildstream/cas/staging/cas-tmpdirTeNNPU": System error in `make_nesteddirectory()` for path "buildstream/carbonOS/test.bst/../../../buildstream-install": std::system_error exception thrown at [buildboxcommon_merklize.cpp:69] [system:13], errMsg = "Failed to open path "foo"", errno : Permission denied"
[00:00:00] FAILURE test.bst: Running commands
[00:00:01] FAILURE [6371c55b] test.bst: buildbox-run failed with returncode 1

An alternative way to trigger it looks something like the following. The failure will happen right after all the integration commands finish running for bst build test2.bst or bst shell --build test2.bst

# test1.bst
kind: stack

depends:
- buildsystems/base.bst

public:
  bst:
    integration-commands:
    - touch /foo
    - chmod 000 /foo

# test2.bst
# <...>

build-depends:
- test1.bst
# <...>

# <...>

The way I actually triggered this in my carbonOS build is an element that runs systemd-sysusers as an integration command. This command creates the files /etc/shadow and /etc/gshadow inside of the container, which BuildStream will try to capture. However, /etc/shadow and /etc/gshadow typically have their mode set to 0000. So, when systemd-sysusers creates the file like this, we get the failure. My workaround for now is to set the integration commands like so:

# <...>
      integration-commands:
      - systemd-sysusers
      - chmod 400 %{sysconfdir}/{,g}shadow
# <...>

[fishrockz]

This is really interesting. Thanks for the report!

TLDR; AFAICT this is a bug in buildbox-fuse or buildbox-casd @juergbi do you know which needs the issue raised?

Long story:

A tiny nit is that bst does not use a container as such, its using buildbox and boublewrap, but its not far off a container lol

Another note is that buildbox records the build into the cache which uses the google REAPI format. Which only stores the execution bit by mandate. FileNode

So just fixing the crash will not allow for your use case..

You may note that the cache can store more info in the optional NodeProperties @juergbi explained to me that this optional data will be correctly staged but that buildbox does not record this optional info.

I would like to add this extra info also, i have had a look at buildbox recently but i have not made much progress.

related issues include #1790 #1792 #1413

[AdrianVovk]

AFAICT this is a bug in buildbox-fuse or buildbox-casd

I agree. I suspect buildbox-fuse. However, I'm opening the issue here, in BuildStream, because I have no idea what to report to buildbox. My bug report to them would amount to "when I do this in bst I get this error from buildbox-fuse". My reproducer exists in BuildStream. Hence the bug report here.

A tiny nit is that bst does not use a container as such

Sure it's not a docker container but it's still a container. It has the behavior of a container anyway. And Bubblewrap is a container runtime (just ask Flatpak 🙂)

Which only stores the execution bit by mandate

That's extremely silly

[fishrockz]

That's extremely silly

YES

Its why elements like
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/elements/vm/minimal/qemu-u-boot.bst?ref_type=heads have to have the "systemd config" in the same element as the packaging, "genimage"

It would be much more re-usable and simpler if the "systemd config" could go in one element and then the "genimage" in another.

Now that NodeProperties has been added to the FileNode im hoping we can get over this limitation soon

[nanonyme]

To be fair, NodeProperties sounds seriously dangerous to implement without being opt-in. BuildStream needs a place to be able to break cache keys when it's done. (so cached artifacts with different permissions are not considered equivalent) I don't mean user necessarily needs to be able to set it on or off but BuildStream to buildbox-casd level there needs to be awareness of it.

[juergbi]

Capture is currently handled by buildbox-casd, so it's a limitation there. A complication is that buildbox-casd generally shouldn't modify files that are being captured, however, it likely would have to modify the mode of unreadable files to be able to capture them. We'll have to figure out the best way to handle this. In my opinion, 0000 is a silly file mode (doesn't improve security) but we still have to deal with this.

[juergbi]

To be fair, NodeProperties sounds seriously dangerous to implement without being opt-in.

Capture of NodeProperties is already opt-in (supported for mtime) and I definitely agree, they must remain opt-in.

[fishrockz]

To be fair, NodeProperties sounds seriously dangerous to implement without being opt-in.

It would need to be opt in for both casd and bst..

I did look into the casd part of the task but i didnt get very far.