Replace #4505 with a different set of workarounds

[chandlerc]

This restores the symlinks for the installation, but teaches the busybox
info search to look for a relative path to the busybox binary itself
before walking through symlinks. This let's it find the tree structure
when directly invoking prefix_root/bin/carbon or similar, either
inside of a Bazel rule or from the command line, and mirrors how we
expect the installed tree to look. This works even when Bazel resolves
the symlink target fully, and potentially to something nonsensical like
a CAS file.

In order to make a convenient Bazel target that can be used with bazel run //toolchain, this adds an override to explicitly set the desired
argv[0] to use when selecting a mode for the busybox and a busybox
binary. Currently, the workaround uses an environment variable because
that required the least amount of plumbing, and seems a useful override
mechanism generally, but I'm open to other approaches.

This should allow a few things to work a bit more nicely:

• It should handle sibling symlinks like clang++ to clang or
  ld.lld to lld, where that symlink in turn points at the busybox.
  We want to use initial argv[0] value to select the mode there.
• It avoids bouncing through Python (or other subprocesses) when
  invoking the carbon binary in Bazel rules, which will be nice for
  building the example code and benchmarking.

It does come at a cost of removing one feature: the initial symlink
can't be some unrelated alias like my_carbon_symlink -- we expect the
first argv[0] name to have the meaningful filename for selecting
a busybox mode.

It also trades the complexity of the Python script for some complexity
in the busybox search in order to look for a relative carbon-busybox
binary. On the whole, I think that tradeoff is worthwhile, but it isn't
free.

[jonmeow]

Maybe I'm missing something, but the use of copy_file to create a symlink looks equivalent to a rollback of #4505. I think bazelbuild/bazel#23620 is correct about workaround options, but let me give my understanding of what's happening.

This change tries to create a symlink chain such as:

./bazel-bin/toolchain/install/run/carbon -> /.../toolchain/install/prefix_root/bin/carbon
/.../toolchain/install/prefix_root/bin/carbon -> /.../toolchain/install/prefix_root/lib/carbon/carbon-busybox
/.../toolchain/install/prefix_root/lib/carbon/carbon-busybox (regular file)

While that's typical output, Bazel could use its CAS (content-addressable storage) cache. In this model, Bazel's promise is to provide something which matches the checksum of the regular file, not the file/symlink structure (this is bazelbuild/bazel#23620). This is also true of carbon-busybox, which is not required to be a regular file: it just needs equivalent content. For example, this is also valid:

./bazel-bin/toolchain/install/run/carbon -> /cas/somechecksum
/.../toolchain/install/prefix_root/bin/carbon -> /cas/somechecksum
/.../toolchain/install/prefix_root/lib/carbon/carbon-busybox -> /cas/somechecksum

Note that things like remote caching can be a lot firmer about removing indirections, even more than the local cache. That's why we need bazelbuild/bazel#23620 to be fixed if we want to use symlinks with Bazel, since it's about retaining symlink structure.

As far as alternative workarounds:

• As described above, I don't expect symlinks to be reliable due to build caching.
• I expect copies for each busyboxed binary to trigger GitHub action runner space constraints. (see other comment)
• Regarding workarounds listed in Preserve symlink structure of remote execution inputs bazelbuild/bazel#23620, declare_symlink is currently not generally available for us.

So that's why I went with something like the 2nd workaround: scripts rather than symlinks.

I believe Google's Bazel support makes much more eager decisions about when to reuse cache content, and how to structure symlinks. If you want to go this route, perhaps it would make sense to verify that it works robustly using your access to Google's infrastructure?

[jonmeow]

Oh, and to be sure, I believe that the main difference prior to #4505 is in changing the name "run_carbon" to "run/carbon". But since it's going back to symlinks, I do think that this is brittle. Maybe I could have been clearer in-person, but I think any solution using symlinks is going to require a split implementation: one for this repository on GitHub, one for Google.

[chandlerc]

Oh, and to be sure, I believe that the main difference prior to #4505 is in changing the name "run_carbon" to "run/carbon".

Yes, it was trying to couple this change with the change to prevent a symlink for the busybox itself.

But thanks for the context on worrying about remote caches forcing even the original busybox to be a symlink into some CAS or other thing that loses all the interesting context. That was what I was missing that #4505 was trying to work around. Sorry if it wasn't clear -- this PR was mostly a question, I wasn't at all confident.

I have another idea that I'm going to play with, but not sure it will work. But now I know how to test that.

[chandlerc]

Ok, after this helpful feedback and some offline discussions, I think a new attempt.

The core of this is to look for a relative carbon-busybox binary prior to walking through symlinks, and an override to use from the Bazel run_tool rule.

Should get us symlink overhead but be resilient to all the perplexities of symlinks (I hope!). PTAL!

[chandlerc]

(Also, just flagging that I updated the PR description with new context.)

[jonmeow]

Understood you intend things to change, I still would appreciate tests demonstrating the behavior that's not intended to work anymore. It would make it clearer that it's an intended break, not an accidental one. (suggestions for specific things below)

[jonmeow]

I'd been minimalist about this, but you could switch MakeDir to std::create_directories so that you don't need the repeat calls.

[jonmeow]

This test is trying to verify that absolute symlinks are correctly handled. That was in specific response to your concern that path concatenation for an absolute symlink would concatenate instead of replacing (also, is_absolute won't validate the difference). IIUC, the behavior here changed and now is a subset of LayerSymlinksInstallTree due to your logic changes; had you considered adjusting the file structure in this test so that it continues to test absolute symlink handling?

[jonmeow]

is_absolute isn't going to test much; a return value of "/bin/carbon" would satisfy this check. Is this test doing enough that you're confident this is the correct absolute path?

[jonmeow]

Would it make sense to add a new test which tests the original symlink order, and that "c" is now the result? i.e., something like OriginalSymlinkNameRetained

[jonmeow]

Yesterday, one of the concerns I mentioned was incorrect install selection. You'd said you're okay with that, but would it be worth adding a test that demonstrates the expected result?

(note, you could share install tree creation via a helper with LayerSymlinksInstallTree)

e.g., choosing "/opt" because that's a thing:

TEST_F(BusyBoxInfoTest, SymlinkAtSameLevelReturnsWrongInstall) {
  // Just enough of the system install that incorrect looks can find the busybox.
  MakeDir(dir_ / "lib");
  MakeDir(dir_ / "lib/carbon");
  MakeFile(dir_ / "lib/carbon/carbon-busybox");

  // What it would look like if someone also did an install in "opt".
  MakeDir(dir_ / "opt");
  MakeDir(dir_ / "opt/lib");
  MakeDir(dir_ / "opt/lib/carbon");
  MakeFile(dir_ / "opt/lib/carbon/carbon-busybox");
  MakeDir(dir_ / "opt/lib/carbon/llvm");
  MakeDir(dir_ / "opt/lib/carbon/llvm/bin");
  MakeSymlink(dir_ / "opt/lib/carbon/llvm/bin/clang", "../../carbon-busybox")
  MakeDir(dir_ / "opt/bin");
  MakeSymlink(dir_ / "opt/bin/carbon", "../lib/carbon/carbon-busybox")

  auto target = MakeSymlink(dir_ / "opt/carbon", "bin/carbon");

  auto info = GetBusyboxInfo(target.string());
  ASSERT_TRUE(info.ok()) << info.error();
  EXPECT_THAT(info->bin_path, Eq(dir_ / "opt/../lib/carbon/carbon-busybox"));
  EXPECT_THAT(info->mode, Eq("clang"));
}