Support cross-compile install

[infinity0]

Amazingly this Just Works, tested cross-compiling aho-corasick from amd64 to armhf on Debian.

[rust-highfive]

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

[matklad]

So looks like we have somewhat conflicting requests:

• Support non-host target for cargo install #3810 asks to add --target to cargo install among other thigs, but in some very specific circumstances.

• cargo install uses my project's .cargo/config target triple and it shouldn't? #5441 asks cargo install to ignore target settings from .carg/config, to make cargo install custom-command from a cross-compiled project work.

If we support --target argument, we probably support .cargo/config as well.

@infinity0 could you explain your use-case a little bit? cargo install installs binaries into the host's bin directory, so binaries installed by cargo install are always run by host and it seems to me that cross-compilation for install, in general, does not make sense. There are specific cases where it makes sense though, like #5441, but I am curious about yours!

[infinity0]

@matklad I'm giving --root to do a staged install, the end results are being distributed onto other computers of the target architecture.

[matklad]

@infinity0 do you distribute only the binaries, or .crates.toml file as well?

If you are interested only in binaries, the currently-unstable cargo build --target $target --release --outupt-dir $root/bin option might a better fit here: cargo install's intended use-case is installing binaries on the local machine.

[infinity0]

I'm currently not distributing the .crates.toml but in Debian a staged install for end user machines is supposed to have no behavioral differences to a local install, whether that's done on a machine for the same or different host. If I use cargo install for a native build then a cross build needs to run pretty much the same process.

I'm not sure what the intended differences between cargo install and --output-dir are, can you clarify?

[matklad]

I'm not sure what the intended differences between cargo install and --output-dir are, can you clarify?

Sure! cargo install --root foo would basically do two things:

• put the binary into ./foo/bin/
• write a cargo-specific metadata file, .crates.toml, to ./foo/.crates.toml

cargo build --output-dir foo compiles the requested Cargo target (--bin or --lib) and puts it to the foo directory.

I.e., install writes some custom cargo-specific metadata, and --output-dir is just a usual build with additional "I'd love to fetch final artifacts from here".

Neither is a perfect fit for creating a binary distribution of the application. install's intended use-case is to install binaries for a current user in a Cargo-specific location (that's why it writes cargo-specific metadata into root/.crates.toml). --output-dir's intended use-case is to handle over compiled binaries to some other automation, which will package them or integrate into other build system or just about anything else.

[infinity0]

When you say "cargo-specific" do you mean "specific to how cargo has its own registered set of installed standalone binaries" as opposed to "specific to cargo as a dependency manager"? If I understood that correctly then it does sound like --output-dir is more suitable for us than install. But I also worry about the future if cargo extends support for standardised locations to install e.g. manpages, that --output-dir would become a second-class citizen to install. How likely is it that it will be stabilised and maintained as good as install is?

[matklad]

When you say "cargo-specific" do you mean "specific to how cargo has its own registered set of installed standalone binaries"

This one. By default, cargo install installs into Cargo-specific location, ~/.cargo/bin, and pays no attention to the conventions of the OS. It is rather different from make install, which tries to put stuff into proper directories like /usr/bin.

But I also worry about the future

Yep, the future is very hazy here! I am not on top of the things in this area, but looks like https://github.com/rust-lang-nursery/cli-wg/issues/8 is the current discussion.

I think that the future directions would look like this:

• --output-dir would certainly become stable and would be well-supported. It is useful in various circumstances, not only packaging, and is easy to support because it actually does very little.

• cargo install might grow some notion of "post install scripts" to install manpages and such, but this will be geared towards installing into ./cargo/bin enclave, and not towards following OS conventions (because there are too many OSes to support them all in Cargo).

• An external tool (stager?) might be written for packaging rust applicaitons. This tool would probably have some integration with cargo, so that running something like cargo dist --deb --rpm --msi would automatically use the tool to produce various packages with an app.

So, for know, I think the best way is to use --output-dir (we probably could stabilize it right now, seems low-risk), or even to just cargo build and copy the binary from ./target. This would be stable. In contrast, using cargo install might break one day, if we add support for man-pages or something. When the proper solution for packaging emerges (which likelly would be different from both bulid/install), switch to that.

[matklad]

Fascinating! Regardless of what I think about the purposes of cargo install, people do use it for packaging: https://github.com/snapcore/snapcraft/blob/4cbe8c1abea0d72b9defa2db7f79264b073c5491/snapcraft/plugins/rust.py#L100-L113.

[infinity0]

Traditionally packagers at least on UNIX/FHS-based systems rely on make install to put stuff in the right place with DESTDIR to do a staged install, and most other buildsystems like cmake have some sort of equivalent. cargo install "looks like" it's doing this but it's not, but it looks very close to fool most people including me :).

I can understand the need to distinguish these though so am happy to use --output-dir when that gets released as an unstable flag.

[alexcrichton]

I think I'd personally be inclined to take this route rather than #5606 after reading up on this. I do think that for the disto use case cargo install is the way to go as it's somewhat future-proof with what we'd like to do with install scripts.

[alexcrichton]

Note that if we do indeed decide to merge this we'd probably want to revert #5606 which I've basically accidentally merged at this point

[infinity0]

FWIW there are actually not that many different possible installation layouts, and everything I've seen can be done using the first few variables prescribed by GNU Make. Only the first few are essential, stuff further below is optional.

Cargo could define its own installation layout for ~/.cargo based on the above variables. Then, to support system-wide installation, cargo install could either support overriding individual variables via the CLI, or just support a new flag --system to install into a "system layout" defined by the distro (e.g. Debian). Under this override, it could install .crates.toml into $(datadir)/cargo (going by the GNU standard I just linked), which on Debian and most other FHS systems would be /usr/share/cargo. If it needs to, it can read .crates.toml from both this system directory and ~/.cargo.

What I just described is a pretty standard structure across package managers.

It is also quite common to use ~/.local as a user-specific prefix (instead of /usr/local) which is defined by systemd but also in use by various other distros. That would be an alternative to ~/.cargo if you don't want to continually make up new directory layouts under it.

[infinity0]

Under this override, it could install .crates.toml into $(datadir)/cargo [..]

Correction, this would not be suitable - since the file gets rewritten every time something new is installed, it would have to maintain a file in $(localstatedir)/cargo (/var/lib/cargo usually) and update it in a post-install hook. Alternatively, don't use a single file .crates.toml but generate a new static file for each binary, and keep these files in a directory under $(datadir)/cargo (/usr/share/cargo usually).

[alexcrichton]

@infinity0 can you include a revert of #5606 in this PR now as well since it'll be required if we merge? Additionally can you add a test that the --target flag works?

[matklad]

To me, both #5606 and this PR seem valid (separately, of course :) ). I don't know the right solution here, but I agree that we should revert #5606 at least because it will break current packagers relying on this behavior, and that we should add --target and explicitly document that you can use install to install to a different machine.

The reasoning behind this my opinion is really fuzzy though, and relies on interaction with yet to be implemented features. I think cargo install can be used for three purposes:

• install rust-specific dev tools, like cargo install wasm-bindgen-cli. For this use-case we currently want cargo install will ignore the target triple specified in a project directory #5606. However, hopefully custom workflows would eliminate the need for this use-case, and instead you would add something like [workflow] wasm-bindgen = "0.1" and make this project local.

• creating canonical distributable artifacts. Here, I think we miss the ability to cargo install static and dynamic libraries. I can envision someone writing a dynamic library with C API in rust with the intention of distributing it just like any over libfoo.so. For this use-case, we'll need some command to produce this canonical artifact, and, preferably, this should be the same command that is used for binaries (and similar desire for install scripts applies to, for example, distribute the header file for the library).

• a low friction way to distribute generally useful CLI tools (think ripgrep) to Rust users specifically, and here CLI group are doing something: https://github.com/rust-lang-nursery/cli-wg/issues/8

All in all, I fear that this in general perhaps is "needs design", but I don't want to block anything because I don't know a single person who is on top of all the things above.

[alexcrichton]

Excellent points @matklad! With those use cases in mind though I don't think it's clear that --target and [build.target] should be rejected though (in my mind at least). In that sense I'd personally be ok landing something like this

[matklad]

In that sense I'd personally be ok landing something like this

Yeah, my current understanding is that we should merge it (+ a revert) as well, I am just not confident that my current understanding is sufficiently informed :-)

[infinity0]

OK I've updated the PR, reverted #5606 as requested and added a test for the new --target behaviour, including using a foreign arch when cross-compile tests are enabled.

[alexcrichton]

@bors: r+

Ok thanks @infinity0!

[bors]

📌 Commit 0774e97 has been approved by alexcrichton

[bors]

⌛ Testing commit 0774e97 with merge 7139192...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 7139192 to master...