buildRustPackage is not very nix-ish

[FlorentBecker]

Hi, I started developping some rust code on nixos, and I find that while buildRustPackage is great for having all of crates.io available, the experience is not quite on par with, say, that of developping in haskell or ocaml on nix.

Specifically:

• Documentation is a bit terse (I only found this wiki page: https://nixos.org/wiki/Rust), and it is not clear if it is up to date.
• In Rust projects with both Rust and C dependencies, having the C dependencies known of nix, and the Rust dependencies known to Cargo is not convenient: Cargo.lock cannot be generated from within a nix-shell because of depsSha256, so one needs the awkward nix-shell -p cargo -p rustc -p
• C dependencies of Rust dependencies of a project need to be discovered and added manually to default.nix (dependencies are forgotten when going through Cargo-land)
• When using nix-shell, builds of rust dependencies are not shared between rust projects.

I'm grouping these into one issue because I'm not sure which of these sub-issues can be fixed in buildRustPackage, and what would need something more akin to cabal2nix where rust dependencies of rust packages are made manifest to nix/within nixpkgs.

[wizeman]

I agree with your points.

Regarding your 3rd point, that may not be very easy to accomplish. I was thinking we'd need something that parses Cargo.toml, but even Cargo.toml doesn't seem to tell you what C dependencies you need, see for example: https://github.com/brson/rust-sdl/blob/master/Cargo.toml
This may be even harder to figure out when libraries are dynamically loaded via dlopen. Case in point, the x11-dl library, which loads its dependencies dynamically: https://github.com/Daggerbot/x11-rs/blob/master/x11-dl/Cargo.toml

If I understand correctly your 4th point, I think it may also not be easy to accomplish at this point in time (but I hope I'm wrong). As far as I know, there is no way to tell cargo how to use an already-built library. I think I asked the Cargo developers how to do this, and IIRC they told me this is not supported because Rust doesn't have a stable ABI yet (which is probably a poor excuse in Nix's case).

[cstrahan]

Regarding the 3rd point, perhaps we could have something that crawls a source tree looking for stuff like #[link(name="SDL")], and then we just need a map from the SO name to the Nixpkgs package name. Perhaps a similar thing could be done for dlopen (at least where dlopen is used with a string litteral).

That might pull in some false positives, but then it's just a matter of removing the unnecessary stuff, which is much easier IMO than trying to hunt down the dependencies in the first place.

[cstrahan]

And regarding the 4th point, as @wizeman notes:

If I understand correctly your 4th point, I think it may also not be easy to accomplish at this point in time (but I hope I'm wrong). As far as I know, there is no way to tell cargo how to use an already-built library. I think I asked the Cargo developers how to do this, and IIRC they told me this is not supported because Rust doesn't have a stable ABI yet (which is probably a poor excuse in Nix's case).

That's precisely what I recall from the last time I looked into Rust on Nix.

[Ericson2314]

As far as I know, there is no way to tell cargo how to use an already-built library.

I'd make these two simplish requests of them:

• Allow configuring/building dependencies with a downstream lockfile
• Allow an unstable way to pass arbitrary flags to rustc without cargo complaining.

Then we should be to do the rest ourselves.

[FlorentBecker]

Allow configuring/building dependencies with a downstream lockfile

Don't overrides allow to do that?

[retrry]

I think the whole rust situation is bound to become worse for some time. I was trying to update cargo to 0.10.0 version, but it breaks buildRustPackage function.
-- Cargo needs write access to registry as they implemented file locking (rust-lang/cargo#2486). And it seems that they now try to write .cargo-lock-index file to registry directory. This can be mitigated by copying whole registry to package build directory with write permissions.
-- Cargo fetch now barfs at NixOS hack with file:///dev/null as it seems that they always try to update registry.

    Updating registry `file:///dev/null`
error: failed to fetch `file:///dev/null`

Caused by:
  [6/-1] Could not find repository from '/dev/null'

If somebody could look into this, as I don't have enough knowledge and experience with this kind of stuff :) Or maybe we should wait for: rust-lang/cargo#2361

[Mic92]

Could cargo-local-registry be used here to improve the situation?

[Ericson2314]

Not very much :(. The problem is caching rlibs, not source.

[FlorentBecker]

Could we use multiple output and have each rust package output both its binaries / rlibs and its source. Then each rust package in nix could depend on the "source" output of its cargo dependencies. I think this would solve issues 2 and 3 of my original report, since rust-to-rust dependencies would be tracked by nix.

[ckauhaus]

See this Reddit discussion + blog post for an alternative approach. Looks interesting.

[8573]

And the relevant nixpkgs issue ticket: #24991.

[Ericson2314]

Can we close this because Carnix and buildRustCrate?

[ckauhaus]

I think so. There are still some points which could be improved, but we'd better handle that separate tickets.

[Ericson2314]

For that matter, is there a proposal to replace all buildRustPackage with Carnix in nixpkgs?

[Mic92]

No. Carnix still generates very large nix expressions that are not shared between packages.

[Ericson2314]

Bad for eval times, but could still evaluate to the same derivation?

[Mic92]

This should be the case. A nodePackages like approach would be great. I am not sure how active the carnix development is, there are some features like editions that are not implemented. @andir who forked the project might have a better idea.

[andir]

Well we have a copy with some fixes in the nix-community group. It's technically a fork but just to have a "backup" of what's proposed for carnix and a copy of the released versions. There was some attempt at mirroring cargo into nixpkgs but as @Mic92 said it's generating very large expressions. Those missing features and the mentioned use-cases are definitely things I and others want to have in carnix. We still have to discuss with the original author. It will probably happen around nixcon.. (Excuses the lack of links & mentions, typed on a crippled phone)

[Ericson2314]

it's generating very large expressions.

All the language infrastructures are hitting this. I really want to open the discussion on import-from-derivation to fix the problem once and for all.

[ghost]

@andir where is that fork? The nest doesn't seem to have any way to search, and I can't see it in your github. I want to contribute.

[andir]

@lifranc https://github.com/nix-community/carnix

[illegalprime]

this looks promising: https://www.hadean.com/blog/managing-rust-dependencies-with-nix-part-ii

[ghost]

@illegalprime It doesnt, since it uses cargo in the build.
@andir Carnix is a mess. There might be some code worth saving thou.

[illegalprime]

@lifranc cargo has a lot of stuff in it that would be hard to maintain in a rewrite (features, editions, etc.), as long as nix has fully control over building each crate, what's wrong with building it with cargo?

[ghost]

what's wrong with building it with cargo?

Never used that word. not promising is what I expressed. I'm certain you can see why.

[illegalprime]

@lifranc hmm... what do you think would be a good solution? (maybe that involves a contribution to rustc or cargo, but that's even more hoops to jump through)

[ghost]

@illegalprime What carnix does (try to do maybe); pull all the data needed into nix expressions. The expressions carnix outputs have the wrong structure imo and feel like they stop at an intermediary step (stil lots of functions and boilerplate code in files that should only contain crates' metadata). So either fix carnix or do it over.