How to use with a build script

[justinrubek]

My crate is using tonic for gRPC, and part of that is compiling a .proto file in build.rs. It seems that either the script isn't being ran, or maybe the resulting files aren't being included in the main build somehow. I'm not sure how to inspect further. I've mostly tried using buildPackage and cargoBuild (and considered mkCargoDerivation) but haven't had any luck. The only thing I've seen that references a build.rs is mkDummySrc but I'm not sure that applies.

I've made a minimal example repo showcasing the error. I'd like to have minimal changes to the non-nix code (or, at least leave it working for developers that don't use nix)

[dpc]

@justinrubek https://github.com/hyperium/tonic/blob/3e40d819cfbd3d5e4e078b79e3c95a43d14d489e/tonic-build/src/lib.rs#L39 ?

Possibly tonic should fail better when it fails to generate the file. I don't think there's an excuse for failing silently.

[dpc]

This file (example.rs generated by tonic) is not there in ./target.

What I do is I run nix build -L .# and here is some stuff I've been trying:

> git diff HEAD
diff --git a/flake.nix b/flake.nix
index 2e08534..2e1bd62 100644
--- a/flake.nix
+++ b/flake.nix
@@ -35,7 +35,7 @@
         inputs',
         ...
       }: let
-        toolchain = inputs'.fenix.packages.minimal.toolchain;
+        toolchain = inputs'.fenix.packages.stable.toolchain;
 
         craneLib = inputs.crane.lib.${system}.overrideToolchain
           toolchain;
@@ -43,6 +43,22 @@
         packages = {
           default = craneLib.buildPackage {
             src = ./.;
+
+            buildInputs = [ pkgs.protobuf ];
+
+            PROTOC = "${pkgs.protobuf}/bin/protoc";
+            PROTOC_INCLUDE = "${pkgs.protobuf}/include";
+
+            preBuild = ''
+              echo $PROTOC
+              echo $PROTOC_INCLUDE
+              echo "-"
+              ls $PROTOC
+              echo "-"
+              ls $PROTOC_INCLUDE
+              echo '='
+              command  protoc
+            '';
           };
         };

I can't debug this more, but maybe this will be helpful.

[dpc]

Hmmm...

diff --git a/crates/example/build.rs b/crates/example/build.rs
index 4e5ff31..d026989 100644
--- a/crates/example/build.rs
+++ b/crates/example/build.rs
@@ -4,5 +4,8 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
         .build_server(true)
         .emit_rerun_if_changed(true)
         .compile(&["proto/example.proto"], &["proto"])?;
-    Ok(())
+    println!("DFDFDF");
+
+
+    Err("FAIL".into())
 }

For some reason this build.rs seems to never get executed...

[dpc]

@ipetkov I might be wrong, but AFAICT cargo just does not recompiles and/or rerun the build.rs file after it was run the first time in the stubbed version?

[justinrubek]

This file (example.rs generated by tonic) is not there in ./target.

What I do is I run nix build -L .# and here is some stuff I've been trying:

> git diff HEAD
diff --git a/flake.nix b/flake.nix
index 2e08534..2e1bd62 100644
--- a/flake.nix
+++ b/flake.nix
@@ -35,7 +35,7 @@
         inputs',
         ...
       }: let
-        toolchain = inputs'.fenix.packages.minimal.toolchain;
+        toolchain = inputs'.fenix.packages.stable.toolchain;
 
         craneLib = inputs.crane.lib.${system}.overrideToolchain
           toolchain;
@@ -43,6 +43,22 @@
         packages = {
           default = craneLib.buildPackage {
             src = ./.;
+
+            buildInputs = [ pkgs.protobuf ];
+
+            PROTOC = "${pkgs.protobuf}/bin/protoc";
+            PROTOC_INCLUDE = "${pkgs.protobuf}/include";
+
+            preBuild = ''
+              echo $PROTOC
+              echo $PROTOC_INCLUDE
+              echo "-"
+              ls $PROTOC
+              echo "-"
+              ls $PROTOC_INCLUDE
+              echo '='
+              command  protoc
+            '';
           };
         };

I can't debug this more, but maybe this will be helpful.

Oops, yes I forgot to include the PROTOC variables because I didn't run into the issue when making the example. Good call.

I haven't found anything much about build scripts in crane so I'd hoped they'd be ran like normal. I've been trying to see if there's some way to manually call them in the buildPhase but I'm not sure how to proceed

[dpc]

@justinrubek Here is a workaround

> git diff HEAD
diff --git a/flake.nix b/flake.nix
index 2e08534..dd03611 100644
--- a/flake.nix
+++ b/flake.nix
@@ -35,14 +35,19 @@
         inputs',
         ...
       }: let
-        toolchain = inputs'.fenix.packages.minimal.toolchain;
+        toolchain = inputs'.fenix.packages.stable.toolchain;
 
         craneLib = inputs.crane.lib.${system}.overrideToolchain
           toolchain;
+
+
       in rec {
         packages = {
           default = craneLib.buildPackage {
             src = ./.;
+
+            buildInputs = [ pkgs.protobuf ];
+            cargoBuildCommand = "touch crates/example/build.rs && cargo build --profile release";
           };
         };
 

@ipetkov The reason is that cargo's build.rs and source files detection must be timestamp based, and this project has exactly one source file named the same as the stubbed one, and one build script, named exactly like the stubbed one, and in nix store everything gets the same filesystem time. :)

Very interesting corner case. :D

[dpc]

Alternatively, you can rename build.rs to something else, and then use build = somethingelse.rs in Cargo.toml, and this will count as a "needs rebuild" from the PoV of cargo.

@ipetkov That might also be a good fix. Just use a really weird name for the build script, so that people don't accidentally name it the same.

[ipetkov]

Oh wow interesting finding for sure! Hadn't considered this fully, but when should the build script run? Sometimes the build script builds dependencies so you may want a non-stubbed version to run with the dummy dependencies, but then again, maybe you want the build script to run with the real sources

The renamed-stub trick might be a good default to apply, still thinking through the implications 🤔

[ipetkov]

Just for posterity, this will be fixed with #122