Verify that src dir wasn't modified by build.rs when publishing #5584

gferon · 2018-05-28T10:23:10Z

Fixes issue #5073.

Implemented during RustFest Paris impl days.

Co-authored-by: Gabriel Feron <[email protected]>

Co-authored-by: boxdot <[email protected]>

rust-highfive · 2018-05-28T10:23:21Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

matklad

Excellent!

matklad · 2018-05-28T11:11:45Z

tests/testsuite/publish.rs

+        .build();
+
+    assert_that(
+        p.cargo("publish")


We can use cargo package here, to avoid the need to setup the publishing infra.

matklad · 2018-05-28T11:40:58Z

tests/testsuite/publish.rs

+            .arg("--index")
+            .arg(publish::registry().to_string()),
+        execs().with_status(101),
+    );


Let's also check that running cargo pacakge --no-verify suppresses this check!

…d while publishing

Co-authored-by: Gabriel Feron <[email protected]>

matklad · 2018-05-28T12:20:30Z

tests/testsuite/package.rs

+    );
+
+    assert_that(
+        p.cargo("package")


This can be more compactly written as p.cargo("package --no-verify"), the testing infra gained ability to split arguments automatically fairly recently.

Co-authored-by: boxdot <[email protected]>

boxdot · 2018-05-28T12:47:30Z

src/doc/src/reference/build-scripts.md

@@ -272,6 +272,11 @@ There’s a couple of points of note here:
  output files should be located. It can use the process’ current working
  directory to find where the input files should be located, but in this case we
  don’t have any input files.
+* In general, build scripts should not modify any files outside of `OUT_DIR`.
+  It may seem fine on the first blush, but it does cause problems when you use


Thanks @matklad for formulating this already in the issue.

matklad · 2018-05-28T14:09:53Z

src/cargo/ops/cargo_package.rs

+        bail!(
+            "Source directory was modified by build.rs during cargo publish. \
+             Build scripts should not modify anything outside of OUT_DIR. \
+             Modified file: {}",


One more thing: let's add use '--no-verify' flag to suppress this check to the error message?

matklad · 2018-05-28T14:16:39Z

@alexcrichton, are we comfortable with enabling this behavior by default?

this might make some existing crates to fail `cargo publish, which is bad (breaking change of sorts), but at the same time is kind of a point of the feature
suppressing the check is easy, one can add --no-verify flag
publish is a rare operation, and in general involves human, so the change wouldn't break someone's CI hopefully.

We can down tone this to a warning perhaps? I personally am fine with a hard error.

alexcrichton · 2018-05-28T16:51:05Z

👍 I'm in favor of enabling this by default

I think the error message should indicate that it can be fixed with the --no-verify flag as @matklad already mentioned but other than that I'm in favor!

matklad · 2018-05-29T09:37:43Z

tests/testsuite/package.rs

+
+Caused by:
+  Source directory was modified by build.rs during cargo publish. \
+Build scripts should not modify anything outside of OUT_DIR. Modified file: [..]src/generated.txt


You need src[/]generated.txt here to pass the test on windows.

matklad · 2018-05-29T09:41:09Z

@bors r+

Thanks @boxdot & @gferon !

bors · 2018-05-29T09:41:10Z

📌 Commit 987ce87 has been approved by matklad

bors · 2018-05-29T09:41:18Z

⌛ Testing commit 987ce87 with merge f35302c...

Verify that src dir wasn't modified by build.rs when publishing Fixes issue #5073. Implemented during RustFest Paris `impl days`.

bors · 2018-05-29T10:17:55Z

☀️ Test successful - status-appveyor, status-travis
Approved by: matklad
Pushing f35302c to master...

iddm · 2018-08-02T17:27:10Z

Can src directory contain immutable code which includes contents of something from build dir written by build.rs? vergen crate, for example.

alexcrichton · 2018-08-02T17:49:27Z

@vityafx of course yeah!

lu-zero · 2018-08-02T19:09:09Z

How does that mix with bindgen? I want to generate the bindings using build.rs and they usually live in src/.

sfackler · 2018-08-02T19:31:06Z

@lu-zero build scripts should place the files they create in the OUT_DIR: https://doc.rust-lang.org/cargo/reference/build-scripts.html#case-study-code-generation

lu-zero · 2018-08-02T19:37:10Z

Looks quite bulky (and in my case quite a bit of churn).

I can live with that, but would be nicer to provide a better mean to hide it.

lu-zero · 2018-08-02T19:44:36Z

Also, forces me to create an empty file with just that line. Since doing

mod foo {
   include!(concat!(env!("OUT_DIR"), "/foo.rs"));
}

Does not work.

lu-zero · 2018-08-02T19:49:07Z

Also having the empty file leads to :

error: an inner attribute is not permitted in this context
 --> /Users/lu_zero/Sources/rust/rav1e/target/debug/build/rav1e-77987e5f4b9641e6/out/aom.rs:3:3

lu-zero · 2018-08-02T19:51:35Z

So basically you made impossible to publish crates that rely on bindgen or there is a way around it?

matklad · 2018-08-02T19:58:55Z

@lu-zero try using setup which is described in bindgen’s docs:

https://rust-lang-nursery.github.io/rust-bindgen/tutorial-3.html
https://rust-lang-nursery.github.io/rust-bindgen/tutorial-4.html

There’s also —no-verify flag which you can use to suppress this check. I would advise against using it, if possible: mutating sources during the build will make lives of consumers of your library harder.

lu-zero · 2018-08-02T20:15:54Z

What is suggested does not seem to work at least with the bindgen 0.37.4.

#![allow(non_upper_case_globals)]
#![allow(non_camel_case_types)]
#![allow(non_snake_case)]

include!(concat!(env!("OUT_DIR"), "/aom.rs"));

The generated file still contains:

/* automatically generated by rust-bindgen */

#![allow(dead_code)]
#![allow(non_camel_case_types)]
#![allow(non_snake_case)]
#![allow(non_upper_case_globals)]

...

The result is :

error: an inner attribute is not permitted in this context
 --> /Users/lu_zero/Sources/rust/rav1e/target/debug/build/rav1e-22d2ecf7978d906c/out/aom.rs:3:3
  |
3 | #![allow(dead_code)]
  |   ^
  |
  = note: inner attributes, like `#![no_std]`, annotate the item enclosing them, and are usually found at the beginning of source files. Outer attributes, like `#[test]`, annotate the item following them.

lu-zero · 2018-08-02T20:40:01Z

Oh, right I forgot to change the build.rs recipe to not append those there.

With Rust 1.28.0: Cargo will now no longer allow you to publish crates with build scripts that modify the src directory. The src directory in a crate should be considered to be immutable. rust-lang/cargo#5584

This is due to rust-lang/cargo#5584 that forbids writing to `src/` and is present since Rust 1.28.

… by `cargo package` (see rust-lang/cargo#5584)

…r caused by `cargo package` (see rust-lang/cargo#5584)" This reverts commit 712c54a.

…h di `OUT_DIR` lalu di-include via `mod.rs`, hal ini dikarenakan oleh: rust-lang/cargo#5584 Tambah beberapa proto message.

boxdot and others added 2 commits May 28, 2018 12:06

Verify that src dir was not modified by build.rs during publish.

ca9aa3d

Co-authored-by: Gabriel Feron <[email protected]>

Add test to check that publish fails when src directory was changed.

8896fdd

Co-authored-by: boxdot <[email protected]>

rust-highfive assigned alexcrichton May 28, 2018

matklad reviewed May 28, 2018

View reviewed changes

gferon and others added 2 commits May 28, 2018 13:59

Add cargo --no-verify test when checking that src dir was not modifie…

ba48ff4

…d while publishing

Document that build scripts should not modify files outside OUT_DIR.

285eb4d

Co-authored-by: Gabriel Feron <[email protected]>

matklad reviewed May 28, 2018

View reviewed changes

gferon and others added 2 commits May 28, 2018 14:42

Provide more context when src files changed while packaging

9c1124f

Co-authored-by: boxdot <[email protected]>

Use auto split argument in test

e31f3c3

gferon force-pushed the master branch from a926a46 to e31f3c3 Compare May 28, 2018 12:43

boxdot reviewed May 28, 2018

View reviewed changes

matklad reviewed May 28, 2018

View reviewed changes

Add more instructions to error message

53480ac

matklad added the relnotes Release-note worthy label May 29, 2018

matklad reviewed May 29, 2018

View reviewed changes

Fix test to work on Windows

987ce87

bors added a commit that referenced this pull request May 29, 2018

Auto merge of #5584 - boxdot:master, r=matklad

f35302c

Verify that src dir wasn't modified by build.rs when publishing Fixes issue #5073. Implemented during RustFest Paris `impl days`.

bors merged commit 987ce87 into rust-lang:master May 29, 2018

matklad mentioned this pull request Jun 2, 2018

Generate files in OUT_DIR lalrpop/lalrpop#353

Merged

dimbleby mentioned this pull request Jul 31, 2018

Relax insistence that cargo project is mounted read-only cross-rs/cross#145

Closed

a1ien mentioned this pull request Aug 7, 2018

Can't use include! macro on generated files stepancheg/rust-protobuf#324

Closed

dwijnand mentioned this pull request Aug 8, 2018

During cargo publish, verify that src directory is not modified by the build.rs #5073

Closed

bjorn-ove mentioned this pull request Aug 18, 2018

Generated files should go into OUT_DIR and not the working directory jmacdonald/amp#94

Merged

This was referenced Aug 22, 2018

Change output location of protobuf generated Rust code hyperledger-archives/sawtooth-core#1825

Merged

Restructure node and change protobuf generation location bitwiseio/sawtooth-pbft#7

Merged

hcpl added a commit to hcpl/mtproto-rs that referenced this pull request Sep 28, 2018

Move generating schema definitions to output dir

d14798f

This is due to rust-lang/cargo#5584 that forbids writing to `src/` and is present since Rust 1.28.

ysimonson added a commit to indradb/indradb that referenced this pull request Oct 28, 2018

Try removing ignore of autogenerated files to circumvent error caused…

712c54a

… by `cargo package` (see rust-lang/cargo#5584)

ysimonson added a commit to indradb/indradb that referenced this pull request Oct 28, 2018

Revert "Try removing ignore of autogenerated files to circumvent erro…

e286f2d

…r caused by `cargo package` (see rust-lang/cargo#5584)" This reverts commit 712c54a.

bjfish mentioned this pull request Apr 13, 2019

runtime-c-api should not copy files to manifest directory wasmerio/wasmer#354

Closed

dalance mentioned this pull request Jun 24, 2019

Output path of process pfnet/rflex#7

Closed

ehuss added this to the 1.28.0 milestone Feb 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Verify that src dir wasn't modified by build.rs when publishing #5584

Verify that src dir wasn't modified by build.rs when publishing #5584

gferon commented May 28, 2018

rust-highfive commented May 28, 2018

matklad left a comment

matklad May 28, 2018

matklad May 28, 2018

matklad May 28, 2018

boxdot May 28, 2018

matklad May 28, 2018

matklad commented May 28, 2018 •

edited

Loading

alexcrichton commented May 28, 2018

matklad May 29, 2018

matklad commented May 29, 2018

bors commented May 29, 2018

bors commented May 29, 2018

bors commented May 29, 2018

iddm commented Aug 2, 2018

alexcrichton commented Aug 2, 2018

lu-zero commented Aug 2, 2018

sfackler commented Aug 2, 2018

lu-zero commented Aug 2, 2018

lu-zero commented Aug 2, 2018

lu-zero commented Aug 2, 2018

lu-zero commented Aug 2, 2018

matklad commented Aug 2, 2018

lu-zero commented Aug 2, 2018 •

edited

Loading

lu-zero commented Aug 2, 2018

Verify that src dir wasn't modified by build.rs when publishing #5584

Verify that src dir wasn't modified by build.rs when publishing #5584

Conversation

gferon commented May 28, 2018

rust-highfive commented May 28, 2018

matklad left a comment

Choose a reason for hiding this comment

matklad May 28, 2018

Choose a reason for hiding this comment

matklad May 28, 2018

Choose a reason for hiding this comment

matklad May 28, 2018

Choose a reason for hiding this comment

boxdot May 28, 2018

Choose a reason for hiding this comment

matklad May 28, 2018

Choose a reason for hiding this comment

matklad commented May 28, 2018 • edited Loading

alexcrichton commented May 28, 2018

matklad May 29, 2018

Choose a reason for hiding this comment

matklad commented May 29, 2018

bors commented May 29, 2018

bors commented May 29, 2018

bors commented May 29, 2018

iddm commented Aug 2, 2018

alexcrichton commented Aug 2, 2018

lu-zero commented Aug 2, 2018

sfackler commented Aug 2, 2018

lu-zero commented Aug 2, 2018

lu-zero commented Aug 2, 2018

lu-zero commented Aug 2, 2018

lu-zero commented Aug 2, 2018

matklad commented Aug 2, 2018

lu-zero commented Aug 2, 2018 • edited Loading

lu-zero commented Aug 2, 2018

matklad commented May 28, 2018 •

edited

Loading

lu-zero commented Aug 2, 2018 •

edited

Loading