-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC for prepublication dependencies #1969
Conversation
I'm guessing based on the CC's that this feature is something Servo is having trouble with? |
@steveklabnik Yes. The RFC spells things out from first principles, but the original context is rust-lang/cargo#2649 |
Thanks for writing up the RFC here @aturon! Overall the feature sounds great to me. Some notes I'd have are:
|
What if you want to both update the |
Keys like in
Are you arguing against the very concept of |
I don't know what you mean by "the very concept". Can you be more precise? I am arguing against adding adding yet another mechanism when the already existing one can just be extended. If you add a new Cargo.toml section, that is a new mechanism for me. If you instead extend I just said that for consistency reasons, the only drawback of putting it into the
Can you tell me of any advantage of making a prepublish category instead of extending the |
I don't understand how allowing regexps in Cf. rust-lang/cargo#2649. |
@nox maybe you have misunderstood what I have been proposing. Its a very hard to describe topic sometimes, I was similarly confused often when reading the RFC. I had read the thread you linked, its included in @aturon 's comment above. The current functionality of a) you may only specify precise versions of crates -- no "^0.1.0" allowed, only "0.1.0". The proposal in the linked thread was originally to get rid of b). My proposal is to keep b) and c) and change a) to: a) you must specify a version range for your crate -- "^0.1.0" and "0.1.0" are both allowed, but omitting version info is not allowed, as well as bare Let me explain it with the example from the RFC. Instead of for the foo crate [prepublish]
xml-rs = { git = "https://github.com/aturon/xml-rs", branch = "0.9.2" }
[dependencies]
xml-rs = "0.9.2" and for the servo crate [prepublish]
xml-rs = { git = "https://github.com/aturon/xml-rs", branch = "0.9.2" }
[dependencies]
foo = { git = "https://github.com/aturon/foo", branch = "fix-xml" } You'd do for the foo crate: [replace]
"xml-rs:0.9" = { git = "https://github.com/aturon/xml-rs", branch = "0.9.2" }
[dependencies]
xml-rs = "0.9.2" and the servo crate [dependencies]
foo = { git = "https://github.com/aturon/foo", branch = "fix-xml" } or if you want to test the new xml-rs in all fitting deps of the servo crate (like in the example): [replace]
"xml-rs:0.9" = { git = "https://github.com/aturon/xml-rs", branch = "0.9.2" }
[dependencies]
foo = { git = "https://github.com/aturon/foo", branch = "fix-xml" } |
Do other package managers in other languages have this problem, or is cargo the only one because of how it resolves duplicate dependencies? If it is a problem in other package managers, how do they deal with it? I'm a little confused around this thought:
Let's say a project is going through the process outlined in the RFC, so that they've submitted a patch to
What if our PR to |
Re: other package managers: I deal with pip (python) extensively and it's possible to hack something kind of like this using the That only really works inside a corporate infrastructure, though. Pip does allow you to specify arbitrary URIs for dependencies, and since it pretty much just takes the first package it encounters which solves a given constraint (not distinguishing between pypi.python.org and any other source) you could theoretically use GitHub projects in the way that prepublish is being suggested. Saying all that makes me think that the fundamental problem being solved by prepublish is one of merging different sources, and that the cargo warning/refusal to publish has to do more with levels of trust ("I trust crates.io more than GitHub") than it does actually prepublishing. For example, I could imagine a corporate environment that doesn't allow publishing if any dependencies are on crates.io, and which would therefore want the same kind of "allow me to test with external crates but give me a warning" behavior. |
This was exactly my first thought on reading this RFC.
IME, it has to do with the multiple resolution aspect; that is, this is because Servo wants to only keep one version of each dependency in the tree, for various reasons. This doesn't affect
Ruby can't do multiple versions in the first place, which means that you end up just having to wait a while. Many ecosystems also don't have nearly as many packages and dependencies; that is, my impression is that |
This is unrelated to |
With Also, why doesn't |
@Ericson2314
I don't think this is actually a problem, or rather it is already a problem with our current tools. If To tie these two points together: unless I am misunderstanding something, it seems we can succinctly define |
A critical difference for me between For example consider a case where serde release 1.0.0. All crates that depend on serde likely now need a major version bump themselves. The work flow for this on Servo would look like:
In this use case as you go down the dependency graph you can immediately start using In contrast to I see these as different enough use cases that a separate section is ok, and otherwise trying to reuse That's a good point! Note that technically I believe builds won't break (because of lock files), but it does mean that if you follow through what the warning says the build may break (for a number of reasons). This aspect seems somewhat fundamental to the feature here though (the final state is unknown given the current state of a repository). I wonder, though, are you basically concluding that we shouldn't issue a warning at all? |
Yes, and that I propose to be fixed! I don't argue that the current Can you read the a,b,c from my comment in #1969 (comment) and the sentence below? Again, I do not propose to keep |
Would you folks mind reading an alternative RFC that outlines my proposed extension of |
So @nox clarified in chat that in the imagined servo workflow, There's nothing stopping anyone from merging but if only one person on a team has to worry about that, and they know about the status of all 16 PRs they're waiting on and whether they've been released or not, then this seems better than not being able to integration test easily while you're waiting 🤷♀️ I feel like the documentation around this should be "if you're going to be waiting a long time for the upstream PRs to be merged and new versions to be cut, and you need to move forward, prefer git dependencies instead". |
Good point. I think we should keep the warning but tweak its wording: not that the |
@yodaldevoid thanks. It seems to me like |
OK, time for some responses!
I think that's not too much of a problem here, given that we'll prevent you from publishing with this tag.
It'd be good to at least have a sketch of the design there, yes.
Yes.
I'm not sure I quite understand the scenario. Could you try outlining it in a step-by-step fashion? Is it fair to interpret the core of your idea as being that we could use the If so, it's an interesting thought, and could be a nice simplification! The word "replace" is not ideal here, but we can deal with that separately.
That kind of thing is always welcome -- it's the whole point of the RFC process :-) I think that your idea has merit, and would love to see it fleshed out some more. It doesn't have to be a whole RFC, though, maybe just a more detailed comment or gist? Whatever you prefer.
Good question. Cargo is definitely somewhat different here because of the duplication strategy and the impact on static type checking. @wycats may be able to comment more.
It's a basic assumption here that, if you're using For mitigation, we can at least make the warning more broadly worded, to point you at these possibilities. We could consider going further, and simply always printing something when you compile with prepublished deps. It's meant to be a short-term state, and that would help others working on the project who may not be aware. I'll update the downsides section with some of these points.
That's certainly an alternative (and I'll update the section accordingly). But the design there is much more complex than you make it out to be, because if we manage this use case through a separate index, then we have a whole set of problems around managing that index. For example, when the crate is published upstream, you'll want to remove it from the index you layered on top. The proposed feature, OTOH, makes it very straightforward to see and manage exactly what is being added.
I'm not sure what you mean -- are you talking about replacing an existing version with a non-existent one, or something else? |
👍 I’ve been confused a couple times when starting work on something new in an existing repo because I forgot I still had |
If possible, I'd prefer for this to be a (obvious) modification to the dependency line itself rather than yet another line at the end telling you you're using prepublished deps. We already have a different "look" for git deps, and I think it makes sense to find a way to call out the use of a prepublished dep in the line where the dep is used. It could also be printed even if the dep is "clean," probably. |
Another potentially viable approach would be to list out groups of dependencies with interesting characteristics at the very end. This could include things like yanked deps, deps using deprecated features, deps with capped lints, and prepublished deps. We could decide which of these groups to include by default (the capped lint one would, I suspect, produce more noise than value if on by default, but could be a nice thing to be allowed to enable with a flag). |
I think I misunderstood until I read your next comment, but here you mean line in the output of running Cargo, not line in a |
@SimonSapin yes, I mean the output of running Cargo. |
@rfcbot concern "bikeshed the name" I'd prefer a less esoteric name from "augment," I suggested some synonyms above, would love to get feedback or other ideas from anyone (of which I most like "patch"). |
"patch" sounds great to me! |
|
@rfcbot resolve "bikeshed the name" |
I'm going into FCP manually, since @withoutboats is traveling and won't be able to formally resolve their concern in the near future. |
s/augment/patch/
Ok looks like 10 days have passed and not much new has come up during FCP, so I'm going to merge. An implementation of this can be found at rust-lang/cargo#4123 |
This is an implementation of [RFC 1969] which adds a new section to top-level manifests: `[patch]`. This section allows you to patch existing sources with new versions of crates, possibly replacing the versions that already exist in the source. More details about this feature can be found in the RFC itself. [RFC 1969]: rust-lang/rfcs#1969
Implement the [patch] section of the manifest This is an implementation of [RFC 1969] which adds a new section to top-level manifests: `[patch]`. This section allows you to augment existing sources with new versions of crates, possibly replacing the versions that already exist in the source. More details about this feature can be found in the RFC itself. [RFC 1969]: rust-lang/rfcs#1969
There should be a better explanation at http://doc.crates.io/manifest.html#the-patch-section how this is different from |
@bennofs the more thorough docs are on this page: http://doc.crates.io/specifying-dependencies.html#overriding-dependencies. |
Changelog: Version 1.21.0 (2017-10-12) ========================== Language -------- - [You can now use static references for literals.][43838] Example: ```rust fn main() { let x: &'static u32 = &0; } ``` - [Relaxed path syntax. Optional `::` before `<` is now allowed in all contexts.][43540] Example: ```rust my_macro!(Vec<i32>::new); // Always worked my_macro!(Vec::<i32>::new); // Now works ``` Compiler -------- - [Upgraded jemalloc to 4.5.0][43911] - [Enabled unwinding panics on Redox][43917] - [Now runs LLVM in parallel during translation phase.][43506] This should reduce peak memory usage. Libraries --------- - [Generate builtin impls for `Clone` for all arrays and tuples that are `T: Clone`][43690] - [`Stdin`, `Stdout`, and `Stderr` now implement `AsRawFd`.][43459] - [`Rc` and `Arc` now implement `From<&[T]> where T: Clone`, `From<str>`, `From<String>`, `From<Box<T>> where T: ?Sized`, and `From<Vec<T>>`.][42565] Stabilized APIs --------------- [`std::mem::discriminant`] Cargo ----- - [You can now call `cargo install` with multiple package names][cargo/4216] - [Cargo commands inside a virtual workspace will now implicitly pass `--all`][cargo/4335] - [Added a `[patch]` section to `Cargo.toml` to handle prepublication dependencies][cargo/4123] [RFC 1969] - [`include` & `exclude` fields in `Cargo.toml` now accept gitignore like patterns][cargo/4270] - [Added the `--all-targets` option][cargo/4400] - [Using required dependencies as a feature is now deprecated and emits a warning][cargo/4364] Misc ---- - [Cargo docs are moving][43916] to [doc.rust-lang.org/cargo](https://doc.rust-lang.org/cargo) - [The rustdoc book is now available][43863] at [doc.rust-lang.org/rustdoc](https://doc.rust-lang.org/rustdoc) - [Added a preview of RLS has been made available through rustup][44204] Install with `rustup component add rls-preview` - [`std::os` documentation for Unix, Linux, and Windows now appears on doc.rust-lang.org][43348] Previously only showed `std::os::unix`. Compatibility Notes ------------------- - [Changes in method matching against higher-ranked types][43880] This may cause breakage in subtyping corner cases. [A more in-depth explanation is available.][info/43880] - [rustc's JSON error output's byte position start at top of file.][42973] Was previously relative to the rustc's internal `CodeMap` struct which required the unstable library `libsyntax` to correctly use. - [`unused_results` lint no longer ignores booleans][43728] [42565]: rust-lang/rust#42565 [42973]: rust-lang/rust#42973 [43348]: rust-lang/rust#43348 [43459]: rust-lang/rust#43459 [43506]: rust-lang/rust#43506 [43540]: rust-lang/rust#43540 [43690]: rust-lang/rust#43690 [43728]: rust-lang/rust#43728 [43838]: rust-lang/rust#43838 [43863]: rust-lang/rust#43863 [43880]: rust-lang/rust#43880 [43911]: rust-lang/rust#43911 [43916]: rust-lang/rust#43916 [43917]: rust-lang/rust#43917 [44204]: rust-lang/rust#44204 [cargo/4123]: rust-lang/cargo#4123 [cargo/4216]: rust-lang/cargo#4216 [cargo/4270]: rust-lang/cargo#4270 [cargo/4335]: rust-lang/cargo#4335 [cargo/4364]: rust-lang/cargo#4364 [cargo/4400]: rust-lang/cargo#4400 [RFC 1969]: rust-lang/rfcs#1969 [info/43880]: rust-lang/rust#44224 (comment) [`std::mem::discriminant`]: https://doc.rust-lang.org/std/mem/fn.discriminant.html
Closed and merged but here's a question, will that allow to patch a crate from crates.io with a crate from another registry, e.g. [patch.crates-io] Gonna try but just throwing it into the round in case I miss the syntax? |
This RFC proposes the concept of prepublication dependencies for Cargo. These dependencies augment a crate index (like crates.io) with new versions of crates that have not yet been published to the index. Dependency resolution then works as if those prepublished versions actually existed in the index. Prepublication dependencies thus act as a kind of "staging index".
Prepublication makes it possible to perform integration testing within a large crate graph before publishing anything to crates.io, and without requiring dependencies to be switched from the crates.io index to git branches. It can, to a degree, simulate an "atomic" change across a large number of crates and repositories, which can then actually be landed in a piecemeal, non-atomic fashion.
Rendered