Add subcommand to download the crate source

[gsingh93]

It would be nice to be able to download the source for a crates.io crate. Currently, I either google the repository name to find the github page, or find the crate on crates.io and find the repository link on the crate page. Not only is this process long, but it also may not always work as some crates may have generic and hard to search names, may be hosted on less popular hosting platforms, or may not have include the repository link in the manifest.

A subcommand would be useful to quickly download the source without running into these issues.

[withoutboats]

cc me

[JanLikar]

How do you think this subcommand should be named? Should it essentially work like git clone?

[gsingh93]

Maybe cargo source or cargo get-source? I prefer the former but don't feel too strongly about the name.

[alexcrichton]

@JanLikar yeah I've thought in the past that cargo clone may be a good name for something like this, it could even check out the git repo if it's available or just download the source from crates.io (depending on what's selected)

[JanLikar]

I'm willing to do this, but I would need some guidance. @alexcrichton, would you be my mentor?

[alexcrichton]

Certainly! You'll likely want to start this outside of this repo and use the cargo crate on crates.io Once you've got that you'll do something along the lines of:

• Construct a Source. This is done through various concrete implementations which are found in src/cargo/sources
• Update the source
• Download the package
• Call get to get the Package from which you can get the source.

You may want to also explore other cargo subcommands to explore how to set up the CLI.

[JanLikar]

Why shouldn't I just work with a copy of the master branch?

[retep998]

@JanLikar Because it is fairly trivial to cargo install an external version so that as soon as you get it written and working, people can immediately use it without having to wait for it to get merged into cargo and proliferate into the version of Rust they use. Also it acts as a sort of stabilization period.

[alexcrichton]

@JanLikar yeah what @retep998 says is right, and we also unfortunately can't have "unstable cargo subcommands" so anything added would be instantly stable, and we may not be ready to commit to this.

[lifthrasiir]

FYI @JanLikar has since created the cargo-clone for this exact purpose. (I note that here because I was asked about the feature and found that the subcommand is not yet mentioned in this issue.)

[joshtriplett]

cargo-clone is useful; however, it would also be nice to have a cargo-download that downloads the .crate file, with checksum verification (and optional specification of a version other than the latest).

[lilianmoraru]

I will give you one example where this would be useful to have directly inside cargo:
In CI builds you need some Rust tool and you would like to cache the build of that, so you don't waste minutes building the tool every time.
Installing another tool here would defeat the purpose - you still sit through cargo's downloads and compilation into a temporary directory that you cannot cache.

[stale]

As there hasn't been any activity here in over 6 months I've marked this as stale and if no further activity happens for 7 days I will close it.

I'm a bot so this may be in error! If this issue should remain open, could someone (the author, a team member, or any interested party) please comment to that effect?

The team would be especially grateful if such a comment included details such as:

• Is this still relevant?
• If so, what is blocking it?
• Is it known what could be done to help move this forward?

Thank you for contributing!

(The cargo team is currently evaluating the use of Stale bot, and using #6035 as the tracking issue to gather feedback.)

If you're reading this comment from the distant future, fear not if this was closed automatically. If you believe it's still an issue please leave a comment and a team member can reopen this issue. Opening a new issue is also acceptable!

[joshtriplett]

This issue still exists.

The next step would be the implementation of a command (cargo download or cargo source, not sure which name makes more sense) which would download the .crate file for a crate, and optionally unpack it into a directory.

[durka]

cargo-download may cover this.

[withoutboats]

I'm inclined to close this issue. Multiple implementations as third party tools exist, and we'd want an RFC to add a built-in command to cargo. I think there's no further progress that can be made in this repository without an RFC.

[joshtriplett]

That's fair; the next step is probably an RFC, not further implementation work.

[epage]

Another potentially similar command: https://github.com/nakabonne/rhack

[Nemo157]

Yet another third-party implementation of this: cargo-dl. The major difference of this (and why I wrote it to replace my previous cargo download usage) is that it verifies the checksums to detect IO issues like cargo-download has. (It also has a couple user-friendly features like grabbing existing tarballs from cargo's cache if they exist, parallel downloading so you can quickly grab a range of versions cargo dl -e serde@=1.0.{0..9}, and not defaulting to dumping binary gzip data to stdout).

[lolbinarycat]

just want to mention two existing commands that fetch package sources: cargo fetch and cargo vendor.

adding a flag to cargo fetch that would allow fetching an individual package and print the directory of that package would be one possible solution.

[epage]

For people who just want to browse source, you can do so at

• https://docs.rs/
• https://diff.rs/

Could people help me understand the use case for this command? I'm wanting to make sure we focus on the right feature set for this.

[ssokolow]

Could people help me understand the use case for this command? I'm wanting to make sure we focus on the right feature set for this.

For me, I can think of two use-cases off the top of my head:

1. Performing analysis on the exact source cargo build will use without having to extract it back out of rustdoc output.
2. As the set up to patch the Cargo.lock'd version of the crate instead of the git HEAD.

In both cases, the common thread is "I want the version Cargo is going to use and I don't want to have to slurp it back out of HTML".

In case 1, "without running any of its code" is also an important constraint since "analysis" may very well be auditing the crate for trustworthiness. (So I imagine, if the version hasn't been Cargo.lock'd yet, the downloader should also do that so any following cargo build will use the SHA256 that was just audited for.)

[epage]

As the set up to patch the Cargo.lock'd version of the crate instead of the git HEAD.

Would #1545 (getting source from package.repository rather than the .crate) be a better fit for this?

[ssokolow]

No. I can already do that manually if that's what I intended. I'm talking about specifically the situation where I want to patch against the exact code published in that specific release on Crates.io.

(In the most extreme case, possibly even where I want to do a fork-and-fix on a Cargo.lock'd crate where all the versions have been yanked and the repo has been taken down.)

Last I checked, Cargo doesn't even have a mechanism to match the SHA256 from Cargo.lock to a specific commit in the repo to verify that they're the same thing aside from "someone got curious a few months ago and hacked together some tooling to do a study."

[epage]

Last I checked, Cargo doesn't even have a mechanism to match the SHA256 from Cargo.lock to a specific commit in the repo to verify that they're the same thing aside from "someone got curious a few months ago and hacked together some tooling to do a study."

The .crate contains the git sha from when it was published. In 1.80, we will start including that sha when --no-verify is used (with a dirty field). That doesn't help with other VCSs, doesn't guarantee that the SHA was pushed to package.repository, and it doesn't protect against people manually constructing a .crate and calling the publish endpoint directly.

[ssokolow]

I stand corrected.

However, that doesn't change that, were any proposed subcommand to work by downloading from package.repository, I'd probably just ignore it and whip up my own solution in the form of a script (either Python or rust-script) which calls cargo fetch, then parses Cargo.lock, finds the appropriate .crate file under ~/.cargo/registry/cache/, and unpacks it.

[MarcoIeni]

My main use case for this feature is release-plz/release-plz#1331

In my tool, I want to download the source of what's published in crates.io to understand if what was published is the same as the current commit.

Other use cases I can think about:

• auditing: I want to make sure that what's published on crates.io is the same of what's in the git repo
• convenience: people might just want to quickly download a crate and play with it in their editor. I know you can browse it in docs.rs but I'm sure many people would feel more productive in their editor.

Is the cargo team willing to review PRs in case somebody (maybe me 👀) tries to implement it? Or are you still evaluating if it's worth adding this command in cargo?

[epage]

Is the cargo team willing to review PRs in case somebody (maybe me 👀) tries to implement it? Or are you still evaluating if it's worth adding this command in cargo?

No, this is not implementation ready as this is marked needs-design (downgraded from needs-RFC). If it was, it would be marked S-accepted.

Generally the path for a new subcommand is

• Develop it third party and work through feedback
• After it has matured and shown there is a strong need, start discussions with the Cargo team for how well it aligns with Cargo and what further feedback rounds might be needed (e.g. cargo add changed in significant ways after talking with the Cargo team despite having a lot of use)
• Propose the command for merge

cargo info had a fast path to being merged because 99% of the command is UX / for humans and doesn't have compatibility guarantees so we can easily evolve it. In particular for our plumbing commands (100% has compatibility guarentees), I'm a bit concerned over the mixed abstraction layers (cargo metadata is both too low level for people and too high level) and think we need to re-evaluate our approach to them.

In my tool, I want to download the source of what's published in crates.io to understand if what was published is the same as the current commit.

FYI if you want anything other than vcs info, then this is complicated as you need the details in the published Cargo.toml but Cargo changes how it generates the published Cargo.toml between releases.

auditing: I want to make sure that what's published on crates.io is the same of what's in the git repo

Have you seen crates.io's work in this direction?