Skip to content

Commit

Permalink
fix(publish): Block until it is in index
Browse files Browse the repository at this point in the history
Originally, crates.io would block on publish requests until the publish
was complete, giving `cargo publish` this behavior by extension.  When
crates.io switched to asynchronous publishing, this intermittently broke
people's workflows when publishing multiple crates.  I say interittent
because it usually works until it doesn't and it is unclear why to the
end user because it will be published by the time they check.  In the
end, callers tend to either put in timeouts (and pray), poll the
server's API, or use `crates-index` crate to poll the index.

This isn't sufficient because
- For any new interested party, this is a pit of failure they'll fall
  into
- crates-index has re-implemented index support incorrectly in the past,
  currently doesn't handle auth, doesn't support `git-cli`, etc.
- None of these previous options work if we were to implement
  workspace-publish support (rust-lang#1169)
- The new sparse registry might increase the publish times, making the
  delay easier to hit manually
- The new sparse registry goes through CDNs so checking the server's API
  might not be sufficient
- Once the sparse registry is available, crates-index users will find
  out when the package is ready in git but it might not be ready through
  the sparse registry because of CDNs

So now `cargo` will block until it sees the package in the index.
- This is checking via the index instead of server APIs in case there
  are propagation delays.  This has the side effect of being noisy
  because of all of the "Updating index" messages.
- This is done unconditionally because cargo used to block and that
  didn't seem to be a problem, blocking by default is the less error
  prone case, and there doesn't seem to be enough justification for a
  "don't block" flag.

The timeout was 5min but I dropped it to 1m.  Unfortunately, I don't
have data from `cargo-release` to know what a reasonable timeout is, so
going ahead and dropping to 60s and assuming anything more is an outage.

Fixes rust-lang#9507
  • Loading branch information
epage committed Oct 11, 2022
1 parent 35c5129 commit 3200e19
Show file tree
Hide file tree
Showing 8 changed files with 137 additions and 54 deletions.
1 change: 1 addition & 0 deletions crates/cargo-test-support/src/compare.rs
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ fn substitute_macros(input: &str) -> String {
("[MIGRATING]", " Migrating"),
("[EXECUTABLE]", " Executable"),
("[SKIPPING]", " Skipping"),
("[WAITING]", " Waiting"),
];
let mut result = input.to_owned();
for &(pat, subst) in &macros {
Expand Down
72 changes: 72 additions & 0 deletions src/cargo/ops/registry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,11 @@ use termcolor::Color::Green;
use termcolor::ColorSpec;

use crate::core::dependency::DepKind;
use crate::core::dependency::Dependency;
use crate::core::manifest::ManifestMetadata;
use crate::core::resolver::CliFeatures;
use crate::core::source::Source;
use crate::core::QueryKind;
use crate::core::{Package, SourceId, Workspace};
use crate::ops;
use crate::ops::Packages;
Expand Down Expand Up @@ -183,6 +185,10 @@ pub fn publish(ws: &Workspace<'_>, opts: &PublishOpts<'_>) -> CargoResult<()> {
reg_id,
opts.dry_run,
)?;
if !opts.dry_run {
let timeout = std::time::Duration::from_secs(60);
wait_for_publish(opts.config, reg_id, pkg, timeout)?;
}

Ok(())
}
Expand Down Expand Up @@ -374,6 +380,72 @@ fn transmit(
Ok(())
}

fn wait_for_publish(
config: &Config,
registry_src: SourceId,
pkg: &Package,
timeout: std::time::Duration,
) -> CargoResult<()> {
let version_req = format!("={}", pkg.version());
let mut source = SourceConfigMap::empty(config)?.load(registry_src, &HashSet::new())?;
let source_description = source.describe();
let query = Dependency::parse(pkg.name(), Some(&version_req), registry_src)?;

let now = std::time::Instant::now();
let sleep_time = std::time::Duration::from_secs(1);
let mut logged = false;
loop {
{
let _lock = config.acquire_package_cache_lock()?;
// Force re-fetching the source
//
// As pulling from a git source is expensive, we track when we've done it within the
// process to only do it once, but we are one of the rare cases that needs to do it
// multiple times
config
.updated_sources()
.remove(&source.replaced_source_id());
source.invalidate_cache();
let summaries = loop {
// Exact to avoid returning all for path/git
match source.query_vec(&query, QueryKind::Exact) {
std::task::Poll::Ready(res) => {
break res?;
}
std::task::Poll::Pending => source.block_until_ready()?,
}
};
if !summaries.is_empty() {
break;
}
}

if timeout < now.elapsed() {
config.shell().warn(format!(
"timed out waiting for `{}` to be in {}",
pkg.name(),
source_description
))?;
break;
}

if !logged {
config.shell().status(
"Waiting",
format!(
"on `{}` to propagate to {} (ctrl-c to wait asynchronously)",
pkg.name(),
source_description
),
)?;
logged = true;
}
std::thread::sleep(sleep_time);
}

Ok(())
}

/// Returns the index and token from the config file for the given registry.
///
/// `registry` is typically the registry specified on the command-line. If
Expand Down
1 change: 1 addition & 0 deletions tests/testsuite/artifact_dep.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1920,6 +1920,7 @@ fn publish_artifact_dep() {
[UPDATING] [..]
[PACKAGING] foo v0.1.0 [..]
[UPLOADING] foo v0.1.0 [..]
[UPDATING] [..]
",
)
.run();
Expand Down
2 changes: 2 additions & 0 deletions tests/testsuite/credential_process.rs
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ Only one of these values may be set, remove one or the other to proceed.
[UPDATING] [..]
[PACKAGING] foo v0.1.0 [..]
[UPLOADING] foo v0.1.0 [..]
[UPDATING] [..]
",
)
.run();
Expand Down Expand Up @@ -208,6 +209,7 @@ fn publish() {
[UPDATING] [..]
[PACKAGING] foo v0.1.0 [..]
[UPLOADING] foo v0.1.0 [..]
[UPDATING] [..]
",
)
.run();
Expand Down
2 changes: 2 additions & 0 deletions tests/testsuite/features_namespaced.rs
Original file line number Diff line number Diff line change
Expand Up @@ -903,6 +903,7 @@ fn publish_no_implicit() {
[UPDATING] [..]
[PACKAGING] foo v0.1.0 [..]
[UPLOADING] foo v0.1.0 [..]
[UPDATING] [..]
",
)
.run();
Expand Down Expand Up @@ -1029,6 +1030,7 @@ fn publish() {
[COMPILING] foo v0.1.0 [..]
[FINISHED] [..]
[UPLOADING] foo v0.1.0 [..]
[UPDATING] [..]
",
)
.run();
Expand Down
Loading

0 comments on commit 3200e19

Please sign in to comment.