Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

publishing: add a CI task to test packaging/building of all published crates individually #11985

Closed
nagisa opened this issue Aug 21, 2024 · 5 comments

Comments

@nagisa
Copy link
Collaborator

nagisa commented Aug 21, 2024

Our current CI tests a combination of all crates in the workspace, which can lead to issues related to crate features especially which are an additive combination. In particular a different crate may be enabling a feature crucial for a crate, meaning the faulty crate would not compile in isolation.

Something like what's in #11983

@eluad

This comment was marked as spam.

@mooori
Copy link
Contributor

mooori commented Sep 6, 2024

I'm running into the following problem: We want to dry-run publish crates near-*. Assume this includes:

  • near-bar which has no dependency on other packages from near-*
  • near-foo which depends on on near-bar

cargo ws publish ... publishes crates in an order that allows satisfying dependencies. Though the point of dry-run is to not actually publish the crates to a registry. Accordingly cargo publish --dry-run -p near-bar succeeds but cargo publish --dry-run -p near-foo fails because near-bar at the new version to be dry-run published cannot be found in the registry.

What if near-bar is a path dependency? My understanding of cargo docs is that this doesn't matter for packaging, which will ignore the path:

Cargo will ignore the path key for dependencies in published packages.

There seems to be no option to enforce using the path dependency. Intuitively this makes sense, as all dependencies of a published crate should be published too.

I can think of the following workarounds:

  1. Add --dry-run to NEAR's cargo-workspaces fork and try to implement a workaround there. I doubt this would be practicable as under the hood cargo commands are invoked.
  2. Try to use a dummy registry to which we can publish packages during PR CI checks. Not sure of how big of an effort this would be.
  3. Publish dummy versions to crates.io that are yanked at the end of the CI run. To me that seems like an abuse of community resources and should therefore be ruled out.

Perhaps I'm missing another workaround that is more promising?

@nagisa
Copy link
Collaborator Author

nagisa commented Sep 6, 2024

Dummy registry would probably be ideal approach here as it would emulate the actual workload we're looking to test here the best. There are some projects that provide such a thing, https://github.com/hirevo/alexandrie/ seems most straightforward from a brief look?

List of all sorts of tools are here: https://github.com/rust-lang/cargo/wiki/Third-party-registries.

@mooori
Copy link
Contributor

mooori commented Sep 6, 2024

Thanks for the pointers. I'll give the dummy registry approach a try.

@mooori
Copy link
Contributor

mooori commented Sep 18, 2024

Some further issues popped up after setting up a dummy registry and publishing crates to them:

  • Using cargo ws publish --registry ... can publish a crate to the dummy registry, but then it times out because it waits for the crate to become available on crates.io. Even though it was made aware of the alternate registry with --registry. Further investigations are needed to find out if this is an issue with cargo-workspaces upstream or the fork used in nearcore.
    • Another possible root cause could be that the third party registry that I used (kellnr) is sending an incorrect response after the crate was uploaded to it, triggering the unexpected behavior of cargo-workspaces
  • Using cargo publish would be possible only after figuring out the correct order in which the near-* crates need to be published since they can depend on each other. cargo workspaces publish can do that, but it has the issue described above.

Making all of this work in CI might require going down some rabbit holes. I'm unassigning myself as I should move on to another project instead.

For anyone who might tackle this in the future, I can recommend kellnr for setting up the dummy registry. It seemed to be the option that's easiest to setup in CI while still providing the required functionality. Heads up that while margo is even easier to set up, it does not (yet) provide the functionality required to cargo publish crates to it. Most software to set up alternate registries seems under active development, so in the future other options might become better suited.

@mooori mooori removed their assignment Sep 18, 2024
@akhi3030 akhi3030 closed this as completed Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants