Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add cargo xtask ls-apis #6561

Merged
merged 18 commits into from
Sep 25, 2024
Merged

add cargo xtask ls-apis #6561

merged 18 commits into from
Sep 25, 2024

Conversation

davepacheco
Copy link
Collaborator

@davepacheco davepacheco commented Sep 12, 2024

The goal of this tool is to programmatically identify potential API dependencies (for Dropshot/Progenitor-based APIs) and use developer-maintained metadata to annotate these dependencies and filter out the bogus ones. While working on this I've found it has a lot of potential for scope creep. I'm creating the PR at this point because I think it's a useful stopping point for now and I'd rather the tooling be in the repo than sitting elsewhere.

The new README has a good summary of what's here.

@davepacheco davepacheco marked this pull request as ready for review September 12, 2024 16:06
Copy link
Contributor

@sunshowers sunshowers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is incredible work, thanks for doing this! I have a few comments but none of them are blocking.

.github/buildomat/build-and-test.sh Outdated Show resolved Hide resolved
dev-tools/ls-apis/Cargo.toml Outdated Show resolved Hide resolved
Comment on lines +32 to +43
via path+file:///home/dap/omicron-merge/out/ls-apis/checkout/maghemite/ddmd#0.1.0
consumed by: mgd (maghemite/mgd)
via path+file:///home/dap/omicron-merge/out/ls-apis/checkout/maghemite/mg-lower#0.1.0
via path+file:///home/dap/omicron-merge/out/ls-apis/checkout/maghemite/mgd#0.1.0
consumed by: omicron-nexus (omicron/nexus)
via path+file:///home/dap/omicron-merge/nexus#[email protected]
consumed by: omicron-sled-agent (omicron/sled-agent)
via path+file:///home/dap/omicron-merge/sled-agent#[email protected]
consumed by: tfportd (dendrite/tfportd)
via path+file:///home/dap/omicron-merge/out/ls-apis/checkout/dendrite/tfportd#0.1.0
consumed by: wicketd (omicron/wicketd)
via path+file:///home/dap/omicron-merge/wicketd#0.1.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on replacing dap with something generic like user?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't mind but I try to avoid munging output that's presented as actual output from a command. It does seem pretty safe here, admittedly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with either :)

dev-tools/ls-apis/api-manifest.toml Show resolved Hide resolved
dev-tools/ls-apis/src/cargo.rs Outdated Show resolved Hide resolved
dev-tools/ls-apis/src/lib.rs Show resolved Hide resolved
dev-tools/ls-apis/src/main.rs Outdated Show resolved Hide resolved
dev-tools/ls-apis/src/workspaces.rs Outdated Show resolved Hide resolved
Comment on lines +148 to +149
// TODO As of this writing, we have two distinct packages called
// "dpd-client":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah this isn't good -- definitely worth fixing, maybe even as a blocker to this PR (which would make this comment moot). Probably easiest to rename the one in Omicron.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not that easy because it may be used by dependencies that aren't in Omicron. I will definitely file an issue for this. I don't consider it a blocker because this is already a problem and not really made worse by having this tool.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, yeah that's unfortunate. Thanks for filing the issue.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #6666

dev-tools/ls-apis/src/workspaces.rs Outdated Show resolved Hide resolved
# Do some test runs of the `ls-apis` command.
banner ls-apis
(
source ./tools/include/force-git-over-https.sh;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, doesn't buildomat already take care of this? @jclulow

Copy link
Collaborator Author

@davepacheco davepacheco Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what's essential here is the export CARGO_NET_GIT_FETCH_WITH_CLI=true. Without this, the CI job fails because it can't access a token that can be used to clone the https URL.

edit: I didn't actually test only setting that variable. But the previous commit on this branch tried removing the source altogether and it failed in that way. Since this is already an https URL, I think that part of the sourced script isn't critical. But the error message for the failure even mentions that if it works from the git CLI, then you should set this variable. I opted to source this script anyway instead of just setting the variable somewhat arbitrarily but it feels like it communicates more context about what's going on.

[[apis]]
client_package_name = "installinator-client"
label = "Wicketd Installinator"
server_package_name = "installinator-api"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth adding a note here saying that "installinator-api" is named after the client, unlike our usual convention of naming APIs after the server.

known_server_packages: &'a BTreeMap<ServerPackageName, &'a ApiMetadata>,

// outputs (structures that we're building up)
errors: Vec<anyhow::Error>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super minor: I generally like passing in error collection like this as a parameter rather than a field, but since the struct is mutable anyway it's a matter of taste. (Essentially, it indicates to callers that this is how errors are handled.)

If it were immutable then passing error collection in as a mutable param is basically required.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's interesting. But I think this would be more unwieldy in this case. This is kind of like a builder with a bunch of different methods that could produce errors. I think it'd be much more annoying to have to pass errors to all of the methods.

dev-tools/ls-apis/src/workspaces.rs Show resolved Hide resolved
@davepacheco
Copy link
Collaborator Author

I think CI failed here on the Helios job because the GitHub token expired during the job. The job took 94 minutes and the token is only good for 60. I will probably fix this by moving the new invocations to a separate job.

@davepacheco davepacheco merged commit 8390a49 into main Sep 25, 2024
18 checks passed
@davepacheco davepacheco deleted the dap/ls-apis branch September 25, 2024 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants