Bzlmod: Support querying/inspecting external dependencies #15365

Wyverald · 2022-04-28T14:32:46Z

We should have a new subcommand in Bazel that allows the user to query/inspect external dependencies.

- `originalDeps` map added to `Module` to keep track of the original dependencies when doing **external dependency graph inspection**, without having to call `SkyFrame` for the `ModuleFileValues` - `ModuleBuilder` added to simplify creating `Module` object for `UnitTests` #15365 PiperOrigin-RevId: 448026884

- `BazelModuleResolutionValue` also stores the `unprunedDepGraph` which will be used for inspection - unit tests added to `SelectionTest` for the un-pruned dep graph #15365 PiperOrigin-RevId: 448028351

- `BazelModuleInspectorValue` - `SkyValue` which stores: - an unpruned dep graph with `ModuleAugment` wrapper nodes which give information about dependants and resolution rules applied - a `modulesIndex` map from *module name* to the corresponding set of `ModuleKeys` - `BazelModuleInspectorFunction` - `SkyFunction` computes the above based on the information inside `BazelModuleResolutionValue` - `BazelModuleInspectorFunctionTest` - UnitTests for the `computeAugmentedGraph` core method of the inspector function #15365 PiperOrigin-RevId: 448029252

andyrinne12 · 2022-05-12T13:33:54Z

I am working on developing an inspection tool for the Bzlmod external module dependency system (Manage external dependencies with Bzlmod | Bazel) (somehow similar to bazel query Bazel Query How-To).

My experience with complex dependency trees is not very vast, therefore I would really appreciate some suggestions on what problems you have encountered and what features may be interesting or useful.

Feel free to suggest improvements on the possible usages as well as on the actual syntax and output of the commands. Here you can find the Design Document.

- ModqueryCommand added for the external dependency inspection `modquery` bazel command - ModqueryOptions added - ModqueryExecutor empty skeleton added to separate query execution logic and print to the injected output stream #15365 PiperOrigin-RevId: 446940802 Change-Id: I18ac702baf739357392248ab05f4270ced3fcf7a

Currently: - `tree` implementation - basic text outputter TODO: - implement `all_paths`, `explain` and `show` - add outputter class which transforms `ResultNode` tree into (text), json and graphviz graph bazelbuild/bazel#15365 PiperOrigin-RevId: 448979280

- `tree`, `all_paths` core implementations (using the same helper graph walkers) - `show` implementation added (mostly uses `Query`'s `TargetOutputter` which is now publicly exposed instead of package-private). - `text`, `json` and `graphviz dot` outputters added - Unit testing for `ModqueryCommand` syntax logic - Unit testing for `ModqueryExecutor` both processing and output logic bazelbuild/bazel#15365 PiperOrigin-RevId: 448979280

- `tree`, `all_paths` core functions implementations (using the same core graph traversal) - Unit testing for `ModqueryCommand` result graph computation logic bazelbuild/bazel#15365 PiperOrigin-RevId: 448979280

- `show` implementation added (mostly uses `Query`'s `TargetOutputter` which is now publicly exposed instead of package-private). - `text`, `json` and `graphviz dot` outputters added - Unit testing for `ModqueryExecutor`output logic bazelbuild/bazel#15365 PiperOrigin-RevId: 451390850

- `tree`, `all_paths` core functions implementations (using the same core graph traversal) - Unit testing for `ModqueryCommand` result graph computation logic bazelbuild/bazel#15365 PiperOrigin-RevId: 448979280

- `ModqueryCommand` added for the external dependency inspection `modquery` bazel command with option parsing logic implemented. - `ModqueryOptions` added. - `ModqueryExecutor` empty skeleton added to separate query execution logic and print to the injected output stream. Dummy test implementations provided for `tree` and `deps` query types. bazelbuild/bazel#15365 PiperOrigin-RevId: 454914018

- `ModqueryCommand` added for the external dependency inspection `modquery` bazel command with option parsing logic implemented. - `ModqueryOptions` added. - `ModqueryExecutor` empty skeleton added to separate query execution logic and print to the injected output stream. Dummy test implementations provided for `tree` and `deps` query types. #15365 PiperOrigin-RevId: 454914018 Change-Id: Ic9cdd71748eafe63fbf67bb74c0a170a12f8647b

Wyverald · 2022-10-04T11:34:52Z

We still need to merge @andyrinne12's remaining changes; they're all sent to me but I haven't had time to review them in depth.

meteorcloudy · 2022-10-04T12:08:55Z

I'm changing this to P2 just in case we don't have enough time to fix all 6.0 release blockers before the cut.

lberki · 2022-10-04T12:31:59Z

SGTM. It would be a pity, but we are quickly running out of room to postpone the 6.0 cut, so I think it's a good decision to put this in the "not a hard requirement" category.

- `tree`, `all_paths` core functions implementations (using the same core graph traversal) - Unit testing for `ModqueryCommand` result graph computation logic bazelbuild#15365 PiperOrigin-RevId: 473344453 Change-Id: I626cf8a83ca42445a2ac81d4bd9c9648e94c2eea

wondfor · 2023-01-03T08:36:48Z

@Wyverald how's progress on this issue? What's still missing from modquery?

bazel modquery still unusable in bazel 6.0.0? but i can get help from " bazel help modquery"

meteorcloudy · 2023-01-04T09:19:43Z

@wondfor We have some changes still under review, we'll work on merging them in Q1 and aim to cherry pick them into Bazel 6.1.

dmkurilov · 2023-02-26T18:06:14Z

We have some changes still under review, we'll work on merging them in Q1

@meteorcloudy, will the changes allow to use the query command with bzlmod? Right now bazel 7.0.0-pre.20230215.2 produces error which doesn't make much sense.

$ bazel query --output=build //:binary
ERROR: error loading package '': Unable to find package for @rules_foreign_cc//foreign_cc:defs.bzl: The repository '@rules_foreign_cc' could not be resolved: Repository '@rules_foreign_cc' is not defined.

dmkurilov · 2023-02-26T18:08:41Z

Is this issue related to #15165? See #15165 (comment)

meteorcloudy · 2023-02-27T10:29:25Z

will the changes allow to use the query command with bzlmod?

bazel query is for querying the build graph, we plan to add bazel mod query to analyze the bazel module dependency graph.

Is this issue related to #15165? See #15165 (comment)

Yes, query the underlying repo rule definition will be supported.

Wyverald · 2023-04-26T17:07:11Z

Current status: targeting 6.3.0 (release early July)

- `show` implementation added (mostly uses `Query`'s `TargetOutputter` which is now publicly exposed instead of package-private). - `text`, `json` and `graphviz dot` outputters added - Unit testing for `ModqueryExecutor`output logic #15365 PiperOrigin-RevId: 542325901 Change-Id: I155a326465355432fbb4436b28aecc0697c3ffab

- Now includes extension usages and repositories inside query graphs along related options - `show` now supports extension-generated repos - Added new subcommand `show_extension` which displays the list of repos generated by that extension and its usages by each module - Since this CL introduces a new argument type to modquery (`<extension>`), refactored modquery argument parsing logic (see `ModuleArg` and `ExtensionArg`). For a user-friendly description, see the `modquery.txt` file. - Added some basic Python integration tests for modquery (more to come). #15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547524086 Change-Id: If1364f01c3be871343edcd5cee94b1180b4b930f

- Since we're considering adding more subcommands that don't exactly "query", such as `bazel mod upgrade` etc. - `bazel modquery show` is renamed to `bazel mod show_repo` (`show_extension` is unchanged) - `bazel modquery tree` is renamed to `bazel mod graph`. bazelbuild/bazel#15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547553222

- Since we're considering adding more subcommands that don't exactly "query", such as `bazel mod upgrade` etc. - `bazel modquery show` is renamed to `bazel mod show_repo` (`show_extension` is unchanged) - `bazel modquery tree` is renamed to `bazel mod graph`. #15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547553222 Change-Id: I74145fdb87c05761692391e6ba47ce8d975f90a9

#15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547554906 Change-Id: I02ab16fe4044888bb13b652e2e75dbfa4f55ead1

- `show` implementation added (mostly uses `Query`'s `TargetOutputter` which is now publicly exposed instead of package-private). - `text`, `json` and `graphviz dot` outputters added - Unit testing for `ModqueryExecutor`output logic #15365 PiperOrigin-RevId: 542325901 Change-Id: I155a326465355432fbb4436b28aecc0697c3ffab

- Now includes extension usages and repositories inside query graphs along related options - `show` now supports extension-generated repos - Added new subcommand `show_extension` which displays the list of repos generated by that extension and its usages by each module - Since this CL introduces a new argument type to modquery (`<extension>`), refactored modquery argument parsing logic (see `ModuleArg` and `ExtensionArg`). For a user-friendly description, see the `modquery.txt` file. - Added some basic Python integration tests for modquery (more to come). #15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547524086 Change-Id: If1364f01c3be871343edcd5cee94b1180b4b930f

#15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547554906 Change-Id: I02ab16fe4044888bb13b652e2e75dbfa4f55ead1

- Since we're considering adding more subcommands that don't exactly "query", such as `bazel mod upgrade` etc. - `bazel modquery show` is renamed to `bazel mod show_repo` (`show_extension` is unchanged) - `bazel modquery tree` is renamed to `bazel mod graph`. #15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547553222 Change-Id: I74145fdb87c05761692391e6ba47ce8d975f90a9

Wyverald · 2023-07-12T18:52:59Z

This is now merged on master, and will be in 6.3.0 (pending cherry-pick PR #18921). See docs (will probably be live soon).

* ModqueryExecutor output logic - `show` implementation added (mostly uses `Query`'s `TargetOutputter` which is now publicly exposed instead of package-private). - `text`, `json` and `graphviz dot` outputters added - Unit testing for `ModqueryExecutor`output logic #15365 PiperOrigin-RevId: 542325901 Change-Id: I155a326465355432fbb4436b28aecc0697c3ffab * Add module extensions to modquery - Now includes extension usages and repositories inside query graphs along related options - `show` now supports extension-generated repos - Added new subcommand `show_extension` which displays the list of repos generated by that extension and its usages by each module - Since this CL introduces a new argument type to modquery (`<extension>`), refactored modquery argument parsing logic (see `ModuleArg` and `ExtensionArg`). For a user-friendly description, see the `modquery.txt` file. - Added some basic Python integration tests for modquery (more to come). #15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547524086 Change-Id: If1364f01c3be871343edcd5cee94b1180b4b930f * New documentation for external deps Rewrote the "external dependencies" and "Bzlmod" pages, organizing them into a subcategory of pages on external dependencies instead. PiperOrigin-RevId: 506646517 Change-Id: Ib3f1d6fb8c33c06e723aeef2eb0a5b7a223cc487 * Document module extension identity Closes #17633 Adds a section to the module extension documentation about the identity of module extensions as described in #17564 (comment). Closes #17634. PiperOrigin-RevId: 513206202 Change-Id: I8cdc8ef836b0a119911b04aa7efb4f9882c72f54 * Minor verbiage changes about MVS + bzlmod This paragraph was a bit misleading, particularly if you are skimming the docs rather than reading top to bottom. Out of context, it sounds like coexistence of multiple major versions of the same module works, when really the next paragraph clarifies that its the opposite. I tweaked things a tad to make it comparison to Go a bit more obvious at the start of the paragraph. I think it might be worth reworking this entire page to be less dependent on Go semantics and simple explain MVS + bzlmod semantics for bzlmod directly (I say this as a bazel user without a ton of go modules experience, so the comparisons aren't useful for all audiences). Closes #17766. PiperOrigin-RevId: 516532030 Change-Id: Ie8a327511c9c26697b2f45501ebf6bbc2b0e2944 * add "archiveType" source.json property to set the http_archive's "typ… …e" value. Closes #17789. PiperOrigin-RevId: 517408668 Change-Id: I7f89db0d2587cde3ff9d77c8657d162981cf32dc * Add a best practices section for extensions PiperOrigin-RevId: 528789449 Change-Id: I1d1e57493fee0e805d953178bd18679fd7e040e4 * Allow overrides in non-root modules They're simply ignored. RELNOTES: Overrides specified by non-root modules no longer cause an error, and are silently ignored instead. They were originally treated as an error to allow for the future possibility of overrides in the transitive dependency graph working together; but we've deemed that infeasible (and even if it was, it'd be so complicated and confusing to users that it would not be a good addition). PiperOrigin-RevId: 529095596 Change-Id: I8b9b7b570b405ee757554accf791d8e4c1ff6528 * Add links to external useful Bzlmod doc and talk PiperOrigin-RevId: 529706168 Change-Id: Iece19039f41bdc80f4acd2621e0d1f5b2ce27ed0 * Add Bzlmod Migration Guide to bazel.build Based on https://docs.google.com/document/d/1JtXIVnXyFZ4bmbiBCr5gsTH4-opZAFf5DMMb-54kES0/edit#heading=h.5mcn15i0e1ch, but with more code snippets as examples and re-organized the guide structure. PiperOrigin-RevId: 543742388 Change-Id: If77ff96b7686f206dd09f5c2453151dee149b087 * `mod` command docs #15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547554906 Change-Id: I02ab16fe4044888bb13b652e2e75dbfa4f55ead1 * Rename `modquery` to `mod` - Since we're considering adding more subcommands that don't exactly "query", such as `bazel mod upgrade` etc. - `bazel modquery show` is renamed to `bazel mod show_repo` (`show_extension` is unchanged) - `bazel modquery tree` is renamed to `bazel mod graph`. #15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547553222 Change-Id: I74145fdb87c05761692391e6ba47ce8d975f90a9 * fix test --------- Co-authored-by: andyrinne12 <[email protected]> Co-authored-by: andyrinne12 <[email protected]> Co-authored-by: Andreas Herrmann <[email protected]> Co-authored-by: Andy Hamon <[email protected]> Co-authored-by: Jon Landis <[email protected]> Co-authored-by: Richard Levasseur <[email protected]> Co-authored-by: Googler <[email protected]>

- Since we're considering adding more subcommands that don't exactly "query", such as `bazel mod upgrade` etc. - `bazel modquery show` is renamed to `bazel mod show_repo` (`show_extension` is unchanged) - `bazel modquery tree` is renamed to `bazel mod graph`. bazelbuild/bazel#15365 Co-authored-by: Xùdōng Yáng <[email protected]> (cherry picked from commit 70e25dd)

Wyverald added type: feature request P1 I'll work on this now. (Assignee required) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. area-Bzlmod Bzlmod-specific PRs, issues, and feature requests labels Apr 28, 2022

Wyverald assigned Wyverald and andyrinne12 Apr 28, 2022

copybara-service bot mentioned this issue May 20, 2022

ModqueryExecutor result graph logic bazelbuild/intellij#3620

Open

copybara-service bot mentioned this issue May 27, 2022

ModqueryExecutor output logic bazelbuild/intellij#3643

Open

dmivankov mentioned this issue Jun 22, 2022

bazel query //external:all is broken, but //external:* still works #15726

Closed

meteorcloudy added P2 We'll consider working on this in future. (Assignee optional) and removed P1 I'll work on this now. (Assignee required) labels Oct 4, 2022

meteorcloudy modified the milestones: 6.0.0 release blockers, 6.0.0 nice to have Oct 5, 2022

meteorcloudy modified the milestones: 6.0.0 nice to have, 6.1.0 release blockers Jan 4, 2023

Wyverald removed this from the 6.1.0 release blockers milestone Feb 16, 2023

copybara-service bot pushed a commit that referenced this issue Jul 12, 2023

mod command docs

1c9a45c

#15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547554906 Change-Id: I02ab16fe4044888bb13b652e2e75dbfa4f55ead1

Wyverald added a commit that referenced this issue Jul 12, 2023

mod command docs

db43aa4

#15365 Co-authored-by: Xùdōng Yáng <[email protected]> PiperOrigin-RevId: 547554906 Change-Id: I02ab16fe4044888bb13b652e2e75dbfa4f55ead1

Wyverald closed this as completed Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bzlmod: Support querying/inspecting external dependencies #15365

Bzlmod: Support querying/inspecting external dependencies #15365

Wyverald commented Apr 28, 2022

andyrinne12 commented May 12, 2022

Wyverald commented Oct 4, 2022

meteorcloudy commented Oct 4, 2022

lberki commented Oct 4, 2022

wondfor commented Jan 3, 2023

meteorcloudy commented Jan 4, 2023 •

edited

Loading

dmkurilov commented Feb 26, 2023

dmkurilov commented Feb 26, 2023

meteorcloudy commented Feb 27, 2023

Wyverald commented Apr 26, 2023

Wyverald commented Jul 12, 2023

Bzlmod: Support querying/inspecting external dependencies #15365

Bzlmod: Support querying/inspecting external dependencies #15365

Comments

Wyverald commented Apr 28, 2022

andyrinne12 commented May 12, 2022

Wyverald commented Oct 4, 2022

meteorcloudy commented Oct 4, 2022

lberki commented Oct 4, 2022

wondfor commented Jan 3, 2023

meteorcloudy commented Jan 4, 2023 • edited Loading

dmkurilov commented Feb 26, 2023

dmkurilov commented Feb 26, 2023

meteorcloudy commented Feb 27, 2023

Wyverald commented Apr 26, 2023

Wyverald commented Jul 12, 2023

meteorcloudy commented Jan 4, 2023 •

edited

Loading