Migrate XLS to bzlmod

[cdleary]

Bazel is moving to a new system for managing external dependencies called bzlmod. To conform to this future we'll need to migrate from using our WORKSPACE file to using a MODULE.bazel file, and presumably registering ourselves in the Bazel Central Registry: https://github.com/bazelbuild/bazel-central-registry

See also the bzlmod migration guide: https://docs.google.com/document/d/1JtXIVnXyFZ4bmbiBCr5gsTH4-opZAFf5DMMb-54kES0/edit

I'm not sure how challenging XLS is to migrate in the grand scheme of things -- I imagine we have at least a few complicating factors:

• "Large" dependencies
  • rules_hdl
  • grpc
  • llvm, z3, ortools etc
  • we also want to start taking a dep on Verible for LSP utilities
• pybind extensions
• pip_install environment
• clang-format pulled down via toolchain
• an optional config that allows use of hermetic toolchain (clang/llvm)

If I'm understanding correctly we probably need to convert to bzlmod from leaves (of all transitive deps) up to the "root" of XLS's workspace. Because converting everything to bzlmod is not going to happen atomically it seems like generally folks would support two concurrent systems in the intermediate state, and the bzlmod would have a "more stripped down" sort of build configuration.

This new system will also have compatibility specifications, see https://docs.google.com/document/d/1ReuBBp4EHnsuvcpfXM6ITDmP2lrOu8DGlePMUKvDnXM/edit#heading=h.4lwo83a0dksg

This came up because of the discussion in #865 Henner pointed out we don't declare/pin our (direct) dependencies. The team had a discussion on the tradeoffs -- ISTM that specifying your leaf dependencies up front doesn't mitigate risk in the current WORKSPACE flow and can make more work to debug/update when you bump your dependencies vs just letting them float. If you assume the code you control can flex more easily than code in dependencies, or your requirements for the dependencies are looser, then you prefer that you do not pin. It may come up that you need to pin sometimes, but ISTM from the discussion that should be done only when necessary because of the work savings advantages of letting the dependencies float.

By contrast, with bzlmod we'd have explicit specifications of compatibility that were properly checked against transitive dependencies, so it seems like The Solution, but the migration work seems like it could be significant given the wrinkles listed above. This issue serves as the reminder we want to figure out the right time to do this / work through the wrinkles noted above.

cc @ted-xie who was helping me think through the tradeoffs / potential roadmap here

[cdleary]

I closed the wrong issue with that commit.

[hzeller]

To prepare for bzlmod (see first proposal #1570, which not to be merged yet) ideally we get more projects that are not yet in Bazel Central Repository into there once we switch (typically by nudging the external project owners); it is not necessary, as it is possible to mix old and new ways, but the more MODULE.bazel the better :)

• For fuzztest, filed Publish on Bazel Central Registry (BCR) ? fuzztest#1350 to be included in https://registry.bazel.build/. Fuzztest is responsible for a lot of additional projects that we have to import.
• For Verible: I'll be talking the voluntary maintainer into adding it on some weekend :)
• zstd: it is already in registry, but we need to use our own bazel file to include the testcorpus (maybe something to upstream for BCR ?)
• There are others that might range from simple (linenoise) to harder (z3, ortools) to get into BCR