From 696953b917527ba876e9372ffc2a94c47bb4ce5b Mon Sep 17 00:00:00 2001 From: Ted Pudlik Date: Mon, 16 Oct 2023 18:41:43 +0000 Subject: [PATCH] SEED-0111: Make Bazel Pigweed's Primary Build System Change-Id: I3c2b143044ad793762f1719cd9e77525db5ac664 Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/171695 Reviewed-by: Armando Montanez Presubmit-Verified: CQ Bot Account Commit-Queue: Ted Pudlik --- seed/0000-index.rst | 2 +- seed/0111-build-systems.rst | 267 ++++++++++++++++++++++++++++++++++++ seed/BUILD.gn | 5 + 3 files changed, 273 insertions(+), 1 deletion(-) create mode 100644 seed/0111-build-systems.rst diff --git a/seed/0000-index.rst b/seed/0000-index.rst index c98410e2d6..369c377fb5 100644 --- a/seed/0000-index.rst +++ b/seed/0000-index.rst @@ -20,7 +20,7 @@ All pending, active, and resolved SEEDs are listed below. 0108-pw_emu-emulators-frontend 0109-comms-buffers 0110: Memory Allocation Interfaces - 0111: Build Systems + 0111-build-systems 0112: Async Poll Model 0113: Modular Bazel C/C++ toolchain API 0114: Channels diff --git a/seed/0111-build-systems.rst b/seed/0111-build-systems.rst new file mode 100644 index 0000000000..b0ea7bd3e4 --- /dev/null +++ b/seed/0111-build-systems.rst @@ -0,0 +1,267 @@ +.. _seed-0111: + +=============================================== +0111: Make Bazel Pigweed's Primary Build System +=============================================== +.. seed:: + :number: 111 + :name: Make Bazel Pigweed's Primary Build System + :status: last_call + :proposal_date: 2023-09-26 + :cl: 171695 + +------- +Summary +------- +This SEED proposes that Pigweed transition to using `Bazel +`_ as its primary build system, replacing `GN +`_ in that role. + +Pigweed is and will continue to be a multi-build-system project. As modular +middleware, Pigweed aspires to be easy to integrate with existing embedded +projects whatever their build system. To facilitate this, we provide BUILD +files for multiple systems (Bazel, CMake, GN, Soong), as well as other +distributables where applicable (npm packages, Python wheels, etc). + +But Pigweed is more than just a collection of modules. Pigweed offers a quick, +ergonomic way to start a new embedded project, as well as developer tooling +that lets it scale from a prototype to production deployment. And if you're +starting a new project using Pigweed from day one, you will ask: which build +system *should* I use? This is what we mean by Pigweed's *primary* build +system. + +Pigweed's primary build system has been GN, but soon will be Bazel. + +---------- +Motivation +---------- +GN has been Pigweed's primary build system since inception, and we've developed +an extensive GN build that was used to successfully ship products at scale. GN +is fast and extensible, and its flexible toolchain abstraction is well-suited +to the multi-target builds that arise in most embedded projects. + +But GN has limitations: + +#. **Small community.** GN is a niche build system: the only major open-source + projects that use it are Chromium and Fuchsia. It is not championed by any + major nonprofit or corporation. Few users or open-source contributors come + to Pigweed with any past experience with GN. +#. **No reusable rulesets.** Pigweed has written and maintains all its GN + rules: for C/C++, for Python, for Go (though those are deprecated). With + Rust entering Pigweed, we are now developing GN rules for Rust. There are + no built-in or community-provided rules we could adopt instead. Developing + all rules in-house gives us flexibility, but requires large up-front and + ongoing investments. +#. **No hermetic builds.** GN offers no sandboxing and relies on timestamps to + decide if outputs need to be rebuilt. This has undesirable consequences: + + * The boundary between the environment produced by + :ref:`module-pw_env_setup` and GN is blurred, making GN-built Pigweed as + a whole hostile to systems like Docker or remote execution services. + * Incremental builds can become corrupted. Deleting the output directory + and environment is an undesirable but necessary piece of every Pigweed + developer's toolkit. + * Reliably running only affected tests in CQ is not possible. + +We would like Pigweed to recommend a build system that does not suffer from these +limitations. + +These limitations are not new. What's changed is the build system landscape. +When Pigweed was started years ago, GN was the best choice for a project +emphasizing multi-target builds. But the alternatives have now matured. + +-------- +Proposal +-------- +The proposal is to make Bazel the recommended build system to use with Pigweed, +and the best overall build system for embedded developers. This will involve a +combination of contributions to Pigweed itself, to existing open-source Bazel +rules we wish to reuse, and when necessary to core Bazel. + +The vision is not merely to achieve feature parity with Pigweed's GN offering +while addressing the limitations identified above, but to fully utilize the +capabilities provided by Bazel to produce the best possible developer +experience. For example, Bazel offers native support for external dependency +management and remote build execution. We will make it easy for Pigweed +projects to leverage features like these. + +* **What about GN?** Pigweed's GN support will continue, focusing on + maintenance rather than new build features. No earlier than 2026, if no + Pigweed projects are using GN, we may remove GN support. *The approval of + this SEED does not imply approval of removing GN support.* This decision is + explicitly deferred until a future date. + +* **What about CMake?** Because of its wide adoption in the C++ community, + CMake will be supported indefinitely at the current level. + +------- +Roadmap +------- +This section lists the high-level milestones for Pigweed's Bazel support, and +then dives into the specific work needed to reach them. + +This roadmap is our plan of record as of the time of writing, but like all SEED +content it represents a snapshot in time. We are not as committed to the +specific dates as we are to the general direction. + +There's no specific action that users must take by any date. But our +recommendations about build system choice (embodied in docs and in what we tell +people when they ask us) will change at some point. + +Milestones +========== +* **M0: Good for Most.** We can recommend Bazel as the build system for most + new projects. We may not have full parity with GN yet, but we're close enough + that the benefits of adopting Bazel exceed the costs, even in the short run. + The target date for this milestone is the end of 2023. + + * Out of scope for M0: Windows support. We have to start somewhere, and we're + starting with Linux and MacOS. + +* **M1: Good for All.** We can recommend Bazel for all new Pigweed projects, + including ones that need Windows support. The target date is end of Q1 + 2024. After this date, we don't expect any new projects to use GN. + +* **M2: Best.** We develop compelling features for embedded within the + Bazel ecosystem. This will happen throughout 2024. + +Technical tracks +================ +There are three main technical tracks: + +* **Configurable toolchains** exist for host and embedded, for C++ and Rust. + A separate upcoming SEED will cover this area in detail, but the high-level + goal is to make it straightforward to create families of related toolchains + for embedded targets. This is required for milestone M0, except for Windows + support, which is part of M1. The overall tracking issue is `b/300458513 + `_. + +* **Core build patterns** (facades, multi-platform build, third-party crate + deps for Rust) are established, documented, and usable. + + * M0: + + * Module configuration is supported in Bazel, `b/234872811 + `_. + * Bazel proto codegen is feature-complete, `b/301328390 + `_. + * Multiplatform build is ergonomic thanks to the adoption of + `platform_data + `_ + and `platform-based flags + `_, `b/301334234 + `_. + * Clang sanitizers (asan, msan, tsan) are easy to enable in the Bazel build, `b/301487567 + `_. + + * M1: + + * On-device testing pattern for Bazel projects developed and documented, `b/301332139 + `_. + * Sphinx documentation can be built with Bazel. + * OSS Fuzz integration through Bazel. + +* **Bootstrap** for Bazel projects is excellent. This includes offering + interfaces to Pigweed developer tooling like :ref:`module-pw_console`, + :ref:`module-pw_cli`, etc. + + * M0: GN-free bootstrap for Bazel-based projects is designed and prototyped, `b/274658181 + `_. + + * M1: Pigweed is straightforward to manage as a Bazel dependency, `b/301336229 + `_. + +* **Onboarding** for users new to Pigweed-on-Bazel is easy thanks to + excellent documentation, including examples. + + * M0: + + * There is a Bazel example project for Pigweed, `b/299994234 + `_. + * We have a "build system support matrix" that compares the features + available in the three main build systems (Bazel, CMake, GN), + `b/301481759 `_. + + * M1: + + * The sample project has Bazel support, `b/302150820 + `_. + +------------ +Alternatives +------------ +The main alternatives to investing in Bazel are championing GN or switching to +a different build system. + +Champion GN +=========== +Pigweed does not have the resources to bring GN to parity with modern build +systems like Bazel, Buck2, or Meson. This is an area where we should partner +with another large project rather than build capabilities ourselves. + +CMake +===== +CMake is `the most popular build system for C++ projects +`_, +by a significant margin. We already offer some CMake support in Pigweed. But +it's not a viable candidate for Pigweed's primary build system: + +* **No multi-toolchain builds** Unlike Bazel and GN, CMake does not support + multi-toolchain builds. +* **No Python or Rust support** Again unlike Bazel and GN, CMake is primarily + focused on building C++ code. But Pigweed is a multilingual project, and + Python and Rust need first-class treatment. +* **No hermetic builds** Unlike Bazel, CMake does not support sandboxing. + +Many developers are attracted to CMake by its IDE support. Fortunately, `IDE +support for Bazel is also well-developed `_. + +Other build systems +=================== +There are other multi-lingual, correctness-emphasizing build systems out there, +most prominently `Meson `_ and `Buck2 +`_. We did not consider them realistic targets for +migration at this time. They offer similar features to Bazel, and we have an +existing Bazel build that's in use by some projects, as well as a closer +relationship with the Bazel community. + +-------------- +Open questions +-------------- +Additional SEEDs related to Bazel support are anticipated but have not yet been +written. They will be linked from here once they exist. + +* SEED-????: pw_toolchain for Bazel +* SEED-????: Pigweed CI/CQ for Bazel + +---------------------------- +Appendix: Why Bazel is great +---------------------------- +This SEED has not focused on why Bazel is a great build system. This is because +we are not choosing Bazel over other major build systems, like Meson or Buck2, +for its specific features. We are motivated to recommend a new build system +because of GN's limitations, and we choose Bazel because we have a pre-existing +community of Bazel users, developers with Bazel experience, and a close +relationship with the Bazel core team. + +But actually, Bazel *is* great! Here are some things we like best about it: + +* **Correct incremental builds.** It's great to be able to trust the build + system to just do the right thing, including on a rebuild. +* **External dependency management.** Bazel can manage external dependencies + for you, including lazily downloading them only when needed. By leveraging + this, we expect to speed up Pigweed bootstrap from several minutes to + several seconds. +* **Remote build execution** Bazel has excellent native support for `executing + build actions in a distributed manner on workers in the cloud + `_. Although embedded builds are typically + small, build latency and infra test latency is a recurring concern among + Pigweed users, and leveraging remote builds should allow us to dramatically + improve performance in this area. +* **Python environment management.** The Python rules for Bazel take care of + standing up a Python interpreter with a project-specific virtual + environment, a functionality we had to develop in-house for our GN build. +* **Multilingual support.** Bazel comes with official or widely adopted + third-party rules for C++, Python, Java, Go, Rust, and other langauges. +* **Active community.** The Bazel Slack is always helpful, and GitHub issues + tend to receive swift attention. diff --git a/seed/BUILD.gn b/seed/BUILD.gn index f547787fdf..76a95b901c 100644 --- a/seed/BUILD.gn +++ b/seed/BUILD.gn @@ -28,6 +28,7 @@ pw_doc_group("docs") { ":0107", ":0108", ":0109", + ":0111", ] } @@ -67,3 +68,7 @@ pw_doc_group("0108") { pw_doc_group("0109") { sources = [ "0109-comms-buffers.rst" ] } + +pw_doc_group("0111") { + sources = [ "0111-build-systems.rst" ] +}