Address questions and comments.

Signed-off-by: Michel Hidalgo <[email protected]>

articles/generalized_interface_generation.md

@@ -25,24 +25,24 @@ This article discusses a generalization of the `rosidl` interface generation pip
 
 ### Overview
 
-For a system architecture-level overview and rationale for the `rosidl` interface generation pipeline, see [here](https://github.com/ros2/ros_core_documentation/blob/e0b7e2bbd026aa466b23cfc5e070144c9114fe29/source/developer_overview.rst#type-specific-interfaces).
+For a system architecture-level overview and rationale for the `rosidl` interface generation pipeline, see [here](https://github.com/ros2/ros_core_documentation/blob/e0b7e2bbd026aa466b23cfc5e070144c9114fe29/source/developer_overview.rst#type-specific-interfaces) (though bear in mind this documentation has not yet been updated to reflect the fact that `rosidl` generators consume `.idl` files nowadays, see [ros2/rosidl#298](https://github.com/ros2/rosidl/pull/298)).
 
 From a structural point-of-view, the `rosidl` interface generation pipeline is comprised of:
 
-- **Generator packages**. These packages provide tooling to generate language-specific, in-memory representations of ROS interfaces. Common runtime (i.e. interface agnostic) functionality is conventionally but not necessarily split into a separate package (e.g. [`rosidl_runtime_c`](https://github.com/ros2/rosidl/tree/master/rosidl_runtime_c), [`rosidl_runtime_cpp`](https://github.com/ros2/rosidl/tree/master/rosidl_runtime_cpp)). It is not unusual for a generator to use another generator's output via language-specific binding machinery (e.g. [`rosidl_generator_py`](https://github.com/ros2/rosidl_python/blob/master/rosidl_generator_py/), which uses Python C bindings to wrap [`rosidl_generator_c`](https://github.com/ros2/rosidl/tree/master/rosidl_generator_c) representations). 
+- **Generator packages**. These packages provide tooling to generate language-specific, in-memory representations of ROS interfaces. Common runtime (i.e. interface agnostic) functionality is conventionally but not necessarily split into a separate package (e.g. [`rosidl_runtime_c`](https://github.com/ros2/rosidl/tree/master/rosidl_runtime_c), [`rosidl_runtime_cpp`](https://github.com/ros2/rosidl/tree/master/rosidl_runtime_cpp)). It is not unusual for a generator to use another generator's output via language-specific binding machinery (e.g. [`rosidl_generator_py`](https://github.com/ros2/rosidl_python/blob/master/rosidl_generator_py/), which uses Python C bindings to wrap [`rosidl_generator_c`](https://github.com/ros2/rosidl/tree/master/rosidl_generator_c) representations), although this dependency is not explicit: generator packages do depend on each other but it is assumed that their generated code will all be built within the same package so that build system dependencies can be established.
 - **Type support packages**. These packages provide tooling to generate language-specific and often middleware-specific support code for middleware implementations to interact (e.g. on publish, on take) with in-memory representations of ROS interfaces. Type support functionality is encapsulated in `rosidl_message_type_support_t`, `rosidl_service_type_support_t`, and `rosidl_action_type_support_t` C structs in read-only global storage, one for each interface type. To support middleware runtime selection, it is not unusual to find language-specific type support packages that do not provide any functionality of their own but defer to middleware-specific type supports accordingly (e.g. [`rosidl_typesupport_c`](https://github.com/ros2/rosidl_typesupport/tree/master/rosidl_typesupport_c), [`rosidl_typesupport_cpp`](https://github.com/ros2/rosidl_typesupport/tree/master/rosidl_typesupport_c)).
 
 Packages for interface definition translation (e.g. [`rosidl_adapter`](https://github.com/ros2/rosidl/tree/master/rosidl_adapter), [`rosidl_generator_dds_idl`](https://github.com/ros2/rosidl_dds/tree/master/rosidl_generator_dds_idl)), interface definition parsing (e.g. [`rosidl_parser`](https://github.com/ros2/rosidl/tree/master/rosidl_parser)), and `ament_cmake` integration (e.g. [`rosidl_cmake`](https://github.com/ros2/rosidl/tree/master/rosidl_cmake/rosidl_cmake)) complement and support generator and type support packages implementation.
 
-Alongside generated source code, generator and type support packages are expected to provide the means to build it by adding to the `rosidl_generate_idl_interfaces`' [`ament` extension point](https://index.ros.org/doc/ros2/Tutorials/Ament-CMake-Documentation/#adding-to-extension-points). By tapping into this extension, downstream `ament_cmake` packages can delegate interface generation to all available packages in the workspace (or any of its underlays).
+Alongside generated source code, generator and type support packages are expected to provide the means to build it by adding to the `rosidl_generate_idl_interfaces`' [`ament` extension point](https://index.ros.org/doc/ros2/Tutorials/Ament-CMake-Documentation/#adding-to-extension-points). By tapping into this extension, downstream `ament_cmake` packages can delegate interface generation to all available packages in the workspace (or any of its underlays). Note that the order in which each generator runs is the order in which each generator is impored into CMake, by virtue of how `ament` extensions work. This order is further obfuscated by the discovery process that the [`rosidl_default_generators`](https://github.com/ros2/rosidl_defaults/tree/master/rosidl_default_generators) package performs. Thus, for all practical purposes this order cannot be relied on, and build system dependencies must be leveraged to establish dependencies across generators' output.
 
 ### Drawbacks
 
 As it stands, the `rosidl` interface generation pipeline:
 
 - **Is strongly coupled with CMake**, and in particular with `ament_cmake` machinery. There's no way for packages using different build systems or build system generators to generate their own interfaces (e.g. pure Python packages cannot generate interfaces nor extend the pipeline). There's no way for external projects using different build systems or build system generators to provide their own generator or type support packages.
 
-- **Favors monolithic builds**. Since interface generation output cannot be controlled at the package level (i.e. all available extensions will be invoked), one and only one package can build and install all generated artifacts. It also implies that the set of artifacts that the package provides depends on the set of generator and type support packages present in the workspace at build-time. This results in package rebuilds to extend or restrict that support (e.g. adding support for a non-core language requires a rebuild of core interface packages), and loose package versioning (e.g. a change in a generator package can induce an API/ABI break in an interface package).
+- **Favors monolithic builds**. Since interface generation output cannot be controlled at the package level (i.e. all available extensions must and will be invoked), one and only one package can build and install all generated artifacts. It also implies that the set of artifacts that the package provides depends on the set of generator and type support packages present in the workspace at build-time. This results in package rebuilds to extend or restrict that support (e.g. adding support for a non-core language requires a rebuild of core interface packages), and loose package versioning (e.g. a change in a generator package can induce an API/ABI break in an interface package).
 
 ## Proposal
 
@@ -65,12 +65,16 @@ As such, it does not fundamentally change the concepts that underpin it nor its
 
 ##### Description
 
-Migrate generation logic to extensible, standardized tools that can compile interface definitions into language-specific type representations and middleware-specific type support code.
+Migrate generation logic to extensible, standardized tools that can compile interface definitions into language-specific type representation and middleware-specific type support code.
+These tools take a set of interface definition files and generate type representation or type support source code.
 Support for new languages and middlewares can be added by external packages via a plugin system.
+Each plugin version is expected to generate the same output for the same input over time.
+Users can (but are not forced to) target specific plugin versions to prevent workspace (or underlay) updates from affecting tool output.
 
 ##### Specification
 
-Interface type representation and type support generation CLIs are roughly equivalent but conceptually different, and thus kept separate. 
+Interface type representation and interface type support are conceptually different pieces of functionality.
+Therefore, their generation CLIs, even if roughly equivalent, are kept separate. 
 This allows them to deviate from each other in the future if need be.
 
 *For interface type representations*
@@ -98,7 +102,8 @@ Additionally, [`setuptools` entrypoints](https://setuptools.readthedocs.io/en/la
 
 ##### Description
 
-Migrate build logic to extensible, standardized tools that can generate [common build specifications](#a-common-build-specification) for language-specific type representations and middleware-specific type support code.
+Migrate build logic to extensible, standardized tools that can generate [common build specifications](#a-common-build-specification) for language-specific type representation and middleware-specific type support code.
+These tools take a set of interface definition files and generate a build tool-agnostic description (aka a [common build specification](#a-common-build-specification)) that outlines how type representation and type support code must be generated (e.g. by invoking a source code generator) and built (e.g. by building a shared library).
 Support for new languages and middlewares can be added by external packages via a plugin system.
 
 ##### Rationale
@@ -108,14 +113,17 @@ It may be one source file or many source files in a hierarchy of directories.
 To build these source files, a command or a script specific to the code generator may have to be executed.
 Thus, in the most general case, build system integration is necessary.
 
-However, integration with any given build system or build system generator couples a large portion of the interface pipeline to that set of tools, which then has to be ported over for projects that use other build systems or build system generators.
-Programs that can bridge across build systems and/or build system generators do exist, but these are rare and often limited to a subset of the functionality that is shared.
+However, integration with any given build-system (generator) A couples a large portion of the interface pipeline to it.
+If a different build-system (generator) B is to be used, users are forced to either:
+* **Delegate from build-system (generator) B to build-system (generator) A**. This is challenging. Build-system (generator) A must be properly embedded in build-system (generator) B, pushing and pulling information across the gap for builds and installs to succeed, adapting invocations of build-system (generator) B to any constraints that build-system (generator) A may impose (e.g. Bazel performs sandboxed builds). There is a performance penalty in doing this.
+* **Port from build-system (generator) A to build-system (generator) B**. This may be (and currently is) a non-trivial effort. Programs that can convert from one build-system (generator) to another do exist, but these are rare and often limited to a subset of the functionality that is shared between source and target.
 
 By generating a build specification in a format designed to be simple yet general, an appropriate build system can be generated on consumption.
 
 ##### Specification
 
-Interface type representation and type support build configuration CLIs are roughly equivalent but conceptually different, and thus kept separate.
+Interface type representation and interface type support are conceptually different pieces of functionality.
+Therefore, their build configuration CLIs, even if roughly equivalent, are kept separate. 
 This allows them to deviate from each other in the future if need be.
 
 *For interface type representations*
@@ -140,41 +148,63 @@ Using Python would ensure tooling interoperability and their supporting librarie
 Additionally, [`setuptools` entrypoints](https://setuptools.readthedocs.io/en/latest/userguide/entry_point.html#advertising-behavior) can be leveraged as plugin system.
 
 Since common build specifications are eminently declarative, most configuration must be resolved prior to generation e.g. for which OS to build.
-This includes dependencies between generated outputs e.g. type support code may depend on type representation code to build, and thus a build specification for the former must ensure the presence of the latter by conditionally generating it if not present.
+This includes dependencies between generated outputs e.g. type support code may depend on type representation code to build.
+Build specification must ensure the presence of all dependencies by conditionally generating them if not present.
+
+There are no requirements nor constraints as to how a build is specified.
+A build may specify some generic components, such as shared libraries, and it may specify an arbitrary command for execution (which could easily be a build tool, such as `cmake`).
+The more generic a build specification is, the better will build-system integration be -- a build-system adapter plugin in `ament_meta_build` cannot workaround a build specification that forces `cmake` to be called.
 
 #### Meta build tool
 
 ##### Description
 
-Develop an extensible, standardized tool that can translate [common build specifications](#a-common-build-specification) into an actual build system or build system generator code.
-Support for new build systems and build system generators can be added by external packages via a plugin system.
+Develop an extensible, standardized tool that can parse [common build specifications](#a-common-build-specification) and generate an actual build system or build system generator code.
+Support for new build systems and build system generators can be added by external packages via a system of __build-system adapter_ plugins.
 
 ##### Specification
 
 ```sh
-ament_meta_build ((-o|--output-file) PATH)? ((-b|--build-system) IDENTIFIER)? PATH?
+ament_meta_build ((-o|--output-file) PATH)? (--build-prefix PATH)? (--install-prefix PATH)? ((-b|--build-system) IDENTIFIER)? PATH?
 ```
 
 Its only positional argument is a path to a common build specification file.
 If no common build specification file is given, it is read from standard input.
 If no output file path is provided, generated build system sources are written to standard output.
-If no build system is specified nor it can be deduced from output file extension, the tool fails.
+If no build-system adapter plugin is specified nor it can be deduced from output file extension, the tool fails.
+If a build prefix is specified, it is used
 If the specific build system is not supported, the tool fails.
 
+##### Implementation considerations
+
+There are no requirements nor constraints as to whether and how any given build-system adapter plugin supports the build specification schema.
+A plugin may generate native build system generator code to build some components, delegate on other plugins (and other build systems) to build some components, and not support some components at all.
+The broader native support is, the lower will build overhead (usually) be and the more useful will an (open source) plugin be to the community.
+
 ### Build system integration
 
-#### `ament_cmake` integration
+It can be achieved by adding thin layers that leverage [build system configuration](#build-system-configuration) and [meta build](#meta-build-tool) tooling to bootstrap an package builds.
 
-It can be achieved by a thin layer that leverages [build system configuration](#build-system-configuration) and [meta build](#meta-build-tool) tooling to bootstrap an `ament_cmake` package i.e. to generate and include CMake code during the configuration stage.
-This layer must bring in package dependencies as there is no build system-agnostic mechanism to do so.
+For instance, an `ament_cmake` integration would generate and include CMake code during the configuration stage:
+
+```cmake
+# Generate C++ type representation build specification
+execute_process(COMMAND rosidl_build_configure --language cpp -I path/to/interface/dependencies/ interface0.idl interface1.idl OUTPUT_FILE build.spec)
+# Translate build specification into CMake code
+execute_process(COMMAND ament_meta_build -b cmake INPUT_FILE build.spec OUTPUT_FILE build.cmake)
+# Use generate CMake code
+include(build.cmake)
+```
+
+It is worth noting that these integration layers must bring in package dependencies on their own, as there is no build system-agnostic mechanism to do so.
 
 ## Appendix
 
 ### A. Common Build Specification
 
-This specification is based on the [Common Package Specification](https://mwoehlke.github.io/cps/) format, a simple and relatively standard format that describes packages for downstream consumption in a build system and build tool agnostic way.
-
-This [patch](TODO(hidmic): draft patch) extends the aforementioned format schema to describe component builds.
+This specification is based on the [Common Package Specification](https://mwoehlke.github.io/cps/) format, a simple and relatively standard format that describes packages for downstream consumption in a build tool-agnostic way.
+This format was not designed to describe builds, and thus why the need to extend its schema.
+This [patch](TODO(hidmic): draft patch) introduces support to describe component builds.
 
 As an example, the following snippet describes a Python package `bar` that depends on a shared library `foo`,
 whose sources are generated by a `foo-code` command.