From e7188842b0e4ce8dfc10a580a959aa4f6c7c56a0 Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Fri, 18 Jan 2019 13:45:54 -0800 Subject: [PATCH 01/12] Proposal for dynamically composed nodes --- articles/150_roslaunch.md | 180 ++++++++++++++++++++++++++++++-------- 1 file changed, 143 insertions(+), 37 deletions(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index dc680debf..a61f2f0d2 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -428,62 +428,168 @@ The remapping design document[^static_remapping] goes into detail on how you can However, as an example of a process with multiple nodes, consider a program that instantiates two camera driver nodes called "camera1" and "camera2" by default. You could configure their namespaces separately by doing something like `camera1:__ns:=left camera2:__ns:=right`. -### Dynamically loaded Nodes +#### Dynamically loading Nodes -A special case of processes with multiple nodes are nodes that do not statically instantiate nodes. -Two separate ways to do this are described below, but in both cases the typical command line arguments (as described in the remapping design document[^static_remapping]) and environment variables may not work as expected. -In stead the method for causing the dynamic instantiation of a node needs to communicate these settings separately. -And because of this, the launch system also needs to know how to do this. +Dynamically loading a node means spawning it in a container process that does not know about the node until it is asked to load it. +A container process is a stand alone executable that loads and executes nodes within itself. -#### By Configuration +##### Container Process API -The first way would be to pass some user define command line arguments to the process which the process would interpret and use to dynamically instantiate some ROS nodes. -There's no current plan to have a generic tool for this, but that's subject to change. -However, since there's no standard tool, the launch system cannot know how to convert ROS specific declarations into a format that can be used to invoke it. +While there will be standard container processes, custom container processes would allow using custom executors or client libraries. +Therefore, there must be a container process API for the launch system to communicate which nodes should be loaded. -Though the user could always generate the appropriate command line arguments for the process and have the launch system treat it as a normal operating system process. -In that case, though, it might be useful to indicate to the launch system that a process may contain nodes, but should be otherwise treated like a plain operating system process. -This would let the launch system still monitor it for ROS specific events and input/output. +The launch system must be able tell the container process what arguments to give to a dynamically loaded node. +This includes command line arguments and client library specific options (e.g. `rclcpp` has `use_intra_process_comms`). +Since the launch system cannot know about all custom containers, the API must include a way to pass unknown arguments (e.g. by passing key-value pairs). -As an example of a "by configuration" dynamic node process, you could have a process that takes a variable number of arguments, where each is a serial port address for a laser scanner, e.g. `/dev/ACM0 /dev/ACM1 ...`, and it would instantiate a laser scanner driver node for each address. -It would would up to this process to figure out how to allow the user to further configure the nodes though. -It would not be possible, in this case, for the launch system to "apply a ROS namespace" to the nodes within it, for example. -For this reason, it's probably not an ideal way to structure nodes until a standardized way to load many nodes into a single process via configuration is defined. +The API will not include setting environment variables per loaded node. +Many languages have APIs to get environment variables, and there is no way to isolate them within a process. -
-RFC: +The following options for an API are being considered. -On the point of a standardized way of having a "dynamic nodes process by configuration", I think it might be worth pursuing this, because it will basically become a more efficient way of launch a bunch of nodes into a single process, more efficient than the "by proxy" way described below. +###### API using Command Line Configuration File +One option for a container processes API is to pass a configuration file with nodes to load via the command line. -The "by proxy" way below is essentially the pattern used by nodelets in ROS 1, but what I'm describing here is essentially like a NodeletManager executable which rather than loading nodelets via Service calls would load and run them based on command line arguments or a config file or something. +Advantages +* No waiting for an API to become available -The "by proxy" way is more convenient when you're running things by hand, but if the roslaunch description looks something like this (just pseudo code): +Disadvatages +* Requires write access to the file system +* Requires parsing a config file +* Cannot tell from the outside if a container process supports this interface +* Cannot tell if and when nodes are loaded or unloaded -```xml - - - +This API could have very low latency to launch nodes since it does not require waiting for discovery. +However, there is no way to get feedback about the success or failure of loaded nodes. +There is also no way to tell a container process to unload a composable node. + +###### API using STDIN + +Another option for a container process API is to pass configuration in via STDIN. + +Advantages +* No waiting for an API to become available +* Works with read-only file systems + +Disadvantages +* Requires parsing a config +* Cannot tell from the outside if a container process supports this interface +* Cannot tell if and when nodes are loaded or unloaded +* Cannot stop dynamically loaded nodes from reading STDIN + +This API could also have very low latency to launch nodes. +However, there also is no way to get feedback about the success or failure of loaded nodes. +STDOUT cannot be used because a composable node logging messages to STDOUT is assumed to be very common and would conflict. +Since STDIN is always available, it would be possible to unload a node via this API. + +###### API using ROS Services or Topics + +Lastly, a container process API may be defined by ROS services or topics. + +Advantages +* No config file parsing +* Works with read-only file systems +* Can indicate if a node was successfully loaded +* Can create API to trigger launch events +* Can tell if a container process supports this interface + +Disadvantages +* Must wait for the service API to become available +* Cannot stop dynamically loaded nodes from creating the same services + +This is the only option discussed which can communicate the success or failure of dynamically launched nodes. +It is also the only option that allows introspection. +However, this option has the highest potential delay from when the container process is spawned to when nodes may be loaded. + +##### Proposed Container process API + +This is a proposal for a container process API to be used by the launch system. + +###### Command Line Arguments +The container process must accept command line arguments including log level, remapping, and parameters. +These command line arguments must not be applied to dynamically launched nodes. +The launch system will pass these arguments to the container process in the same way it would pass them to a node. +If a remap rule would apply to a launch service, the launch system should try to use the remapped service name instead. + +###### ROS Services +The container process must offer all of the following services. + +* `_launch/load_node` +* `_launch/unload_node` +* `_launch/list_nodes` + +The services are hidden to avoid colliding with user created services. +`load_node` will be called by the launch system when a composable node is to be dynamically loaded, and `unload_node` destroys a composable node. +`list_nodes` is not called by launch system, and is only provided for introspection. + +####### load_node + +``` +# A ROS package the composable node can be found in +string package_name +# a plugin within that package +string plugin_name + +# Name the composable node should use +string node_name +# Namespace the composable node should use +string namespace +# Values from message rcl_interfaces/Log +uint8 log_level +# Remap rules +# TODO(sloretz) rcl_interfaces message for remap rules? +string[] remap_rules +# Parameters to set +rcl_interfaces/Parameter[] parameters + +# key/value arguments that are specific to a type of container process +rcl_interfaces/Parameter extra_arguments +--- +# True if the node was successfully loaded +bool success +# Name of the loaded composable node (including namespace) +string full_node_name +``` + +####### unload_node - +``` +# Full node name including namespace +string full_node_name +--- +# True if the node existed and was unloaded +bool success +``` + +####### list_nodes + +``` +--- +# List of full node names including namespace +string[] full_node_names ``` -Whether or not the above results in three processes (one container and two proxy processes) or a single process, probably doesn't matter to the user or at least would be fine with the user in exchange for less overhead. +###### Exit Code +If the container proces is asked to shutdown due to normal [Termination], then the exit code must be 0. +If it exits due to an error then exit code must be any other number. -It would be more efficient because you don't need to have a node in each proxy to actually make the Service call to the load that node the user wants to run, and you don't need to maintain a bond between the container and the proxy processes. +##### Parallel vs Sequential Loading of Node -So I'm interested in what others think about coming up with a standardized executable which can load and configure many nodes at once from command line arguments or a config file. -
+If it is possible to load multiple nodes in parallel, then it needs to be decided how to load the nodes. +The container process should load nodes as soon as it is asked. +It should be up to the launch system to decide whether to load nodes in parallel or sequentially. +If multiple nodes of the same type are to be launched, then the launch system should load the nodes sequentially so each is able to remap it's name before the next is loaded. +If they are of different types then the launch system may choose to try to load them in parallel, where the exact order they get loaded is determined by chance or the container process. -#### By Proxy +##### Registration of Composable Nodes -The other way to implement a process which dynamically instantiates ROS nodes, is to send the request to instantiate a node asynchronously from another process. -So the "container" process would start up and provide a service (ROS Service or otherwise) that lets external processes request a node be instantiated with a given set of configurations. +How Composable nodes are registered is not defined by this document. +Instead, the container process is responsible for knowing how to find nodes it is asked to load. +For example, a container process might use pluginlib for rclcpp nodes, or python entry points for rclpy nodes. -One form this can take is by providing a "proxy" process which doesn't actually run the node, but instead loads it into an already running container and just stays running until the node is shutdown in the remote container. -This pattern is used by the "nodelets" in ROS 1. -It's a useful pattern when you're running nodes by hand but you want them to share a process and don't want to write your own program to do that. +##### Duplicate Node Handling -There are a few other forms this can take, but the common thread between them is that a process instantiates nodes dynamically based on asynchronous input from external actors (proxy's), and that the configuration for those nodes is communicated through something other than command line arguments and environment variables. +If a container process is asked to load a node with a full node name matching an existing node, then it must reject the request. ## Event Subsystem From c9a4d0386cbe8b7cae6e2efb38b48ca8a1825634 Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Fri, 18 Jan 2019 14:30:14 -0800 Subject: [PATCH 02/12] allow multiple extra_arguments --- articles/150_roslaunch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index a61f2f0d2..a03c593c8 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -543,7 +543,7 @@ string[] remap_rules rcl_interfaces/Parameter[] parameters # key/value arguments that are specific to a type of container process -rcl_interfaces/Parameter extra_arguments +rcl_interfaces/Parameter[] extra_arguments --- # True if the node was successfully loaded bool success From 77461c1069036d727be7a3484314c50f3a6f09c7 Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Fri, 18 Jan 2019 14:32:47 -0800 Subject: [PATCH 03/12] Allow node_name and namespace to be empty --- articles/150_roslaunch.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index a03c593c8..884ee15e6 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -530,9 +530,9 @@ string package_name # a plugin within that package string plugin_name -# Name the composable node should use +# Name the composable node should use, or empty to use the node's default name string node_name -# Namespace the composable node should use +# Namespace the composable node should use, or empty to use the node's default namespace string namespace # Values from message rcl_interfaces/Log uint8 log_level From 5bd6a5c8f2da5dd5cf1aabe3a677b48d750ded73 Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Fri, 18 Jan 2019 14:35:33 -0800 Subject: [PATCH 04/12] Human readable error message --- articles/150_roslaunch.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index 884ee15e6..de6b21831 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -547,6 +547,8 @@ rcl_interfaces/Parameter[] extra_arguments --- # True if the node was successfully loaded bool success +# Human readable error message if success is false, else empty string +string error_messsage # Name of the loaded composable node (including namespace) string full_node_name ``` @@ -559,6 +561,8 @@ string full_node_name --- # True if the node existed and was unloaded bool success +# Human readable error message if success is false, else empty string +string error_messsage ``` ####### list_nodes From e0008cf721014d53f56c05862677182f8add4288 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Fri, 18 Jan 2019 14:39:17 -0800 Subject: [PATCH 05/12] Update articles/150_roslaunch.md Co-Authored-By: sloretz --- articles/150_roslaunch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index de6b21831..3da7ec108 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -577,7 +577,7 @@ string[] full_node_names If the container proces is asked to shutdown due to normal [Termination], then the exit code must be 0. If it exits due to an error then exit code must be any other number. -##### Parallel vs Sequential Loading of Node +##### Parallel vs Sequential Loading of Nodes If it is possible to load multiple nodes in parallel, then it needs to be decided how to load the nodes. The container process should load nodes as soon as it is asked. From 90480aeeabb904d2693b3f6e0c4e3ae72567ff88 Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Fri, 18 Jan 2019 18:25:34 -0800 Subject: [PATCH 06/12] Assign nodes unique ids, but still forbid duplicates --- articles/150_roslaunch.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index 3da7ec108..056561cd5 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -524,6 +524,13 @@ The services are hidden to avoid colliding with user created services. ####### load_node +If a container process is asked to load a node with a full node name matching an existing node, then it must reject the request. +This is to avoid conflicts in features that assume node name uniqueness, like parameters. + +A container process must assign the node a unique id when it is loaded. +The id of a loaded node instance never changes. +Two nodes in the same container process must never have the same id, and there should be a significant time delay before an id is reused. + ``` # A ROS package the composable node can be found in string package_name @@ -551,13 +558,15 @@ bool success string error_messsage # Name of the loaded composable node (including namespace) string full_node_name +# A unique identifier for the loaded node +uint64 unique_id ``` ####### unload_node ``` -# Full node name including namespace -string full_node_name +# Container specific unique id of a loaded node +uint64 unique_id --- # True if the node existed and was unloaded bool success @@ -571,6 +580,8 @@ string error_messsage --- # List of full node names including namespace string[] full_node_names +# corresponding unique ids (must have same length as full_node_names) +uint64[] unique_ids ``` ###### Exit Code @@ -591,10 +602,6 @@ How Composable nodes are registered is not defined by this document. Instead, the container process is responsible for knowing how to find nodes it is asked to load. For example, a container process might use pluginlib for rclcpp nodes, or python entry points for rclpy nodes. -##### Duplicate Node Handling - -If a container process is asked to load a node with a full node name matching an existing node, then it must reject the request. - ## Event Subsystem This section of the article covers the event subsystem within the launch system, which is responsible for generating events and reporting them to users and itself so that those events can be handled. From 339259174b648d06f702674d5a987093a8cd6159 Mon Sep 17 00:00:00 2001 From: Michel Hidalgo Date: Tue, 22 Jan 2019 10:27:58 -0800 Subject: [PATCH 07/12] Update articles/150_roslaunch.md Co-Authored-By: sloretz --- articles/150_roslaunch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index 056561cd5..f81948133 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -453,7 +453,7 @@ One option for a container processes API is to pass a configuration file with no Advantages * No waiting for an API to become available -Disadvatages +Disadvantages * Requires write access to the file system * Requires parsing a config file * Cannot tell from the outside if a container process supports this interface From 656132161b9701ef9c184a1451964fd71a77e0c2 Mon Sep 17 00:00:00 2001 From: Michel Hidalgo Date: Tue, 22 Jan 2019 10:28:06 -0800 Subject: [PATCH 08/12] Update articles/150_roslaunch.md Co-Authored-By: sloretz --- articles/150_roslaunch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index f81948133..d4f722888 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -585,7 +585,7 @@ uint64[] unique_ids ``` ###### Exit Code -If the container proces is asked to shutdown due to normal [Termination], then the exit code must be 0. +If the container process is asked to shutdown due to normal [Termination], then the exit code must be 0. If it exits due to an error then exit code must be any other number. ##### Parallel vs Sequential Loading of Nodes From e21211be0d6484759d036c2069f9434a801dabfb Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Tue, 22 Jan 2019 16:21:10 -0800 Subject: [PATCH 09/12] Section to list --- articles/150_roslaunch.md | 122 +++++++++++++++++++------------------- 1 file changed, 61 insertions(+), 61 deletions(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index d4f722888..bdc18e749 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -522,67 +522,67 @@ The services are hidden to avoid colliding with user created services. `load_node` will be called by the launch system when a composable node is to be dynamically loaded, and `unload_node` destroys a composable node. `list_nodes` is not called by launch system, and is only provided for introspection. -####### load_node - -If a container process is asked to load a node with a full node name matching an existing node, then it must reject the request. -This is to avoid conflicts in features that assume node name uniqueness, like parameters. - -A container process must assign the node a unique id when it is loaded. -The id of a loaded node instance never changes. -Two nodes in the same container process must never have the same id, and there should be a significant time delay before an id is reused. - -``` -# A ROS package the composable node can be found in -string package_name -# a plugin within that package -string plugin_name - -# Name the composable node should use, or empty to use the node's default name -string node_name -# Namespace the composable node should use, or empty to use the node's default namespace -string namespace -# Values from message rcl_interfaces/Log -uint8 log_level -# Remap rules -# TODO(sloretz) rcl_interfaces message for remap rules? -string[] remap_rules -# Parameters to set -rcl_interfaces/Parameter[] parameters - -# key/value arguments that are specific to a type of container process -rcl_interfaces/Parameter[] extra_arguments ---- -# True if the node was successfully loaded -bool success -# Human readable error message if success is false, else empty string -string error_messsage -# Name of the loaded composable node (including namespace) -string full_node_name -# A unique identifier for the loaded node -uint64 unique_id -``` - -####### unload_node - -``` -# Container specific unique id of a loaded node -uint64 unique_id ---- -# True if the node existed and was unloaded -bool success -# Human readable error message if success is false, else empty string -string error_messsage -``` - -####### list_nodes - -``` ---- -# List of full node names including namespace -string[] full_node_names -# corresponding unique ids (must have same length as full_node_names) -uint64[] unique_ids -``` +1. load_node + + If a container process is asked to load a node with a full node name matching an existing node, then it must reject the request. + This is to avoid conflicts in features that assume node name uniqueness, like parameters. + + A container process must assign the node a unique id when it is loaded. + The id of a loaded node instance never changes. + Two nodes in the same container process must never have the same id, and there should be a significant time delay before an id is reused. + + ``` + # A ROS package the composable node can be found in + string package_name + # a plugin within that package + string plugin_name + + # Name the composable node should use, or empty to use the node's default name + string node_name + # Namespace the composable node should use, or empty to use the node's default namespace + string namespace + # Values from message rcl_interfaces/Log + uint8 log_level + # Remap rules + # TODO(sloretz) rcl_interfaces message for remap rules? + string[] remap_rules + # Parameters to set + rcl_interfaces/Parameter[] parameters + + # key/value arguments that are specific to a type of container process + rcl_interfaces/Parameter[] extra_arguments + --- + # True if the node was successfully loaded + bool success + # Human readable error message if success is false, else empty string + string error_messsage + # Name of the loaded composable node (including namespace) + string full_node_name + # A unique identifier for the loaded node + uint64 unique_id + ``` + +2. unload_node + + ``` + # Container specific unique id of a loaded node + uint64 unique_id + --- + # True if the node existed and was unloaded + bool success + # Human readable error message if success is false, else empty string + string error_messsage + ``` + +3. list_nodes + + ``` + --- + # List of full node names including namespace + string[] full_node_names + # corresponding unique ids (must have same length as full_node_names) + uint64[] unique_ids + ``` ###### Exit Code If the container process is asked to shutdown due to normal [Termination], then the exit code must be 0. From aa18ecaffc9e7685a22ab9e172c5fbcdbc59d8a0 Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Tue, 22 Jan 2019 17:58:02 -0800 Subject: [PATCH 10/12] More generic wording about container processes --- articles/150_roslaunch.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index bdc18e749..6de2612ab 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -503,16 +503,16 @@ However, this option has the highest potential delay from when the container pro ##### Proposed Container process API -This is a proposal for a container process API to be used by the launch system. +This is a proposal for an API a launch system will use to interact with container processes. ###### Command Line Arguments -The container process must accept command line arguments including log level, remapping, and parameters. +A container process must accept command line arguments including log level, remapping, and parameters. These command line arguments must not be applied to dynamically launched nodes. -The launch system will pass these arguments to the container process in the same way it would pass them to a node. +The launch system will pass these arguments to a container process in the same way it would pass them to a node. If a remap rule would apply to a launch service, the launch system should try to use the remapped service name instead. ###### ROS Services -The container process must offer all of the following services. +A container process must offer all of the following services. * `_launch/load_node` * `_launch/unload_node` From 6c1d79d9abda03b4c5014cd26fe5b638063840f9 Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Wed, 23 Jan 2019 16:19:19 -0800 Subject: [PATCH 11/12] namespace -> node_namespace --- articles/150_roslaunch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index 6de2612ab..3dded6381 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -540,7 +540,7 @@ The services are hidden to avoid colliding with user created services. # Name the composable node should use, or empty to use the node's default name string node_name # Namespace the composable node should use, or empty to use the node's default namespace - string namespace + string node_namespace # Values from message rcl_interfaces/Log uint8 log_level # Remap rules From d98bb98b67d3bd5df9af1bb0ec5f2a741e20b80c Mon Sep 17 00:00:00 2001 From: Shane Loretz Date: Wed, 30 Jan 2019 08:59:54 -0800 Subject: [PATCH 12/12] _launch/ -> ~/_container/ Signed-off-by: Shane Loretz --- articles/150_roslaunch.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/articles/150_roslaunch.md b/articles/150_roslaunch.md index 3dded6381..f8aac86c3 100644 --- a/articles/150_roslaunch.md +++ b/articles/150_roslaunch.md @@ -514,9 +514,9 @@ If a remap rule would apply to a launch service, the launch system should try to ###### ROS Services A container process must offer all of the following services. -* `_launch/load_node` -* `_launch/unload_node` -* `_launch/list_nodes` +* `~/_container/load_node` +* `~/_container/unload_node` +* `~/_container/list_nodes` The services are hidden to avoid colliding with user created services. `load_node` will be called by the launch system when a composable node is to be dynamically loaded, and `unload_node` destroys a composable node.