From ac459840a79e13b5651db4502d1d60424d5e4aa8 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Mon, 27 Mar 2023 14:25:54 +0200 Subject: [PATCH 001/107] Checkin work in progress --- build_instructions/devcontainers.md | 101 ++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 build_instructions/devcontainers.md diff --git a/build_instructions/devcontainers.md b/build_instructions/devcontainers.md new file mode 100644 index 000000000..66659addb --- /dev/null +++ b/build_instructions/devcontainers.md @@ -0,0 +1,101 @@ +# Dev Containers + +Alternatively you can use dev containers to build the project if you prefer a streamlined setup experience. This means you can use the same tools and dependencies as the rest of the team, including our Continuous Integration (CI) workflows, without worrying about installing dependencies on your host machine. Additionally, using Dev Containers makes it simple to switch between local or remote development environments, such as GitHub Codespaces. More info on Dev Containers can be found here: + +- [Development Containers](https://containers.dev/) + - An open specification for enriching containers with development specific content and settings +- [Developing inside a Container](https://code.visualstudio.com/docs/remote/containers) + - Learn how to use Visual Studio Code to develop inside a Docker container +- [GitHub Codespaces overview](https://docs.github.com/en/codespaces/overview) + - A development environment hosted in the cloud + +## What, why, how? + +Lets briefly explain what dev containers are, why you should use them, and how they work. Here, we'll assume the use of VS Code, but the same concepts apply to other supporting tools and services, including alternative CLIs, IDEs, etc. such as: + +- [Dev Container CLI](https://github.com/devcontainers/cli) + - A reference implementation for the open specification +- [JetBrains Space | Develop in Dev Environment](https://www.jetbrains.com/help/space/develop-in-a-dev-environment.html) + - Using Dev Containers with JetBrain based products +- [Supporting tools]() + - List of tools and services supporting the development container specification + +### What is a Dev Container? + +A dev container is a Docker container that has all the tools and dependencies you need to develop the project. It runs in a self-contained environment and is isolated from other containers and your host machine. This lets you reliably develop for the project anywhere, notably for linux distributions targeted by ROS, regardless of your host machine's operating system. + +### Why use a Dev Container? + +A dev container provides a common and consistent development environment. It ensures that everyone on the team is using the same tools and dependencies. It also makes it easy to switch between projects because each project can use a different container. This is especially useful if you work on multiple projects that use different versions of the same tools and dependencies, such as different versions of ROS. + +### How do Dev Containers work? + +When you open the project in VS Code, VS Code checks for the dev container configuration nested within the `.devcontainer` folder under the project's root directory. If it finds one, it can prompt you to reopen the project in a container. If you choose to do so, it launches the container, connects to it, and mounts your project folder inside the container. You can then use VS Code in the container just as you would locally. While setting up the container, VS Code can also attempt to passthrough useful aspects of your local environment, such as git user configurations, X11 sockets, and more. + +This is quite similar to earlier tools used to customize and run docker containers for development: + +- [rocker | ROS + Docker](https://github.com/osrf/rocker) + - A tool to run docker containers with overlays and convenient options for things like GUIs etc. + - Developed by [Open Robotics](https://www.openrobotics.org/) +- [ADE Development Environment](https://ade-cli.readthedocs.io/en/latest/) + - A modular Docker-based tool to ensure developers have a common, consistent development environment + - Developed by [Apex.AI](https://www.apex.ai/) + +## Prerequisites + +To use dev containers, you'll need the following: + +- [Docker Engine](https://docs.docker.com/engine/install/) installed and running on the host machine +- [Visual Studio Code](https://code.visualstudio.com/) installed on any remote machine +- [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension installed in VS Code + +Alternatively, you could use GitHub Codespaces directly from the project repo: + +- [Creating a codespace for a repository](https://docs.github.com/en/codespaces/developing-in-codespaces/creating-a-codespace-for-a-repository?tool=webui) + - How to create a codespace for repository via GitHub CLI, VS Code, or Web browser + +## Creating Dev Containers + +Creating a dev container is as simple as opening the project in VS Code by either: following the prompts to reopen the project in a container, or explicitly opening the command palette and selecting `Remote-Containers: Reopen in Container`. This will create a new container, install any extensions specified in the project's default `.devcontainer/devcontainer.json` config file, and mount the project's root directory within the container. Once the container is created, VS Code will connect to it and you can start developing. + +While you wait, feel free to stretch your legs, grab a coffee, or continue to read the following subsections to learn more about how dev containers are constructed. Note: opening the output log in VS Code allows you to watch how this sausage is made. + +### Building the image + +When first creating Dev Containers, any supporting tool or service used will download and build the docker images needed to run the container. This includes pulling any parent images the project's Dockerfile builds `FROM`, as well as any tags or layers declared via `cacheFrom`, as specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until such layers are updated and pushed to the image registry. + +Specifically, for this project, the default `devcontainer.json` file targets the `dever` stage within the project's root Dockerfile, the stage that also includes handy tools for developing the project, such as bash auto completion. This stage is in turn built `FROM` the `builder` stage, the stage that only includes the dependencies needed for building the project, as reused by the project's CI. + +To speed up the initial build, images layers from this `builder` stage are cached by pulling the same image tag used by the project's CI, hosted from the image registry. This ensures your local dev container replicates our CI environment as close as possible, while benefiting from any cached work preemptively performed by the CI. Yet, this still allows you to customize the project's Dockerfile and rebuild the container, without needing to update CI images to reflect your local modifications. + +Once the base image from the target stage is built, the supporting tool or service may then add additional layers to the image, such as installing additional [features](https://containers.dev/features) or customizations. For VS Code, this also includes some fancy file caching for any extensions to install later. Once this custom image is built, it is then used to start the dev container. + +### Starting the container + +When first creating Dev Containers, any supporting tool or service will invoke a sequence of commands specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until the container is rebuilt, triggered by either updating the Dockerfile, base image, or `.devcontainer/` config. + +Specifically, for this project, the default `devcontainer.json` config executes the `onCreateCommand` to initially colcon cache, clean, and build the overlay workspace for the project. This ensures the workspace is precompiled and ready to use, while also ensuring any changes to the project's source code are reflected in the container. This is especially useful for: + +- IntelliSense + - Enables VS Code extensions to parse auto generated code + - Applicable for ROS package defining messages and services files + - Necessary for code modeling, navigation, and syntax highlighting +- Caching + - Enables Codespace Prebuilds to cache the workspace artifacts + - Applicable for reducing startup time when spawning new Codespaces + - Necessary for limiting costs from CPU and storage usage + +While the colcon workspace is being built, VS Code will simultaneously install any specified extensions and settings. Next the `updateContentCommand` is executed, which reruns whenever the container is started or restarted. Specifically, for this project, this command re-cleans and re-builds the same colcon workspace as before, but only for invalidated packages detected by colcon cache using the lockfiles initialized during the `onCreateCommand`. This caching behavior also replicates the project's CI workflow. This is especially useful for: + +- Branching + - Enables caching of workspace artifacts when switching between branches + - Applicable for reviewing pull requests without rebuilding entire container + - Necessary for reducing startup time when spawning new Codespaces + +Finally, the `postCreateCommand` is executed, which also reruns whenever the container is started or restarted. Specifically, for this project, this command makes a last few tweaks to the user's environment to improve the development experience. + +To speed up subsequent startups, volumes are mounted to store the ccache directory, while the environment is set to enable ccache via colcon mixins. + +## Using Dev Containers + +## From 9de030482f17b41779ccc9648b6371488c8701b1 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Mon, 27 Mar 2023 16:18:52 +0200 Subject: [PATCH 002/107] Add section on Security --- build_instructions/devcontainers.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/build_instructions/devcontainers.md b/build_instructions/devcontainers.md index 66659addb..6e2d777d3 100644 --- a/build_instructions/devcontainers.md +++ b/build_instructions/devcontainers.md @@ -99,3 +99,11 @@ To speed up subsequent startups, volumes are mounted to store the ccache directo ## Using Dev Containers ## + + +## Security + +A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. More info on trusting workspaces and extensions in general can be found here: + +- [Workspace Trust](https://code.visualstudio.com/docs/editor/workspace-trust) + - VS Code user guid on trusting and configure workspaces From 7827bc6c9c6b3614ed0f512d049b9724dcf64272 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Mon, 27 Mar 2023 17:04:00 +0200 Subject: [PATCH 003/107] Add description of container runtime and runArgs --- build_instructions/devcontainers.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/build_instructions/devcontainers.md b/build_instructions/devcontainers.md index 6e2d777d3..56c3f814e 100644 --- a/build_instructions/devcontainers.md +++ b/build_instructions/devcontainers.md @@ -94,7 +94,18 @@ While the colcon workspace is being built, VS Code will simultaneously install a Finally, the `postCreateCommand` is executed, which also reruns whenever the container is started or restarted. Specifically, for this project, this command makes a last few tweaks to the user's environment to improve the development experience. -To speed up subsequent startups, volumes are mounted to store the ccache directory, while the environment is set to enable ccache via colcon mixins. +To speed up subsequent startups, volumes are mounted to the container store a persistent ccache directory, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) capabilities and connected using the [host](https://docs.docker.com/network/host/) network mode. This is especially useful for: + +- Hybrid development + - Enables connecting ROS nodes external to the container + - Applicable for debugging or visualizing distributed systems + - Necessary for DDS discovery and shared memory transport +- Device connectivity + - Enables hardware forwarding from host machine to container + - Applicable for ROS package using sensors and actuators + - Necessary for some GPU drivers and USB devices + +Note that these `runArgs` in the `devcontainer.json` config can be further customized, either expanded or or narrowed in scope, to better suit your desired development environment. The current configuration is merely the project default in order to be the most flexible and useful for the widest range of development use cases. ## Using Dev Containers From 67a0f2e3fec8da5c39b51dd3c774aa9e14b70823 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Mon, 27 Mar 2023 18:19:46 +0200 Subject: [PATCH 004/107] Add to using dev container --- build_instructions/devcontainers.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/build_instructions/devcontainers.md b/build_instructions/devcontainers.md index 56c3f814e..3994624d3 100644 --- a/build_instructions/devcontainers.md +++ b/build_instructions/devcontainers.md @@ -109,6 +109,10 @@ Note that these `runArgs` in the `devcontainer.json` config can be further custo ## Using Dev Containers +Once the dev container has been created and setup completed, VS Code will open a new workspace directly from the project's root directory, which itself is mounted within the source directory in the overlay colcon workspace. From here you can build, test, and debug the project as you normally would, with the added benefit of having the project's dependencies, intellisense, linters, and other extensions pre-configured and ready to use. + +So to build or test the project, simply open a terminal, cd to the root of the colcon workspace, and run the usual colcon commands. You may also want to incorporate the same colcon verbs used by the setup commands from the `devcontainer.json` config file to further automate your local development workflow. + ## From 16ff8e14acf5bebbef4e634671240803525070cf Mon Sep 17 00:00:00 2001 From: ruffsl Date: Mon, 27 Mar 2023 18:21:30 +0200 Subject: [PATCH 005/107] Add subsection on Terminals --- build_instructions/devcontainers.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/build_instructions/devcontainers.md b/build_instructions/devcontainers.md index 3994624d3..32c66a1b1 100644 --- a/build_instructions/devcontainers.md +++ b/build_instructions/devcontainers.md @@ -64,7 +64,7 @@ While you wait, feel free to stretch your legs, grab a coffee, or continue to re When first creating Dev Containers, any supporting tool or service used will download and build the docker images needed to run the container. This includes pulling any parent images the project's Dockerfile builds `FROM`, as well as any tags or layers declared via `cacheFrom`, as specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until such layers are updated and pushed to the image registry. -Specifically, for this project, the default `devcontainer.json` file targets the `dever` stage within the project's root Dockerfile, the stage that also includes handy tools for developing the project, such as bash auto completion. This stage is in turn built `FROM` the `builder` stage, the stage that only includes the dependencies needed for building the project, as reused by the project's CI. +Specifically, for this project, the default `devcontainer.json` file targets the `dever` stage within the project's root Dockerfile, the stage that also includes handy tools for developing the project, such as bash auto completion. This stage is in turn built `FROM` the `builder` stage, the stage that only includes the dependencies needed for building the project, as reused by the project's CI. For example, the `dever` stage modifies `/etc/bash.bashrc` to automatically source `install/setup.bash` from the underlay workspace, ensuring all VS Code extensions are loaded with the correct environment, while avoiding any race conditions during installation and startup. To speed up the initial build, images layers from this `builder` stage are cached by pulling the same image tag used by the project's CI, hosted from the image registry. This ensures your local dev container replicates our CI environment as close as possible, while benefiting from any cached work preemptively performed by the CI. Yet, this still allows you to customize the project's Dockerfile and rebuild the container, without needing to update CI images to reflect your local modifications. @@ -113,8 +113,15 @@ Once the dev container has been created and setup completed, VS Code will open a So to build or test the project, simply open a terminal, cd to the root of the colcon workspace, and run the usual colcon commands. You may also want to incorporate the same colcon verbs used by the setup commands from the `devcontainer.json` config file to further automate your local development workflow. -## +### Terminals +If you prefer using alternate terminal emulators, rather than the built-in VS Code terminal, you can also open a separate shell session directly using devcontainer CLI, or simply via docker exec. Note that new shell sessions started via `exec` will not automatically inherit the same environment setup by the `postCreateCommand` from the `devcontainer.json` config file. So you may need to manually source any necessary scripts, such as `install/setup.bash` from the underlay workspace. + +- [Dev Container CLI](https://code.visualstudio.com/docs/devcontainers/devcontainer-cli) + - `devcontainer exec --workspace-folder bash` +- [docker exec +](https://docs.docker.com/engine/reference/commandline/exec/) + - `docker exec -it bash` ## Security From b05f07052603c6afc57ed9c348110a1452ef6ac6 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Mon, 27 Mar 2023 18:39:47 +0200 Subject: [PATCH 006/107] Add subsection on Lifecycle --- build_instructions/devcontainers.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/build_instructions/devcontainers.md b/build_instructions/devcontainers.md index 32c66a1b1..0c8d1124d 100644 --- a/build_instructions/devcontainers.md +++ b/build_instructions/devcontainers.md @@ -123,6 +123,13 @@ If you prefer using alternate terminal emulators, rather than the built-in VS Co ](https://docs.docker.com/engine/reference/commandline/exec/) - `docker exec -it bash` +### Lifecycle + +While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces: + +- [The codespace lifecycle](https://docs.github.com/en/codespaces/getting-started/the-codespace-lifecycle) + - Maintain your data throughout the entire codespace lifecycle + ## Security A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. More info on trusting workspaces and extensions in general can be found here: From 3bb9e625265aa73318c8c6b6ad6c58b783ef9ce7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 28 Mar 2023 12:53:57 +0200 Subject: [PATCH 007/107] Add reminder to Lifecycle about pushing to remote --- build_instructions/devcontainers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build_instructions/devcontainers.md b/build_instructions/devcontainers.md index 0c8d1124d..071eda31b 100644 --- a/build_instructions/devcontainers.md +++ b/build_instructions/devcontainers.md @@ -125,7 +125,7 @@ If you prefer using alternate terminal emulators, rather than the built-in VS Co ### Lifecycle -While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces: +While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces, so be sure to commit and push local changes to a remote repository: - [The codespace lifecycle](https://docs.github.com/en/codespaces/getting-started/the-codespace-lifecycle) - Maintain your data throughout the entire codespace lifecycle From b3bded1feac73a5fb77d1c734d3f342de12bd2c7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 28 Mar 2023 19:12:56 +0200 Subject: [PATCH 008/107] Add subsection on Rebuilding --- build_instructions/devcontainers.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/build_instructions/devcontainers.md b/build_instructions/devcontainers.md index 071eda31b..2bd5e5d00 100644 --- a/build_instructions/devcontainers.md +++ b/build_instructions/devcontainers.md @@ -130,6 +130,28 @@ While using the dev container, try and keep in mind the lifecycle of the contain - [The codespace lifecycle](https://docs.github.com/en/codespaces/getting-started/the-codespace-lifecycle) - Maintain your data throughout the entire codespace lifecycle +### Rebuilding + +From time to time, you may need to rebuild the dev container, either because the base image, or `.devcontainer/` config was updated, or simply out of wanting a new fresh development environment. To do so, simply open the Command Palette (Ctrl+Shift+P) and select the `Remote-Containers: Rebuild Container` command. For example, you may need to rebuild the dev container when: + +- Pulling newer images from a container registry + - specifically, image tags built `FROM` in the Dockerfile + - or tags listed under `cacheFrom` in `devcontainer.json` + - periodically done manually to ensure local environment reflects CI +- Updating the dev container configuration + - specifically when modifying dependent stages in the `Dockerfile` + - or when modifying `./devcontainer` files and commands + - where build cache reuse correlates with severity of changes made + +If necessary, you can also rebuild the container from scratch, e.i. without caching from docker, by selecting the `Remote-Containers: Rebuild Container Without Cache` command instead. Rebuilding without caching may be necessary when: + +- Needing to update the base image + - specifically if dev container configurations remain unmodified + - to forcefully rerun a `RUN` directive in the Dockerfile + - such as unchanged `apt upgrade` or `rosdep update` commands + +Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory. The management of these volumes is left for the developers discretion, and can be done via the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/), or the [VS Code Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker). However, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. + ## Security A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. More info on trusting workspaces and extensions in general can be found here: From f0a0f2b50122bc19ff1c66cbfe4c19fd136e34e1 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 11:46:49 +0200 Subject: [PATCH 009/107] Rename file to own subtree --- build_instructions/devcontainers.md => dev_containers/index.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename build_instructions/devcontainers.md => dev_containers/index.md (100%) diff --git a/build_instructions/devcontainers.md b/dev_containers/index.md similarity index 100% rename from build_instructions/devcontainers.md rename to dev_containers/index.md From 6880fe0ab498552f6ae638ad2af656b2a10117e1 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 12:19:27 +0200 Subject: [PATCH 010/107] Enable markdown support for Sphinx https://stackoverflow.com/a/33797841/2577586 --- .circleci/config.yml | 2 +- README.md | 2 +- conf.py | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 084b735e8..e1a5695bc 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -17,7 +17,7 @@ references: apt install openjdk-8-jre -y apt install bash git openssh-server -y apt install python3-pip python3 -y - pip3 install sphinx==3.5.0 docutils==0.14 sphinx_rtd_theme breathe==4.28.0 sphinxcontrib-plantuml jinja2==3.0.3 + pip3 install sphinx==3.5.0 docutils==0.14 sphinx_rtd_theme breathe==4.28.0 sphinxcontrib-plantuml jinja2==3.0.3 myst-parser make_docs: &make_docs run: command: | diff --git a/README.md b/README.md index 72dda5ae5..4add7f786 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ This folder holds the source and configuration files used to generate the Dependencies for Build: * `sudo apt install python3-pip` -* `pip3 install sphinx==3.5.0 breathe==4.28.0 sphinx_rtd_theme sphinxcontrib-plantuml jinja2==3.0.3` +* `pip3 install sphinx==3.5.0 breathe==4.28.0 sphinx_rtd_theme sphinxcontrib-plantuml jinja2==3.0.3 myst-parser` Build the docs locally with `make html` and you'll find the built docs entry point in `_build/html/index.html`. diff --git a/conf.py b/conf.py index 856ef8218..8ab9140c7 100644 --- a/conf.py +++ b/conf.py @@ -30,7 +30,7 @@ # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['breathe', 'sphinx.ext.graphviz', 'sphinxcontrib.plantuml', 'sphinx.ext.extlinks'] +extensions = ['breathe', 'myst_parser', 'sphinx.ext.graphviz', 'sphinxcontrib.plantuml', 'sphinx.ext.extlinks'] graphviz_output_format='png' graphviz_dot_args=[ @@ -48,7 +48,7 @@ # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] -source_suffix = '.rst' +source_suffix = ['.rst', '.md'] # The master toctree document. master_doc = 'index' From f1aa71f09981d5739c9d7ea3387d66e0f1017995 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 12:23:43 +0200 Subject: [PATCH 011/107] Add dev_containers to root index --- index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/index.rst b/index.rst index 2e4d8b21d..b1607a88b 100644 --- a/index.rst +++ b/index.rst @@ -137,6 +137,7 @@ Below is an example of the TB3 navigating in a small lounge. getting_started/index.rst build_instructions/index.rst + dev_containers/index.md concepts/index.rst setup_guides/index.rst tutorials/index.rst From 293e78e4676314d0d07c45e96c61a295a3acfeca Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 12:31:04 +0200 Subject: [PATCH 012/107] Fix missing URL --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 2bd5e5d00..e246087e3 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -17,7 +17,7 @@ Lets briefly explain what dev containers are, why you should use them, and how t - A reference implementation for the open specification - [JetBrains Space | Develop in Dev Environment](https://www.jetbrains.com/help/space/develop-in-a-dev-environment.html) - Using Dev Containers with JetBrain based products -- [Supporting tools]() +- [Supporting tools](https://containers.dev/supporting) - List of tools and services supporting the development container specification ### What is a Dev Container? From 6601960a61339cd31dc1d9a9b06163bd38c03ba2 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 14:40:11 +0200 Subject: [PATCH 013/107] Spin out content into development guide to shorten the introduction --- .../dev_containers_docs/development_guide.md | 93 +++++++++++++++++++ dev_containers/index.md | 92 ------------------ 2 files changed, 93 insertions(+), 92 deletions(-) create mode 100644 dev_containers/dev_containers_docs/development_guide.md diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md new file mode 100644 index 000000000..8c2899d89 --- /dev/null +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -0,0 +1,93 @@ +# Development Guide + +### Building the image + +When first creating Dev Containers, any supporting tool or service used will download and build the docker images needed to run the container. This includes pulling any parent images the project's Dockerfile builds `FROM`, as well as any tags or layers declared via `cacheFrom`, as specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until such layers are updated and pushed to the image registry. + +Specifically, for this project, the default `devcontainer.json` file targets the `dever` stage within the project's root Dockerfile, the stage that also includes handy tools for developing the project, such as bash auto completion. This stage is in turn built `FROM` the `builder` stage, the stage that only includes the dependencies needed for building the project, as reused by the project's CI. For example, the `dever` stage modifies `/etc/bash.bashrc` to automatically source `install/setup.bash` from the underlay workspace, ensuring all VS Code extensions are loaded with the correct environment, while avoiding any race conditions during installation and startup. + +To speed up the initial build, images layers from this `builder` stage are cached by pulling the same image tag used by the project's CI, hosted from the image registry. This ensures your local dev container replicates our CI environment as close as possible, while benefiting from any cached work preemptively performed by the CI. Yet, this still allows you to customize the project's Dockerfile and rebuild the container, without needing to update CI images to reflect your local modifications. + +Once the base image from the target stage is built, the supporting tool or service may then add additional layers to the image, such as installing additional [features](https://containers.dev/features) or customizations. For VS Code, this also includes some fancy file caching for any extensions to install later. Once this custom image is built, it is then used to start the dev container. + +### Starting the container + +When first creating Dev Containers, any supporting tool or service will invoke a sequence of commands specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until the container is rebuilt, triggered by either updating the Dockerfile, base image, or `.devcontainer/` config. + +Specifically, for this project, the default `devcontainer.json` config executes the `onCreateCommand` to initially colcon cache, clean, and build the overlay workspace for the project. This ensures the workspace is precompiled and ready to use, while also ensuring any changes to the project's source code are reflected in the container. This is especially useful for: + +- IntelliSense + - Enables VS Code extensions to parse auto generated code + - Applicable for ROS package defining messages and services files + - Necessary for code modeling, navigation, and syntax highlighting +- Caching + - Enables Codespace Prebuilds to cache the workspace artifacts + - Applicable for reducing startup time when spawning new Codespaces + - Necessary for limiting costs from CPU and storage usage + +While the colcon workspace is being built, VS Code will simultaneously install any specified extensions and settings. Next the `updateContentCommand` is executed, which reruns whenever the container is started or restarted. Specifically, for this project, this command re-cleans and re-builds the same colcon workspace as before, but only for invalidated packages detected by colcon cache using the lockfiles initialized during the `onCreateCommand`. This caching behavior also replicates the project's CI workflow. This is especially useful for: + +- Branching + - Enables caching of workspace artifacts when switching between branches + - Applicable for reviewing pull requests without rebuilding entire container + - Necessary for reducing startup time when spawning new Codespaces + +Finally, the `postCreateCommand` is executed, which also reruns whenever the container is started or restarted. Specifically, for this project, this command makes a last few tweaks to the user's environment to improve the development experience. + +To speed up subsequent startups, volumes are mounted to the container store a persistent ccache directory, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) capabilities and connected using the [host](https://docs.docker.com/network/host/) network mode. This is especially useful for: + +- Hybrid development + - Enables connecting ROS nodes external to the container + - Applicable for debugging or visualizing distributed systems + - Necessary for DDS discovery and shared memory transport +- Device connectivity + - Enables hardware forwarding from host machine to container + - Applicable for ROS package using sensors and actuators + - Necessary for some GPU drivers and USB devices + +Note that these `runArgs` in the `devcontainer.json` config can be further customized, either expanded or or narrowed in scope, to better suit your desired development environment. The current configuration is merely the project default in order to be the most flexible and useful for the widest range of development use cases. + +## Using Dev Containers + +Once the dev container has been created and setup completed, VS Code will open a new workspace directly from the project's root directory, which itself is mounted within the source directory in the overlay colcon workspace. From here you can build, test, and debug the project as you normally would, with the added benefit of having the project's dependencies, intellisense, linters, and other extensions pre-configured and ready to use. + +So to build or test the project, simply open a terminal, cd to the root of the colcon workspace, and run the usual colcon commands. You may also want to incorporate the same colcon verbs used by the setup commands from the `devcontainer.json` config file to further automate your local development workflow. + +### Terminals + +If you prefer using alternate terminal emulators, rather than the built-in VS Code terminal, you can also open a separate shell session directly using devcontainer CLI, or simply via docker exec. Note that new shell sessions started via `exec` will not automatically inherit the same environment setup by the `postCreateCommand` from the `devcontainer.json` config file. So you may need to manually source any necessary scripts, such as `install/setup.bash` from the underlay workspace. + +- [Dev Container CLI](https://code.visualstudio.com/docs/devcontainers/devcontainer-cli) + - `devcontainer exec --workspace-folder bash` +- [docker exec +](https://docs.docker.com/engine/reference/commandline/exec/) + - `docker exec -it bash` + +### Lifecycle + +While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces, so be sure to commit and push local changes to a remote repository: + +- [The codespace lifecycle](https://docs.github.com/en/codespaces/getting-started/the-codespace-lifecycle) + - Maintain your data throughout the entire codespace lifecycle + +### Rebuilding + +From time to time, you may need to rebuild the dev container, either because the base image, or `.devcontainer/` config was updated, or simply out of wanting a new fresh development environment. To do so, simply open the Command Palette (Ctrl+Shift+P) and select the `Remote-Containers: Rebuild Container` command. For example, you may need to rebuild the dev container when: + +- Pulling newer images from a container registry + - specifically, image tags built `FROM` in the Dockerfile + - or tags listed under `cacheFrom` in `devcontainer.json` + - periodically done manually to ensure local environment reflects CI +- Updating the dev container configuration + - specifically when modifying dependent stages in the `Dockerfile` + - or when modifying `./devcontainer` files and commands + - where build cache reuse correlates with severity of changes made + +If necessary, you can also rebuild the container from scratch, e.i. without caching from docker, by selecting the `Remote-Containers: Rebuild Container Without Cache` command instead. Rebuilding without caching may be necessary when: + +- Needing to update the base image + - specifically if dev container configurations remain unmodified + - to forcefully rerun a `RUN` directive in the Dockerfile + - such as unchanged `apt upgrade` or `rosdep update` commands + +Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory. The management of these volumes is left for the developers discretion, and can be done via the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/), or the [VS Code Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker). However, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. diff --git a/dev_containers/index.md b/dev_containers/index.md index e246087e3..263e8d7b0 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -60,98 +60,6 @@ Creating a dev container is as simple as opening the project in VS Code by eithe While you wait, feel free to stretch your legs, grab a coffee, or continue to read the following subsections to learn more about how dev containers are constructed. Note: opening the output log in VS Code allows you to watch how this sausage is made. -### Building the image - -When first creating Dev Containers, any supporting tool or service used will download and build the docker images needed to run the container. This includes pulling any parent images the project's Dockerfile builds `FROM`, as well as any tags or layers declared via `cacheFrom`, as specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until such layers are updated and pushed to the image registry. - -Specifically, for this project, the default `devcontainer.json` file targets the `dever` stage within the project's root Dockerfile, the stage that also includes handy tools for developing the project, such as bash auto completion. This stage is in turn built `FROM` the `builder` stage, the stage that only includes the dependencies needed for building the project, as reused by the project's CI. For example, the `dever` stage modifies `/etc/bash.bashrc` to automatically source `install/setup.bash` from the underlay workspace, ensuring all VS Code extensions are loaded with the correct environment, while avoiding any race conditions during installation and startup. - -To speed up the initial build, images layers from this `builder` stage are cached by pulling the same image tag used by the project's CI, hosted from the image registry. This ensures your local dev container replicates our CI environment as close as possible, while benefiting from any cached work preemptively performed by the CI. Yet, this still allows you to customize the project's Dockerfile and rebuild the container, without needing to update CI images to reflect your local modifications. - -Once the base image from the target stage is built, the supporting tool or service may then add additional layers to the image, such as installing additional [features](https://containers.dev/features) or customizations. For VS Code, this also includes some fancy file caching for any extensions to install later. Once this custom image is built, it is then used to start the dev container. - -### Starting the container - -When first creating Dev Containers, any supporting tool or service will invoke a sequence of commands specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until the container is rebuilt, triggered by either updating the Dockerfile, base image, or `.devcontainer/` config. - -Specifically, for this project, the default `devcontainer.json` config executes the `onCreateCommand` to initially colcon cache, clean, and build the overlay workspace for the project. This ensures the workspace is precompiled and ready to use, while also ensuring any changes to the project's source code are reflected in the container. This is especially useful for: - -- IntelliSense - - Enables VS Code extensions to parse auto generated code - - Applicable for ROS package defining messages and services files - - Necessary for code modeling, navigation, and syntax highlighting -- Caching - - Enables Codespace Prebuilds to cache the workspace artifacts - - Applicable for reducing startup time when spawning new Codespaces - - Necessary for limiting costs from CPU and storage usage - -While the colcon workspace is being built, VS Code will simultaneously install any specified extensions and settings. Next the `updateContentCommand` is executed, which reruns whenever the container is started or restarted. Specifically, for this project, this command re-cleans and re-builds the same colcon workspace as before, but only for invalidated packages detected by colcon cache using the lockfiles initialized during the `onCreateCommand`. This caching behavior also replicates the project's CI workflow. This is especially useful for: - -- Branching - - Enables caching of workspace artifacts when switching between branches - - Applicable for reviewing pull requests without rebuilding entire container - - Necessary for reducing startup time when spawning new Codespaces - -Finally, the `postCreateCommand` is executed, which also reruns whenever the container is started or restarted. Specifically, for this project, this command makes a last few tweaks to the user's environment to improve the development experience. - -To speed up subsequent startups, volumes are mounted to the container store a persistent ccache directory, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) capabilities and connected using the [host](https://docs.docker.com/network/host/) network mode. This is especially useful for: - -- Hybrid development - - Enables connecting ROS nodes external to the container - - Applicable for debugging or visualizing distributed systems - - Necessary for DDS discovery and shared memory transport -- Device connectivity - - Enables hardware forwarding from host machine to container - - Applicable for ROS package using sensors and actuators - - Necessary for some GPU drivers and USB devices - -Note that these `runArgs` in the `devcontainer.json` config can be further customized, either expanded or or narrowed in scope, to better suit your desired development environment. The current configuration is merely the project default in order to be the most flexible and useful for the widest range of development use cases. - -## Using Dev Containers - -Once the dev container has been created and setup completed, VS Code will open a new workspace directly from the project's root directory, which itself is mounted within the source directory in the overlay colcon workspace. From here you can build, test, and debug the project as you normally would, with the added benefit of having the project's dependencies, intellisense, linters, and other extensions pre-configured and ready to use. - -So to build or test the project, simply open a terminal, cd to the root of the colcon workspace, and run the usual colcon commands. You may also want to incorporate the same colcon verbs used by the setup commands from the `devcontainer.json` config file to further automate your local development workflow. - -### Terminals - -If you prefer using alternate terminal emulators, rather than the built-in VS Code terminal, you can also open a separate shell session directly using devcontainer CLI, or simply via docker exec. Note that new shell sessions started via `exec` will not automatically inherit the same environment setup by the `postCreateCommand` from the `devcontainer.json` config file. So you may need to manually source any necessary scripts, such as `install/setup.bash` from the underlay workspace. - -- [Dev Container CLI](https://code.visualstudio.com/docs/devcontainers/devcontainer-cli) - - `devcontainer exec --workspace-folder bash` -- [docker exec -](https://docs.docker.com/engine/reference/commandline/exec/) - - `docker exec -it bash` - -### Lifecycle - -While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces, so be sure to commit and push local changes to a remote repository: - -- [The codespace lifecycle](https://docs.github.com/en/codespaces/getting-started/the-codespace-lifecycle) - - Maintain your data throughout the entire codespace lifecycle - -### Rebuilding - -From time to time, you may need to rebuild the dev container, either because the base image, or `.devcontainer/` config was updated, or simply out of wanting a new fresh development environment. To do so, simply open the Command Palette (Ctrl+Shift+P) and select the `Remote-Containers: Rebuild Container` command. For example, you may need to rebuild the dev container when: - -- Pulling newer images from a container registry - - specifically, image tags built `FROM` in the Dockerfile - - or tags listed under `cacheFrom` in `devcontainer.json` - - periodically done manually to ensure local environment reflects CI -- Updating the dev container configuration - - specifically when modifying dependent stages in the `Dockerfile` - - or when modifying `./devcontainer` files and commands - - where build cache reuse correlates with severity of changes made - -If necessary, you can also rebuild the container from scratch, e.i. without caching from docker, by selecting the `Remote-Containers: Rebuild Container Without Cache` command instead. Rebuilding without caching may be necessary when: - -- Needing to update the base image - - specifically if dev container configurations remain unmodified - - to forcefully rerun a `RUN` directive in the Dockerfile - - such as unchanged `apt upgrade` or `rosdep update` commands - -Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory. The management of these volumes is left for the developers discretion, and can be done via the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/), or the [VS Code Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker). However, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. - ## Security A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. More info on trusting workspaces and extensions in general can be found here: From 839e4d2cb68b876687c5d94fc5f80e9eb59e68c4 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 14:42:10 +0200 Subject: [PATCH 014/107] Add guides to index --- .../dev_containers_docs/visualization_guide.md | 1 + dev_containers/index.md | 14 +++++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-) create mode 100644 dev_containers/dev_containers_docs/visualization_guide.md diff --git a/dev_containers/dev_containers_docs/visualization_guide.md b/dev_containers/dev_containers_docs/visualization_guide.md new file mode 100644 index 000000000..7b676fca3 --- /dev/null +++ b/dev_containers/dev_containers_docs/visualization_guide.md @@ -0,0 +1 @@ +# Visualization Guide diff --git a/dev_containers/index.md b/dev_containers/index.md index 263e8d7b0..5e4e8dead 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -9,6 +9,13 @@ Alternatively you can use dev containers to build the project if you prefer a st - [GitHub Codespaces overview](https://docs.github.com/en/codespaces/overview) - A development environment hosted in the cloud +```{toctree} +:hidden: + +dev_containers_docs/development_guide.md +dev_containers_docs/visualization_guide.md +``` + ## What, why, how? Lets briefly explain what dev containers are, why you should use them, and how they work. Here, we'll assume the use of VS Code, but the same concepts apply to other supporting tools and services, including alternative CLIs, IDEs, etc. such as: @@ -58,7 +65,12 @@ Alternatively, you could use GitHub Codespaces directly from the project repo: Creating a dev container is as simple as opening the project in VS Code by either: following the prompts to reopen the project in a container, or explicitly opening the command palette and selecting `Remote-Containers: Reopen in Container`. This will create a new container, install any extensions specified in the project's default `.devcontainer/devcontainer.json` config file, and mount the project's root directory within the container. Once the container is created, VS Code will connect to it and you can start developing. -While you wait, feel free to stretch your legs, grab a coffee, or continue to read the following subsections to learn more about how dev containers are constructed. Note: opening the output log in VS Code allows you to watch how this sausage is made. +While you wait, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about how dev containers are constructed. Note: opening the output log in VS Code allows you to watch how this sausage is made. + +- [](dev_containers_docs/development_guide.md) + - How to develop Nav2 using dev containers and supporting tools +- [](dev_containers_docs/visualization_guide.md) + - How to visualize Nav2 using local or remote headless containers ## Security From 9f7ed795d47612d43271bd82e173824f2f073c92 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 15:37:35 +0200 Subject: [PATCH 015/107] Fix subheading levels to prevent warning as described here: https://myst-parser.readthedocs.io/en/latest/syntax/organising_content.html#document-structure --- dev_containers/dev_containers_docs/development_guide.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 8c2899d89..7463de30a 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -1,5 +1,7 @@ # Development Guide +## Creating Dev Containers + ### Building the image When first creating Dev Containers, any supporting tool or service used will download and build the docker images needed to run the container. This includes pulling any parent images the project's Dockerfile builds `FROM`, as well as any tags or layers declared via `cacheFrom`, as specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until such layers are updated and pushed to the image registry. From 2fd240c8982ff554b7ad013506644f4d529ac0ed Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 15:41:45 +0200 Subject: [PATCH 016/107] Add reference to using remote docker host as an alternative to codespaces --- dev_containers/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 5e4e8dead..423ac44f9 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -56,10 +56,12 @@ To use dev containers, you'll need the following: - [Visual Studio Code](https://code.visualstudio.com/) installed on any remote machine - [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension installed in VS Code -Alternatively, you could use GitHub Codespaces directly from the project repo: +Alternatively, you could use GitHub Codespaces directly from the project repo, or any other remote host machine: - [Creating a codespace for a repository](https://docs.github.com/en/codespaces/developing-in-codespaces/creating-a-codespace-for-a-repository?tool=webui) - How to create a codespace for repository via GitHub CLI, VS Code, or Web browser +- [Develop on a remote Docker host](https://code.visualstudio.com/remote/advancedcontainers/develop-remote-host) + - How to connect VS Code to a remote Docker host using SSH or TCP sockets ## Creating Dev Containers From 2a70f542c3aabcff13ec9e6392b0b288fff116b7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:04:14 +0200 Subject: [PATCH 017/107] Update section Getting stared --- dev_containers/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 423ac44f9..7e608811d 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -63,11 +63,11 @@ Alternatively, you could use GitHub Codespaces directly from the project repo, o - [Develop on a remote Docker host](https://code.visualstudio.com/remote/advancedcontainers/develop-remote-host) - How to connect VS Code to a remote Docker host using SSH or TCP sockets -## Creating Dev Containers +## Getting started -Creating a dev container is as simple as opening the project in VS Code by either: following the prompts to reopen the project in a container, or explicitly opening the command palette and selecting `Remote-Containers: Reopen in Container`. This will create a new container, install any extensions specified in the project's default `.devcontainer/devcontainer.json` config file, and mount the project's root directory within the container. Once the container is created, VS Code will connect to it and you can start developing. +Getting started using dev containers is as simple as opening the project in VS Code by either: following the notification prompt to reopen the project in a container, or explicitly opening the command palette (Crtl+Shift+P) and selecting `Remote-Containers: Reopen in Container`. This will create a new container, install any extensions specified in the project's default `.devcontainer/devcontainer.json` config file, and mount the project's root directory as the workspace folder. Once the container is created, VS Code will connect to it and you can start developing. -While you wait, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about how dev containers are constructed. Note: opening the output log in VS Code allows you to watch how this sausage is made. +While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about how dev containers are constructed. Note: clicking the `Starting Dev Container (show log)` notification in VS Code allows you to observe how the sausage is made. - [](dev_containers_docs/development_guide.md) - How to develop Nav2 using dev containers and supporting tools From c8a030d5648b73f07478290f6f539a47f7569554 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:12:52 +0200 Subject: [PATCH 018/107] Enalbe admonitions and callouts highlighting By adding "colon_fence" to myst_enable_extensions https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#syntax-colon-fence --- conf.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/conf.py b/conf.py index 8ab9140c7..70eab0c73 100644 --- a/conf.py +++ b/conf.py @@ -32,6 +32,8 @@ # ones. extensions = ['breathe', 'myst_parser', 'sphinx.ext.graphviz', 'sphinxcontrib.plantuml', 'sphinx.ext.extlinks'] +myst_enable_extensions = ['colon_fence'] + graphviz_output_format='png' graphviz_dot_args=[ '-Nfontname="verdana"', From 024d0ed693ebcb5d4d6363069b57870e8c32d7cc Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:15:48 +0200 Subject: [PATCH 019/107] Use tip admonition --- dev_containers/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 7e608811d..51bbf1a86 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -67,7 +67,11 @@ Alternatively, you could use GitHub Codespaces directly from the project repo, o Getting started using dev containers is as simple as opening the project in VS Code by either: following the notification prompt to reopen the project in a container, or explicitly opening the command palette (Crtl+Shift+P) and selecting `Remote-Containers: Reopen in Container`. This will create a new container, install any extensions specified in the project's default `.devcontainer/devcontainer.json` config file, and mount the project's root directory as the workspace folder. Once the container is created, VS Code will connect to it and you can start developing. -While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about how dev containers are constructed. Note: clicking the `Starting Dev Container (show log)` notification in VS Code allows you to observe how the sausage is made. +While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about how dev containers are constructed. + +:::{tip} +Clicking the `Starting Dev Container (show log)` notification in VS Code allows you to observe how the sausage is made. +::: - [](dev_containers_docs/development_guide.md) - How to develop Nav2 using dev containers and supporting tools From 1792267b05619fb96e4ebc22b243c96c1ec883ad Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:18:47 +0200 Subject: [PATCH 020/107] Use node admonition --- dev_containers/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dev_containers/index.md b/dev_containers/index.md index 51bbf1a86..2ab696dcf 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -56,12 +56,14 @@ To use dev containers, you'll need the following: - [Visual Studio Code](https://code.visualstudio.com/) installed on any remote machine - [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension installed in VS Code +:::{note} Alternatively, you could use GitHub Codespaces directly from the project repo, or any other remote host machine: - [Creating a codespace for a repository](https://docs.github.com/en/codespaces/developing-in-codespaces/creating-a-codespace-for-a-repository?tool=webui) - How to create a codespace for repository via GitHub CLI, VS Code, or Web browser - [Develop on a remote Docker host](https://code.visualstudio.com/remote/advancedcontainers/develop-remote-host) - How to connect VS Code to a remote Docker host using SSH or TCP sockets +::: ## Getting started From 10c2c76a5132b5fbce7e51d5a39600f2ed0423b0 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:31:02 +0200 Subject: [PATCH 021/107] Rewording --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 2ab696dcf..dd9a6385a 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -69,7 +69,7 @@ Alternatively, you could use GitHub Codespaces directly from the project repo, o Getting started using dev containers is as simple as opening the project in VS Code by either: following the notification prompt to reopen the project in a container, or explicitly opening the command palette (Crtl+Shift+P) and selecting `Remote-Containers: Reopen in Container`. This will create a new container, install any extensions specified in the project's default `.devcontainer/devcontainer.json` config file, and mount the project's root directory as the workspace folder. Once the container is created, VS Code will connect to it and you can start developing. -While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about how dev containers are constructed. +While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about creating and user dev containers, or how to visualize and leverage graphical user interfaces from a headless development environment. :::{tip} Clicking the `Starting Dev Container (show log)` notification in VS Code allows you to observe how the sausage is made. From ec62d89b177c1d33360b7396f3aff4cf43bae7f1 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:31:37 +0200 Subject: [PATCH 022/107] Fromatting --- dev_containers/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index dd9a6385a..35f2c1cad 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -69,15 +69,15 @@ Alternatively, you could use GitHub Codespaces directly from the project repo, o Getting started using dev containers is as simple as opening the project in VS Code by either: following the notification prompt to reopen the project in a container, or explicitly opening the command palette (Crtl+Shift+P) and selecting `Remote-Containers: Reopen in Container`. This will create a new container, install any extensions specified in the project's default `.devcontainer/devcontainer.json` config file, and mount the project's root directory as the workspace folder. Once the container is created, VS Code will connect to it and you can start developing. -While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about creating and user dev containers, or how to visualize and leverage graphical user interfaces from a headless development environment. - :::{tip} Clicking the `Starting Dev Container (show log)` notification in VS Code allows you to observe how the sausage is made. ::: -- [](dev_containers_docs/development_guide.md) +While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about creating and user dev containers, or how to visualize and leverage graphical user interfaces from a headless development environment. + +- **[](dev_containers_docs/development_guide.md)** - How to develop Nav2 using dev containers and supporting tools -- [](dev_containers_docs/visualization_guide.md) +- **[](dev_containers_docs/visualization_guide.md)** - How to visualize Nav2 using local or remote headless containers ## Security From 782bddec16ee9a0dbbdca0dde30552406b6a9a2f Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:43:01 +0200 Subject: [PATCH 023/107] Use caution admonition --- dev_containers/index.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 35f2c1cad..c6ec727ce 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -82,7 +82,13 @@ While waiting for the initial setup, feel free to stretch your legs, grab a coff ## Security -A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. More info on trusting workspaces and extensions in general can be found here: +A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. + +:::{caution} +Ensure you trust the authors and contents of a workspaces before launching any dev containers. +::: + +More info on trusting workspaces and extensions in general can be found here: - [Workspace Trust](https://code.visualstudio.com/docs/editor/workspace-trust) - VS Code user guid on trusting and configure workspaces From 26c0d5c21296b172ca53dfe5cbff4675496d9fe3 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:54:23 +0200 Subject: [PATCH 024/107] Use hint admonition --- dev_containers/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index c6ec727ce..8044c0981 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -18,7 +18,10 @@ dev_containers_docs/visualization_guide.md ## What, why, how? -Lets briefly explain what dev containers are, why you should use them, and how they work. Here, we'll assume the use of VS Code, but the same concepts apply to other supporting tools and services, including alternative CLIs, IDEs, etc. such as: +Lets briefly explain what dev containers are, why you should use them, and how they work. + +:::{hint} +Here we'll assume the use of VS Code, but still applies to alternative tools and services, including other CLIs, IDEs, etc. such as: - [Dev Container CLI](https://github.com/devcontainers/cli) - A reference implementation for the open specification @@ -26,6 +29,7 @@ Lets briefly explain what dev containers are, why you should use them, and how t - Using Dev Containers with JetBrain based products - [Supporting tools](https://containers.dev/supporting) - List of tools and services supporting the development container specification +::: ### What is a Dev Container? From e16df70ddb3246840432d4a12175730b3f705ee4 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 16:58:13 +0200 Subject: [PATCH 025/107] Wording --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 8044c0981..106c96f22 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -89,7 +89,7 @@ While waiting for the initial setup, feel free to stretch your legs, grab a coff A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. :::{caution} -Ensure you trust the authors and contents of a workspaces before launching any dev containers. +Ensure you trust the authors and contents of a workspaces before launching related dev containers. ::: More info on trusting workspaces and extensions in general can be found here: From e05586d5fede26ec0ce26fb9f6dfcee5babf3547 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 17:04:52 +0200 Subject: [PATCH 026/107] Wording --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 106c96f22..b1413b1dd 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -66,7 +66,7 @@ Alternatively, you could use GitHub Codespaces directly from the project repo, o - [Creating a codespace for a repository](https://docs.github.com/en/codespaces/developing-in-codespaces/creating-a-codespace-for-a-repository?tool=webui) - How to create a codespace for repository via GitHub CLI, VS Code, or Web browser - [Develop on a remote Docker host](https://code.visualstudio.com/remote/advancedcontainers/develop-remote-host) - - How to connect VS Code to a remote Docker host using SSH or TCP sockets + - How to connect VS Code to a remote Docker host using SSH tunnels or TCP sockets ::: ## Getting started From 71b04848b50df11da8b19a1ccce8819801ccb0e8 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 17:06:59 +0200 Subject: [PATCH 027/107] Formatting --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index b1413b1dd..8dbf7d9f2 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -16,7 +16,7 @@ dev_containers_docs/development_guide.md dev_containers_docs/visualization_guide.md ``` -## What, why, how? +## What, Why, How? Lets briefly explain what dev containers are, why you should use them, and how they work. From 5ce6c6ae375c35772cbae71802ce638898664f17 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 17:48:16 +0200 Subject: [PATCH 028/107] Add section intros --- .../dev_containers_docs/development_guide.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 7463de30a..0b9a127b8 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -1,7 +1,21 @@ # Development Guide +In this guide, we'll walk through the process of creating and using dev containers for the project. While included subsections will provide greater detail on the various aspects of the process, complete comprehension of the entire guide is not required to get started, but is recommended for those interested in how dev containers work, or how to customize or optimize them for their own personal workflows. + ## Creating Dev Containers +Before creating a dev container, you'll want to choose the exact configuration to use. By default the `.devcontainer/devcontainer.json` configuration is selected, however you can also choose any other `devcontainer.json` file in the `.devcontainer/` directory, where such configurations can be nested to provide greater customization: either by targeting different stages within different Dockerfiles, overriding any merged metadata or default properties, or inclusion of additional extensions and alternate commands. + +:::{seealso} +The specification, reference, and schema for the `devcontainer.json` config file format can be found here: +- [Specification](https://containers.dev/implementors/spec) + - Development Container Specification +- [Reference](https://containers.dev/implementors/json_reference) + - Metadata and properties reference +- [Schema](https://containers.dev/implementors/json_schema) + - JSON schema for `devcontainer.json` +::: + ### Building the image When first creating Dev Containers, any supporting tool or service used will download and build the docker images needed to run the container. This includes pulling any parent images the project's Dockerfile builds `FROM`, as well as any tags or layers declared via `cacheFrom`, as specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until such layers are updated and pushed to the image registry. From d0651e105a0fff2184850c208b77a194b25f87a8 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 17:51:48 +0200 Subject: [PATCH 029/107] Wording --- dev_containers/dev_containers_docs/development_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 0b9a127b8..23adadc6b 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -1,6 +1,6 @@ # Development Guide -In this guide, we'll walk through the process of creating and using dev containers for the project. While included subsections will provide greater detail on the various aspects of the process, complete comprehension of the entire guide is not required to get started, but is recommended for those interested in how dev containers work, or how to customize or optimize them for their own personal workflows. +In this guide, we'll walk through the process of creating and using dev containers for the project. While included subsections will provide greater detail on the various aspects of the process, complete comprehension of the entire guide is not required to get started, but is recommended for those interested in how dev containers work, or how to customize and optimize them for their own personal workflows. ## Creating Dev Containers From b9c4c2e7db0f338e672745419fc97699b3f5ea7c Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 18:04:52 +0200 Subject: [PATCH 030/107] Add seealso callout for Nav2 CI --- dev_containers/dev_containers_docs/development_guide.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 23adadc6b..547e35a6e 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -24,6 +24,13 @@ Specifically, for this project, the default `devcontainer.json` file targets the To speed up the initial build, images layers from this `builder` stage are cached by pulling the same image tag used by the project's CI, hosted from the image registry. This ensures your local dev container replicates our CI environment as close as possible, while benefiting from any cached work preemptively performed by the CI. Yet, this still allows you to customize the project's Dockerfile and rebuild the container, without needing to update CI images to reflect your local modifications. +:::{seealso} +More details on the project's CI and related docker image registry can be found here: + +- [Chronicles of Caching and Containerising CI for Nav2](https://vimeo.com/649647161/5b0c278e6c) + - Video presentation from ROS World 2021 - Ruffin White +::: + Once the base image from the target stage is built, the supporting tool or service may then add additional layers to the image, such as installing additional [features](https://containers.dev/features) or customizations. For VS Code, this also includes some fancy file caching for any extensions to install later. Once this custom image is built, it is then used to start the dev container. ### Starting the container From a868f26a58c53e76fb5b72d0af1cdadb7d230c9e Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 18:14:06 +0200 Subject: [PATCH 031/107] Add hint callout for colcon extentions --- dev_containers/dev_containers_docs/development_guide.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 547e35a6e..6e664f2a8 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -55,6 +55,15 @@ While the colcon workspace is being built, VS Code will simultaneously install a - Applicable for reviewing pull requests without rebuilding entire container - Necessary for reducing startup time when spawning new Codespaces +:::{hint} +More documentation about these additional colcon verb extensions can be found here: + +- [colcon-cache](https://github.com/ruffsl/colcon-cache) + - A colcon extension to cache the processing of packages +- [colcon-clean](https://github.com/colcon/colcon-clean) + - A colcon extension to clean package workspaces +::: + Finally, the `postCreateCommand` is executed, which also reruns whenever the container is started or restarted. Specifically, for this project, this command makes a last few tweaks to the user's environment to improve the development experience. To speed up subsequent startups, volumes are mounted to the container store a persistent ccache directory, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) capabilities and connected using the [host](https://docs.docker.com/network/host/) network mode. This is especially useful for: From d15bec2da4bd48d4328c8dbe81c0eeec4d2c1bd7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 18:57:56 +0200 Subject: [PATCH 032/107] Use attention admonition about runArgs --- dev_containers/dev_containers_docs/development_guide.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 6e664f2a8..196db59b4 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -77,7 +77,9 @@ To speed up subsequent startups, volumes are mounted to the container store a pe - Applicable for ROS package using sensors and actuators - Necessary for some GPU drivers and USB devices -Note that these `runArgs` in the `devcontainer.json` config can be further customized, either expanded or or narrowed in scope, to better suit your desired development environment. The current configuration is merely the project default in order to be the most flexible and useful for the widest range of development use cases. +:::{attention} +These `runArgs` in the `devcontainer.json` config can be further customized, either expanded or or narrowed in scope, to better suit your desired development environment. The current configuration is merely the project default in order to be the most flexible and useful for the widest range of development use cases. +::: ## Using Dev Containers From 9b3deef54a34eb6ebd35e9e7dc8a5fdcff950775 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 11 Apr 2023 18:59:30 +0200 Subject: [PATCH 033/107] Update remarks on volumes to explain changes from: - https://github.com/ros-planning/navigation2/pull/3524 --- .../dev_containers_docs/development_guide.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 196db59b4..64daa8608 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -66,7 +66,18 @@ More documentation about these additional colcon verb extensions can be found he Finally, the `postCreateCommand` is executed, which also reruns whenever the container is started or restarted. Specifically, for this project, this command makes a last few tweaks to the user's environment to improve the development experience. -To speed up subsequent startups, volumes are mounted to the container store a persistent ccache directory, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) capabilities and connected using the [host](https://docs.docker.com/network/host/) network mode. This is especially useful for: +To speed up subsequent startups, volumes are mounted to the container store a persistent ccache and colcon workspace, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). These volumes are labeled using the [`devcontainerId`](https://containers.dev/implementors/json_reference/#variables-in-devcontainerjson) variable, which uniquely identify the dev container on a Docker host, allowing us refer to an common identifier that is unique to the dev container, while remaining stable across rebuilds. This is especially useful for: + +- Caching + - Enables colcon workspaces and ccache to persist between container rebuilds + - Applicable for avoiding re-compilation when modifying dev container config files + - Necessary for quickly customizing image or features without rebuilding from scratch + +:::{tip} +While these volumes are uniquely named, you may like renaming them locally to further organize or segment works-in-progress. E.g. appending branch names to the volume name to quickly switch between pull requests and cached colcon workspaces. +::: + +Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) capabilities and connected using the [host](https://docs.docker.com/network/host/) network mode. This is especially useful for: - Hybrid development - Enables connecting ROS nodes external to the container From 34098e579f58f1ca9b30ad92c255b829f0e81e5c Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 13:49:45 +0200 Subject: [PATCH 034/107] Grammar --- dev_containers/dev_containers_docs/development_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 64daa8608..9c5f37622 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -66,7 +66,7 @@ More documentation about these additional colcon verb extensions can be found he Finally, the `postCreateCommand` is executed, which also reruns whenever the container is started or restarted. Specifically, for this project, this command makes a last few tweaks to the user's environment to improve the development experience. -To speed up subsequent startups, volumes are mounted to the container store a persistent ccache and colcon workspace, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). These volumes are labeled using the [`devcontainerId`](https://containers.dev/implementors/json_reference/#variables-in-devcontainerjson) variable, which uniquely identify the dev container on a Docker host, allowing us refer to an common identifier that is unique to the dev container, while remaining stable across rebuilds. This is especially useful for: +To speed up subsequent startups, volumes that are mounted to the container store a persistent ccache and colcon workspace, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). These volumes are labeled using the [`devcontainerId`](https://containers.dev/implementors/json_reference/#variables-in-devcontainerjson) variable, which uniquely identify the dev container on a Docker host, allowing us to refer to a common identifier that is unique to the dev container, while remaining stable across rebuilds. This is especially useful for: - Caching - Enables colcon workspaces and ccache to persist between container rebuilds @@ -74,7 +74,7 @@ To speed up subsequent startups, volumes are mounted to the container store a pe - Necessary for quickly customizing image or features without rebuilding from scratch :::{tip} -While these volumes are uniquely named, you may like renaming them locally to further organize or segment works-in-progress. E.g. appending branch names to the volume name to quickly switch between pull requests and cached colcon workspaces. +While these volumes are uniquely named, you could rename them locally to further organize or segment works-in-progress. E.g. appending branch names to the volume name to quickly switch between pull requests and cached colcon workspaces. ::: Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) capabilities and connected using the [host](https://docs.docker.com/network/host/) network mode. This is especially useful for: From d37e81885c11fabb0fd594e54e3dc73c61079df3 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 14:53:43 +0200 Subject: [PATCH 035/107] Update notes on current runArgs with added links for reference --- dev_containers/dev_containers_docs/development_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 9c5f37622..c7da9b203 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -77,7 +77,7 @@ To speed up subsequent startups, volumes that are mounted to the container store While these volumes are uniquely named, you could rename them locally to further organize or segment works-in-progress. E.g. appending branch names to the volume name to quickly switch between pull requests and cached colcon workspaces. ::: -Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) capabilities and connected using the [host](https://docs.docker.com/network/host/) network mode. This is especially useful for: +Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) and non-default [Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities), connected using the [host](https://docs.docker.com/network/host/) network mode and [IPC](https://docs.docker.com/engine/reference/run/#ipc-settings---ipc) and [PID](https://docs.docker.com/engine/reference/run/#pid-settings---pid) spaces, with a relaxed [security configuration](https://docs.docker.com/engine/reference/run/#security-configuration) and seccomp confinement for native debugging and external connectivity. This is especially useful for: - Hybrid development - Enables connecting ROS nodes external to the container From 8693a3e845be5970008227b0796d8b74e355f55a Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 14:56:07 +0200 Subject: [PATCH 036/107] Add seealso callout for details on used runArgs --- .../dev_containers_docs/development_guide.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index c7da9b203..2f408a756 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -92,6 +92,17 @@ Additionally, the container is granted [privileged](https://docs.docker.com/engi These `runArgs` in the `devcontainer.json` config can be further customized, either expanded or or narrowed in scope, to better suit your desired development environment. The current configuration is merely the project default in order to be the most flexible and useful for the widest range of development use cases. ::: +:::{seealso} +More details on using DDS, debuggers, or devices with Docker containers can be found here: + +- [How to Communicate Across Docker Containers Using the Host Driver](https://community.rti.com/kb/how-use-rti-connext-dds-communicate-across-docker-containers-using-host-driver) + - Using the `host` network driver to access all network interfaces of the host machine from the Docker container +- [Communicate between two Docker containers using DDS and shared memory](https://community.rti.com/kb/communicate-between-two-docker-containers-using-rti-connext-dds-and-shared-memory) + - Enabling containers to communicate with one another and with the host machine using interprocess communication (IPC) +- [Debugging programs running inside Docker containers, in production](https://nvartolomei.com/debugging-programs-running-inside-docker-containers--in-production/) + - Using tools like strace, perf, gdb when debugging programs running inside containers +::: + ## Using Dev Containers Once the dev container has been created and setup completed, VS Code will open a new workspace directly from the project's root directory, which itself is mounted within the source directory in the overlay colcon workspace. From here you can build, test, and debug the project as you normally would, with the added benefit of having the project's dependencies, intellisense, linters, and other extensions pre-configured and ready to use. From eced357b9835c3f17f84cd92171ddf8aded1b46a Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 15:01:26 +0200 Subject: [PATCH 037/107] Update remark about volumes --- dev_containers/dev_containers_docs/development_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 2f408a756..02dd763a1 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -146,4 +146,4 @@ If necessary, you can also rebuild the container from scratch, e.i. without cach - to forcefully rerun a `RUN` directive in the Dockerfile - such as unchanged `apt upgrade` or `rosdep update` commands -Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory. The management of these volumes is left for the developers discretion, and can be done via the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/), or the [VS Code Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker). However, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. +Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory or colcon workspace. The management of these volumes is left for the developers discretion, and can be done via the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/), or the [VS Code Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker). However, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. From e20e5223a413ca36a8b22ea0e7e16513a5fe822a Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 15:42:05 +0200 Subject: [PATCH 038/107] Formatting --- conf.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/conf.py b/conf.py index 70eab0c73..9b68dc7ea 100644 --- a/conf.py +++ b/conf.py @@ -30,7 +30,13 @@ # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['breathe', 'myst_parser', 'sphinx.ext.graphviz', 'sphinxcontrib.plantuml', 'sphinx.ext.extlinks'] +extensions = [ + 'breathe', + 'myst_parser', + 'sphinx.ext.graphviz', + 'sphinxcontrib.plantuml', + 'sphinx.ext.extlinks', +] myst_enable_extensions = ['colon_fence'] From 63dd66b45409d81c6a0f77cb8ba213e6bccb6cd9 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 15:42:46 +0200 Subject: [PATCH 039/107] Sort Formatting --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index 9b68dc7ea..5b4209eb4 100644 --- a/conf.py +++ b/conf.py @@ -33,9 +33,9 @@ extensions = [ 'breathe', 'myst_parser', + 'sphinx.ext.extlinks', 'sphinx.ext.graphviz', 'sphinxcontrib.plantuml', - 'sphinx.ext.extlinks', ] myst_enable_extensions = ['colon_fence'] From 59e50f4b50eb637cd05021930ecab583127685ca Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 15:47:27 +0200 Subject: [PATCH 040/107] Install sphinx_copybutton for handy copybutton - https://sphinx-copybutton.readthedocs.io/ --- .circleci/config.yml | 2 +- README.md | 6 +++--- conf.py | 1 + scripts/docker/Dockerfile | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index e1a5695bc..2cfbcc4b4 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -17,7 +17,7 @@ references: apt install openjdk-8-jre -y apt install bash git openssh-server -y apt install python3-pip python3 -y - pip3 install sphinx==3.5.0 docutils==0.14 sphinx_rtd_theme breathe==4.28.0 sphinxcontrib-plantuml jinja2==3.0.3 myst-parser + pip3 install sphinx==3.5.0 docutils==0.14 sphinx_rtd_theme breathe==4.28.0 sphinxcontrib-plantuml jinja2==3.0.3 myst-parser sphinx_copybutton make_docs: &make_docs run: command: | diff --git a/README.md b/README.md index 4add7f786..eb608dd33 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,10 @@ https://navigation.ros.org/ This folder holds the source and configuration files used to generate the [Navigation2 documentation](https://navigation.ros.org) web site. -Dependencies for Build: +Dependencies for Build: * `sudo apt install python3-pip` -* `pip3 install sphinx==3.5.0 breathe==4.28.0 sphinx_rtd_theme sphinxcontrib-plantuml jinja2==3.0.3 myst-parser` +* `pip3 install sphinx==3.5.0 breathe==4.28.0 sphinx_rtd_theme sphinxcontrib-plantuml jinja2==3.0.3 myst-parser sphinx_copybutton` Build the docs locally with `make html` and you'll find the built docs entry point in `_build/html/index.html`. -Any images, diagrams, or videos are subject to their own copyrights, trademarks, and licenses. +Any images, diagrams, or videos are subject to their own copyrights, trademarks, and licenses. diff --git a/conf.py b/conf.py index 5b4209eb4..091f74a30 100644 --- a/conf.py +++ b/conf.py @@ -33,6 +33,7 @@ extensions = [ 'breathe', 'myst_parser', + 'sphinx_copybutton', 'sphinx.ext.extlinks', 'sphinx.ext.graphviz', 'sphinxcontrib.plantuml', diff --git a/scripts/docker/Dockerfile b/scripts/docker/Dockerfile index 9b6747f68..b1d5c13a2 100644 --- a/scripts/docker/Dockerfile +++ b/scripts/docker/Dockerfile @@ -5,7 +5,7 @@ FROM alpine:latest RUN apk --no-cache add python3 RUN python3 -m ensurepip -RUN pip3 install sphinx==1.7.5 docutils==0.14 sphinx_rtd_theme breathe==4.9.1 sphinxcontrib-plantuml +RUN pip3 install sphinx==1.7.5 docutils==0.14 sphinx_rtd_theme breathe==4.9.1 sphinxcontrib-plantuml sphinx_copybutton RUN apk --no-cache add make RUN apk --no-cache add doxygen RUN apk --no-cache add graphviz From 0ae0ca780acf25060fe6b8e66f4d1291cbbd290e Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 15:51:24 +0200 Subject: [PATCH 041/107] Simplify install instructions --- build_instructions/index.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index 46e9f657a..211ff2dcc 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -11,11 +11,6 @@ You may install it via the following to get the latest stable released version: ``sudo apt install ros--navigation2 ros--nav2-bringup ros--turtlebot3*`` -(For Ubuntu 20.04 use this command as the parsing of wildcards have been changed: - - ``sudo apt install ros--navigation2 ros--nav2-bringup '~ros--turtlebot3-.*'`` - - Build ***** From 6d98af1a5cf0f922e759fbea7728d687166d824c Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 15:53:42 +0200 Subject: [PATCH 042/107] Format install instructions --- build_instructions/index.rst | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index 211ff2dcc..2a60998bd 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -9,7 +9,13 @@ Install Nav2 and its dependencies are released as binaries. You may install it via the following to get the latest stable released version: - ``sudo apt install ros--navigation2 ros--nav2-bringup ros--turtlebot3*`` +.. code:: bash + + source /opt/ros//setup.bash + sudo apt install \ + ros-$ROS_DISTRO-navigation2 \ + ros-$ROS_DISTRO-nav2-bringup \ + ros-$ROS_DISTRO-turtlebot3* Build ***** @@ -62,7 +68,7 @@ All development is done using the ``rolling`` distribution on Nav2's ``main`` br Build Nav2 Main --------------- -Now that ROS 2 ``rolling`` is installed, we have to install our dependencies and build Nav2 itself. +Now that ROS 2 ``rolling`` is installed, we have to install our dependencies and build Nav2 itself. We'll create a new workspace, ``nav2_ws`` and clone the Nav2 project into it. Afterwards, we'll use ``rosdep`` to automatically find and install our dependencies that were not included in the core ROS 2 install itself (``behaviortree.CPP``, ``ompl``, etc). If you would like to use a custom version of any of these dependencies, simply overlay them in your ``nav2_ws`` and it will use those rather than the binary installed versions. From e471b5eb2742b7885f1bf0cbdb217636a880b567 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 19:30:24 +0200 Subject: [PATCH 043/107] Wording --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 8dbf7d9f2..a3b09901d 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -1,6 +1,6 @@ # Dev Containers -Alternatively you can use dev containers to build the project if you prefer a streamlined setup experience. This means you can use the same tools and dependencies as the rest of the team, including our Continuous Integration (CI) workflows, without worrying about installing dependencies on your host machine. Additionally, using Dev Containers makes it simple to switch between local or remote development environments, such as GitHub Codespaces. More info on Dev Containers can be found here: +You can use dev containers to build the project if you prefer a streamlined setup experience. This means you can use the same tools and dependencies as the rest of the team, including our Continuous Integration (CI) workflows, without worrying about installing dependencies on your host machine. Additionally, using Dev Containers makes it simple to switch between local or remote development environments, such as GitHub Codespaces. More info on Dev Containers can be found here: - [Development Containers](https://containers.dev/) - An open specification for enriching containers with development specific content and settings From 898377da09262ac5124eb189b65b4b808d2fd017 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 19:32:08 +0200 Subject: [PATCH 044/107] Add missing annotating in order to reference it elsewhare - https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html --- dev_containers/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/dev_containers/index.md b/dev_containers/index.md index a3b09901d..32519380f 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -1,3 +1,4 @@ +(dev-containers)= # Dev Containers You can use dev containers to build the project if you prefer a streamlined setup experience. This means you can use the same tools and dependencies as the rest of the team, including our Continuous Integration (CI) workflows, without worrying about installing dependencies on your host machine. Additionally, using Dev Containers makes it simple to switch between local or remote development environments, such as GitHub Codespaces. More info on Dev Containers can be found here: From 1c3996f3e1908900e7f75b061cd914b82d987c0d Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 22:56:05 +0200 Subject: [PATCH 045/107] Update build instructions --- build_instructions/index.rst | 97 ++++++++++++++++++++---------------- 1 file changed, 55 insertions(+), 42 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index 2a60998bd..0442b0e86 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -20,70 +20,83 @@ You may install it via the following to get the latest stable released version: Build ***** -There are 3 ways to build Nav2. -Building for a specific released distribution (e.g. ``foxy``, ``galactic``), build Nav2 on main branch using a quickstart setup script, or building main branch manually. +There are a few ways to build Nav2: + +* Using Released Distribution Binaries +* Using Rolling Development Source + +.. tip:: + For a *repeatable*, *reproducible* and *streamlined* development experience, check the Nav2 documentation on using :ref:`dev-containers`! .. rst-class:: content-collapse -For Released Distributions -========================== +Released Distribution Binaries +============================== -Install ROS ------------ +To build Nav2, you'll first need to build or install ROS 2 and related development tools, including: ``colcon``, ``rosdep`` and ``vcstool``. -Please install ROS 2 via the usual `install instructions `_ for your desired distribution. -Note: ``-devel`` was the branch naming schema pre-``galactic``. -For Galactic and newer, it is simply ````. +.. seealso:: + For more information on building or installing ROS 2 distros, see the official documentation: -Build Nav2 ----------- + * `ROS 2 Installation `_ + * `Install development tools and ROS tools `_ -We're going to create a new workspace, ``nav2_ws``, clone our Nav2 branch into it, and build. -``rosdep`` will be used to get the dependency binaries for Nav2 in your specific distribution. +Once your environment is setup, clone the repo, install all dependencies, and build the workspace: -.. code:: bash +.. attention:: + For ROS 2 distros prior to ``galactic``, the branch naming schema was ``-devel``. - mkdir -p ~/nav2_ws/src - cd ~/nav2_ws/src - git clone https://github.com/ros-planning/navigation2.git --branch -devel - cd ~/nav2_ws - rosdep install -y -r -q --from-paths src --ignore-src --rosdistro - colcon build --symlink-install +.. code:: bash -Note: You need to change ``--rosdistro`` to the selected ROS 2 distribution name (e.g ``foxy``, ``galactic``). + source /opt/ros//setup.bash + export NAV2_WS=~/nav2_ws + mkdir -p $NAV2_WS/src && cd $NAV2_WS + git clone https://github.com/ros-planning/navigation2.git --branch $ROS_DISTRO ./src/navigation2 + rosdep install -y \ + --from-paths ./src \ + --ignore-src + colcon build \ + --symlink-install +You can then ``source $NAV2_WS/install/setup.bash`` to get ready for demonstrations! +.. hint:: + For more examples on building Nav2 from released distribution binaries, checkout `distro.Dockerfile `_. .. rst-class:: content-collapse +Using Rolling Development Source +================================ -For Main Branch Development -=========================== +Building Nav2 using rolling development source is similar to building Nav2 from released distribution binaries, where instead you build dependencies from source using the main development branches for all ROS based packages. -Build ROS 2 Main ----------------- -Build or install ROS 2 ``rolling`` using the `build instructions `_ provided in the ROS 2 documentation. -All development is done using the ``rolling`` distribution on Nav2's ``main`` branch and cherry-picked over to released distributions during syncs (if ABI compatible). +.. seealso:: + For more information on building ROS 2 from source, see the official documentation: -Build Nav2 Main ---------------- + * `ROS 2 Building from source `_ -Now that ROS 2 ``rolling`` is installed, we have to install our dependencies and build Nav2 itself. -We'll create a new workspace, ``nav2_ws`` and clone the Nav2 project into it. -Afterwards, we'll use ``rosdep`` to automatically find and install our dependencies that were not included in the core ROS 2 install itself (``behaviortree.CPP``, ``ompl``, etc). -If you would like to use a custom version of any of these dependencies, simply overlay them in your ``nav2_ws`` and it will use those rather than the binary installed versions. +Once your environment is setup, clone the repo, import all dependencies, and build the workspace: -.. code:: bash +.. attention:: + Be sure to check that all dependencies you need are included and uncommented in the ``.repos`` file. - mkdir -p ~/nav2_ws/src - cd ~/nav2_ws/src - git clone https://github.com/ros-planning/navigation2.git --branch main - cd ~/nav2_ws - rosdep install -y -r -q --from-paths src --ignore-src --rosdistro rolling - colcon build --symlink-install - source install/setup.bash +.. code:: bash -You are now ready for the demonstrations! + source /install/setup.bash + export NAV2_WS=~/nav2_ws + mkdir -p $NAV2_WS/src && cd $NAV2_WS + git clone https://github.com/ros-planning/navigation2.git --branch $ROS_DISTRO ./src/navigation2 + vcs import ./src < ./src/navigation2/tools/underlay.repos + rosdep install -y \ + --from-paths ./src \ + --ignore-src + colcon build \ + --symlink-install + +You can then ``source $NAV2_WS/install/setup.bash`` to get ready for demonstrations! + +.. hint:: + For more examples on building Nav2 from rolling development source, checkout `source.Dockerfile `_. Docker ****** From 9edf13b8bdb9f96d29d8236fea0554954982f526 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 22:56:47 +0200 Subject: [PATCH 046/107] Update docker instructions --- build_instructions/index.rst | 53 +++++++++++++----------------------- 1 file changed, 19 insertions(+), 34 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index 0442b0e86..febc33015 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -24,6 +24,7 @@ There are a few ways to build Nav2: * Using Released Distribution Binaries * Using Rolling Development Source +* Using Docker Container Images .. tip:: For a *repeatable*, *reproducible* and *streamlined* development experience, check the Nav2 documentation on using :ref:`dev-containers`! @@ -98,51 +99,35 @@ You can then ``source $NAV2_WS/install/setup.bash`` to get ready for demonstrati .. hint:: For more examples on building Nav2 from rolling development source, checkout `source.Dockerfile `_. -Docker -****** - -The official Dockerhub entries are primarily for use in the Nav2 CI, but they may also be used for development. It is useful to have a docker image that tracks Nav2 ``main`` branch. The ``Dockerfile`` in the root of the repository is recommended for production use, set to your distribution of choice. - -It is though generally recomended to install Nav2 releases from the apt repository inside a container if you'd like to use our released binaries. - .. rst-class:: content-collapse -Building Docker Container -========================= - -To build an image from the Dockerfile in the Nav2 folder: -First, clone the repo to your local system (or see Building the source above) - - -.. code:: bash - - sudo docker build -t nav2/latest . +Using Docker Container Images +============================= -If proxies are needed: +Building Nav2 using Docker container images provides a repeatable and reproducible environment to automate and self document the entire setup process. Instead of manually invoking the development tools as documented above, you can leverage the project's Dockerfiles to build and install Nav2 for various distributions. -.. code:: bash - - sudo docker build -t nav2/latest --build-arg http_proxy=http://proxy.my.com:### --build-arg https_proxy=http://proxy.my.com:### . +.. seealso:: + For more information on installing Docker or leaning about Dockerfiles, see the official documentation: -Note: You may also need to configure your docker for DNS to work. See article here for details: https://development.robinwinslow.uk/2016/06/23/fix-docker-networking-dns/ + * `Docker Engine `_ + * `Dockerfile reference `_ -If you would like to build from dockerhub cache to speed up the build +Once your system is setup, you can build the Nav2 Dockerfile from the root of the repo: .. code:: bash - sudo docker pull rosplanning/navigation2:main - sudo docker build -t nav2/latest --cache-from rosplanning/navigation2:main . - -.. rst-class:: content-collapse - -Using DockerHub Container -========================= -We allow for you to pull the latest docker image from the main branch at any time. As new releases and tags are made, docker containers on docker hub will be versioned as well to chose from. -This docker image will not contain a built overlay, and you must build the overlay Nav2 workspace yourself (see Build Nav2 Main up above). + export ROS_DISTRO=rolling + git clone https://github.com/ros-planning/navigation2.git --branch $ROS_DISTRO + docker build --tag navigation2:$ROS_DISTRO \ + --build-arg FROM_IMAGE=ros:$ROS_DISTRO \ + --build-arg OVERLAY_MIXINS="release ccache lld" \ + --cache-from ghcr.io/ros-planning/navigation2:$ROS_DISTRO \ + ./navigation2 -.. code:: bash +The `docker build `_ command above creates a tagged image using the `Dockerfile` from the context specified using the path to the repo, where build-time variables are set using additional arguments, e.g. passing a set of `colcon mixins `_ to configure the workspace build. Check the ``ARG`` directives in the `Dockerfile` to discover other build-time variables available. - sudo docker pull rosplanning/navigation2:main +.. tip:: + The images cached from are used for Nav2 CI, but can also be used with Nav2 :ref:`dev-containers`! !!!! From 2fd7f7f4deb06c65871346b718a3fa193d453d6d Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 23:06:05 +0200 Subject: [PATCH 047/107] Default branch is sadly not named rolling would be consistently convenient if it was --- build_instructions/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index febc33015..eebac4d65 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -117,11 +117,11 @@ Once your system is setup, you can build the Nav2 Dockerfile from the root of th .. code:: bash export ROS_DISTRO=rolling - git clone https://github.com/ros-planning/navigation2.git --branch $ROS_DISTRO + git clone https://github.com/ros-planning/navigation2.git --branch main docker build --tag navigation2:$ROS_DISTRO \ --build-arg FROM_IMAGE=ros:$ROS_DISTRO \ --build-arg OVERLAY_MIXINS="release ccache lld" \ - --cache-from ghcr.io/ros-planning/navigation2:$ROS_DISTRO \ + --cache-from ghcr.io/ros-planning/navigation2:main \ ./navigation2 The `docker build `_ command above creates a tagged image using the `Dockerfile` from the context specified using the path to the repo, where build-time variables are set using additional arguments, e.g. passing a set of `colcon mixins `_ to configure the workspace build. Check the ``ARG`` directives in the `Dockerfile` to discover other build-time variables available. From a7362a9a194e87368960bf7a24b5ad7d35fc70c7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 23:44:57 +0200 Subject: [PATCH 048/107] Add descriptions for each build method --- build_instructions/index.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index eebac4d65..e2e073a0c 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -22,10 +22,18 @@ Build There are a few ways to build Nav2: -* Using Released Distribution Binaries +* Using Released Distribution Binaries + + * Simply build Nav2 using installable binary dependencies + * Using Rolling Development Source + + * Intricately build Nav2 using from latest source dependencies + * Using Docker Container Images + * Quickly build Nav2 using cached images and templated Dockerfiles + .. tip:: For a *repeatable*, *reproducible* and *streamlined* development experience, check the Nav2 documentation on using :ref:`dev-containers`! From 23c44ad8853a6caafd84bfda5b8f9aaf4feedd68 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 12 Apr 2023 23:50:01 +0200 Subject: [PATCH 049/107] Wording --- build_instructions/index.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index e2e073a0c..043643039 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -20,17 +20,17 @@ You may install it via the following to get the latest stable released version: Build ***** -There are a few ways to build Nav2: +There are a few ways to build Nav2 using: -* Using Released Distribution Binaries +* Released Distribution Binaries * Simply build Nav2 using installable binary dependencies -* Using Rolling Development Source +* Rolling Development Source * Intricately build Nav2 using from latest source dependencies -* Using Docker Container Images +* Docker Container Images * Quickly build Nav2 using cached images and templated Dockerfiles @@ -74,8 +74,8 @@ You can then ``source $NAV2_WS/install/setup.bash`` to get ready for demonstrati .. rst-class:: content-collapse -Using Rolling Development Source -================================ +Rolling Development Source +========================== Building Nav2 using rolling development source is similar to building Nav2 from released distribution binaries, where instead you build dependencies from source using the main development branches for all ROS based packages. @@ -109,8 +109,8 @@ You can then ``source $NAV2_WS/install/setup.bash`` to get ready for demonstrati .. rst-class:: content-collapse -Using Docker Container Images -============================= +Docker Container Images +======================= Building Nav2 using Docker container images provides a repeatable and reproducible environment to automate and self document the entire setup process. Instead of manually invoking the development tools as documented above, you can leverage the project's Dockerfiles to build and install Nav2 for various distributions. From a0775f2b6f9dd97b7f0722f0f8ea5760f03af9fd Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 00:08:34 +0200 Subject: [PATCH 050/107] Expand docker docs with links --- build_instructions/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index 043643039..01c06d5e0 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -132,10 +132,10 @@ Once your system is setup, you can build the Nav2 Dockerfile from the root of th --cache-from ghcr.io/ros-planning/navigation2:main \ ./navigation2 -The `docker build `_ command above creates a tagged image using the `Dockerfile` from the context specified using the path to the repo, where build-time variables are set using additional arguments, e.g. passing a set of `colcon mixins `_ to configure the workspace build. Check the ``ARG`` directives in the `Dockerfile` to discover other build-time variables available. +The `docker build `_ command above creates a tagged image using the `Dockerfile` from the context specified using the path to the repo, where build-time variables are set using additional arguments, e.g. passing a set of `colcon mixins `_ to configure the workspace build. Check the ``ARG`` directives in the `Dockerfile` to discover all build-time variables available. The command also specifies an `external cache source `_ to pull the latest cached image from Nav2's `Container Registry `_ to speed up the build process. .. tip:: - The images cached from are used for Nav2 CI, but can also be used with Nav2 :ref:`dev-containers`! + The images cached from above are used for Nav2 CI, but can also be used with Nav2 :ref:`dev-containers`! !!!! From e14625390d55fca6cfc9c63ed81ee8b739c7fda7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 11:22:33 +0200 Subject: [PATCH 051/107] Formatting --- build_instructions/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index 01c06d5e0..565f41f9f 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -22,9 +22,9 @@ Build There are a few ways to build Nav2 using: -* Released Distribution Binaries +* Released Distribution Binaries - * Simply build Nav2 using installable binary dependencies + * Simply build Nav2 using installable binary dependencies * Rolling Development Source From 4fa977b0240241f9b5dfcf73d9ffca2c0beb690d Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 11:25:54 +0200 Subject: [PATCH 052/107] Add note that rolling is tracked via main branch --- build_instructions/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index 565f41f9f..952994de7 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -53,7 +53,7 @@ To build Nav2, you'll first need to build or install ROS 2 and related developme Once your environment is setup, clone the repo, install all dependencies, and build the workspace: .. attention:: - For ROS 2 distros prior to ``galactic``, the branch naming schema was ``-devel``. + For ROS 2 distros prior to Galactic, the branch naming schema was ``-devel``, while the default branch for Rolling is ``main``. .. code:: bash From 7f645c9dd445834acd49e29367e35907a5858615 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:48:27 +0200 Subject: [PATCH 053/107] Add tip for development workflow --- dev_containers/dev_containers_docs/development_guide.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 02dd763a1..d91cb661a 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -105,9 +105,11 @@ More details on using DDS, debuggers, or devices with Docker containers can be f ## Using Dev Containers -Once the dev container has been created and setup completed, VS Code will open a new workspace directly from the project's root directory, which itself is mounted within the source directory in the overlay colcon workspace. From here you can build, test, and debug the project as you normally would, with the added benefit of having the project's dependencies, intellisense, linters, and other extensions pre-configured and ready to use. +Once the dev container has been created and setup completed, VS Code will open a new workspace directly from the project's root directory, which itself is mounted within the source directory in the overlay colcon workspace. From here you can build, test, and debug the project as you normally would, with the added benefit of having the project's dependencies, intellisense, linters, and other extensions pre-configured and ready to use. Simply open a new terminal (Crtl+Shift+`), cd to the root of the colcon workspace, and run the usual colcon commands. -So to build or test the project, simply open a terminal, cd to the root of the colcon workspace, and run the usual colcon commands. You may also want to incorporate the same colcon verbs used by the setup commands from the `devcontainer.json` config file to further automate your local development workflow. +:::{tip} +You can incorporate the same scripts used by the `devcontainer.json` config file to further automate your local development workflow. +::: ### Terminals From 276a2fabb4649c491fb7021709aab3c0a12ee4b7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:49:28 +0200 Subject: [PATCH 054/107] Add attention to terminals --- dev_containers/dev_containers_docs/development_guide.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index d91cb661a..3e834c396 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -113,14 +113,18 @@ You can incorporate the same scripts used by the `devcontainer.json` config file ### Terminals -If you prefer using alternate terminal emulators, rather than the built-in VS Code terminal, you can also open a separate shell session directly using devcontainer CLI, or simply via docker exec. Note that new shell sessions started via `exec` will not automatically inherit the same environment setup by the `postCreateCommand` from the `devcontainer.json` config file. So you may need to manually source any necessary scripts, such as `install/setup.bash` from the underlay workspace. +If you prefer using an alternate terminal emulator, rather than the built-in VS Code terminal, you can open a separate shell session by simply using the Dev Container CLI or directly using the Docker CLI via the `exec` subcommands. - [Dev Container CLI](https://code.visualstudio.com/docs/devcontainers/devcontainer-cli) - - `devcontainer exec --workspace-folder bash` + - `devcontainer exec --workspace-folder $NAV2_WS/src/navigation2 bash` - [docker exec ](https://docs.docker.com/engine/reference/commandline/exec/) - `docker exec -it bash` +:::{attention} +Shell sessions spawned directly via `docker exec` do not set the same environment that `devcontainer exec` does using `userEnvProbe`. Additional environment variables include `REMOTE_CONTAINERS_IPC`, `REMOTE_CONTAINERS_SOCKETS` and are used by vscode, ssh and X11. +::: + ### Lifecycle While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces, so be sure to commit and push local changes to a remote repository: From 646bf961beb93fbb6ff0b394ef9429bdf38e9297 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:51:07 +0200 Subject: [PATCH 055/107] Add hint to terminals using the dropdown class as it isn't too significant --- .../dev_containers_docs/development_guide.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 3e834c396..d5e414497 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -125,6 +125,17 @@ If you prefer using an alternate terminal emulator, rather than the built-in VS Shell sessions spawned directly via `docker exec` do not set the same environment that `devcontainer exec` does using `userEnvProbe`. Additional environment variables include `REMOTE_CONTAINERS_IPC`, `REMOTE_CONTAINERS_SOCKETS` and are used by vscode, ssh and X11. ::: +:::{hint} +:class: dropdown +The environment provided by `userEnvProbe` can be sourced manually. E.g. for the default `loginInteractiveShell` probe: + +``` bash +find /tmp -type f -path "*/devcontainers-*/env-loginInteractiveShell.json" -exec \ + jq -r 'to_entries | .[] | "\(.key)=\(.value | @sh)"' {} \; > .env +source .env +``` +::: + ### Lifecycle While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces, so be sure to commit and push local changes to a remote repository: From bdd35eec4a0137b58f2e10268f620f980702803a Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:52:01 +0200 Subject: [PATCH 056/107] Add important to lifecycle --- dev_containers/dev_containers_docs/development_guide.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index d5e414497..05ec625d3 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -138,10 +138,14 @@ source .env ### Lifecycle -While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces, so be sure to commit and push local changes to a remote repository: +While using the dev container, try and keep in mind the lifecycle of the container itself. Specifically, containers are ephemeral, meaning they are normally destroyed and recreated whenever the dev environment is rebuilt or updated. Subsequently, a best practice is to avoid storing any persistent data within the container, and instead utilize the project's source directory, or a separate mounted volume. When altering the development environment inside the container, try to remember to codify your changes into the Dockerfile, or the `devcontainer.json` config file, so that they can be easily reproduced and shared with others. + +:::{important} +This is particularly important when the host machine is inherently ephemeral as well, as the case may be when using cloud based environments such as Codespaces, so be sure to commit and push local changes to a remote repository: - [The codespace lifecycle](https://docs.github.com/en/codespaces/getting-started/the-codespace-lifecycle) - Maintain your data throughout the entire codespace lifecycle +::: ### Rebuilding From 0545b81f4ce113e9a0fabf1f2800c6ffc347eece Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:55:10 +0200 Subject: [PATCH 057/107] Add caution to rebuilding --- dev_containers/dev_containers_docs/development_guide.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 05ec625d3..a058a6942 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -149,7 +149,13 @@ This is particularly important when the host machine is inherently ephemeral as ### Rebuilding -From time to time, you may need to rebuild the dev container, either because the base image, or `.devcontainer/` config was updated, or simply out of wanting a new fresh development environment. To do so, simply open the Command Palette (Ctrl+Shift+P) and select the `Remote-Containers: Rebuild Container` command. For example, you may need to rebuild the dev container when: +From time to time, you may need to rebuild the dev container, either because the base image, or `.devcontainer/` config was updated, or simply out of wanting a new fresh development environment. To do so, simply open the Command Palette (Ctrl+Shift+P) and select the `Remote-Containers: Rebuild Container` command. + +:::{caution} +Rebuilding the container will destroy any changes made to the container itself, such as installing additional packages, or modifying the environment. However, the project's source directory, and any mounted volumes, will remain unaffected. +::: + +For example, you may need to rebuild the dev container when: - Pulling newer images from a container registry - specifically, image tags built `FROM` in the Dockerfile From 2f4eaf63a916f6439567800cea40304398f2a4d8 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:56:21 +0200 Subject: [PATCH 058/107] Add details to rebuilding without cache --- dev_containers/dev_containers_docs/development_guide.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index a058a6942..75d2f0e0f 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -166,7 +166,9 @@ For example, you may need to rebuild the dev container when: - or when modifying `./devcontainer` files and commands - where build cache reuse correlates with severity of changes made -If necessary, you can also rebuild the container from scratch, e.i. without caching from docker, by selecting the `Remote-Containers: Rebuild Container Without Cache` command instead. Rebuilding without caching may be necessary when: +When necessary, you can also rebuild the container from scratch, e.i. without caching from docker, by selecting the `Remote-Containers: Rebuild Container Without Cache` command. This instead omits the `--cache-from` flag from the `docker buildx` command, while also adding the `--no-cache` and `--pull` flags to prevent caching from any existing image layers, using only the latest images from a container registry. + +Rebuilding without caching may be necessary when: - Needing to update the base image - specifically if dev container configurations remain unmodified From 801ab3d886f004ed64b3514a002b03993dc9ffda Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:56:53 +0200 Subject: [PATCH 059/107] Add caution to rebuilding without cache --- dev_containers/dev_containers_docs/development_guide.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 75d2f0e0f..8bebcc440 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -168,6 +168,10 @@ For example, you may need to rebuild the dev container when: When necessary, you can also rebuild the container from scratch, e.i. without caching from docker, by selecting the `Remote-Containers: Rebuild Container Without Cache` command. This instead omits the `--cache-from` flag from the `docker buildx` command, while also adding the `--no-cache` and `--pull` flags to prevent caching from any existing image layers, using only the latest images from a container registry. +:::{caution} +Rebuilding the container without cache may likely pull newer images from a container registry or install newer packages, as is common when developing for ROS 2 Rolling. You may then want to clean your overlay volume to avoid ABI incompatibilities or stale artifacts. +::: + Rebuilding without caching may be necessary when: - Needing to update the base image From dd3add4db05376c025fa008b7c3aaf584c6c1065 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:57:39 +0200 Subject: [PATCH 060/107] Add tip to rebuilding --- .../dev_containers_docs/development_guide.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 8bebcc440..0dc428457 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -179,4 +179,13 @@ Rebuilding without caching may be necessary when: - to forcefully rerun a `RUN` directive in the Dockerfile - such as unchanged `apt upgrade` or `rosdep update` commands -Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory or colcon workspace. The management of these volumes is left for the developers discretion, and can be done via the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/), or the [VS Code Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker). However, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. +Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory or colcon workspace. While the management of these volumes is left up the developers discretion, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. + +:::{tip} +Docker volume management can be done via the Docker CLI, or the VS Code Docker extension: + +* [Docker Volume CLI](https://docs.docker.com/engine/reference/commandline/volume) + * Manage volumes using subcommands to create, inspect, list, remove, or prune volumes +* [VS Code Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker) + * Makes it easy to create, manage, and debug containerized applications +::: From e298804dadd91f70788967613113c0c4523edf8b Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 15:58:42 +0200 Subject: [PATCH 061/107] Split Visualization Guide into separate PR --- dev_containers/dev_containers_docs/visualization_guide.md | 1 - dev_containers/index.md | 2 -- 2 files changed, 3 deletions(-) delete mode 100644 dev_containers/dev_containers_docs/visualization_guide.md diff --git a/dev_containers/dev_containers_docs/visualization_guide.md b/dev_containers/dev_containers_docs/visualization_guide.md deleted file mode 100644 index 7b676fca3..000000000 --- a/dev_containers/dev_containers_docs/visualization_guide.md +++ /dev/null @@ -1 +0,0 @@ -# Visualization Guide diff --git a/dev_containers/index.md b/dev_containers/index.md index 32519380f..5000d9b22 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -82,8 +82,6 @@ While waiting for the initial setup, feel free to stretch your legs, grab a coff - **[](dev_containers_docs/development_guide.md)** - How to develop Nav2 using dev containers and supporting tools -- **[](dev_containers_docs/visualization_guide.md)** - - How to visualize Nav2 using local or remote headless containers ## Security From 113d8b6c05a2f58c216b582992a99d0c5fd75e1e Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 16:05:09 +0200 Subject: [PATCH 062/107] Use seealso admonition --- dev_containers/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dev_containers/index.md b/dev_containers/index.md index 5000d9b22..44dcbb16a 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -91,7 +91,9 @@ A word of caution when using dev containers: they are powerful tools, but can be Ensure you trust the authors and contents of a workspaces before launching related dev containers. ::: +:::{seealso} More info on trusting workspaces and extensions in general can be found here: - [Workspace Trust](https://code.visualstudio.com/docs/editor/workspace-trust) - VS Code user guid on trusting and configure workspaces +::: From a9159c88fc99d1584512b1cddaae31d394b789da Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 16:06:11 +0200 Subject: [PATCH 063/107] Upgrade security caution to danger to stand out using the color red --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 44dcbb16a..586f7fadd 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -87,7 +87,7 @@ While waiting for the initial setup, feel free to stretch your legs, grab a coff A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. -:::{caution} +:::{danger} Ensure you trust the authors and contents of a workspaces before launching related dev containers. ::: From c1c01bb5e0c1bddcaeb65747ae64a27b886506ec Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 13 Apr 2023 16:08:17 +0200 Subject: [PATCH 064/107] Wording --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index 586f7fadd..d5e153d41 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -88,7 +88,7 @@ While waiting for the initial setup, feel free to stretch your legs, grab a coff A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. :::{danger} -Ensure you trust the authors and contents of a workspaces before launching related dev containers. +Ensure you trust the authors and contents of workspaces before launching derived dev containers. ::: :::{seealso} From 2ab52643b24727536347f2921c15b240706a1c50 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Fri, 14 Apr 2023 10:53:46 +0200 Subject: [PATCH 065/107] Wording --- dev_containers/dev_containers_docs/development_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 0dc428457..3e064607b 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -179,7 +179,7 @@ Rebuilding without caching may be necessary when: - to forcefully rerun a `RUN` directive in the Dockerfile - such as unchanged `apt upgrade` or `rosdep update` commands -Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory or colcon workspace. While the management of these volumes is left up the developers discretion, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. +Specifically, for this project, volumes remain unaffected by this rebuilding process: i.e. those used to mount the ccache directory or colcon workspace. While volume management is left to the user's discretion, other projects may of course handle this differently, so be sure to check the `./devcontainer` configuration to inspect how various container resources may be managed. :::{tip} Docker volume management can be done via the Docker CLI, or the VS Code Docker extension: From aee86e71c6c95ba9ec58e48def2c98e24ef54b2d Mon Sep 17 00:00:00 2001 From: ruffsl Date: Fri, 14 Apr 2023 12:15:51 +0200 Subject: [PATCH 066/107] Add tip about logfile the exact command name may change, so just hint about namespace instead - https://github.com/microsoft/vscode-remote-release/issues/8018 --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index d5e153d41..e09b3351b 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -75,7 +75,7 @@ Alternatively, you could use GitHub Codespaces directly from the project repo, o Getting started using dev containers is as simple as opening the project in VS Code by either: following the notification prompt to reopen the project in a container, or explicitly opening the command palette (Crtl+Shift+P) and selecting `Remote-Containers: Reopen in Container`. This will create a new container, install any extensions specified in the project's default `.devcontainer/devcontainer.json` config file, and mount the project's root directory as the workspace folder. Once the container is created, VS Code will connect to it and you can start developing. :::{tip} -Clicking the `Starting Dev Container (show log)` notification in VS Code allows you to observe how the sausage is made. +Clicking the `Starting Dev Container (show log)` notification in VS Code allows you to observe in live time how the sausage is made, while typing `Dev Containers: Show Log` into the command palette will list all the available commands to review and revisit these log files later. ::: While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about creating and user dev containers, or how to visualize and leverage graphical user interfaces from a headless development environment. From bc736e03f78447209b626d5c2a82010f35d5c772 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Fri, 14 Apr 2023 12:16:08 +0200 Subject: [PATCH 067/107] Typo --- dev_containers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_containers/index.md b/dev_containers/index.md index e09b3351b..13929fbb8 100644 --- a/dev_containers/index.md +++ b/dev_containers/index.md @@ -78,7 +78,7 @@ Getting started using dev containers is as simple as opening the project in VS C Clicking the `Starting Dev Container (show log)` notification in VS Code allows you to observe in live time how the sausage is made, while typing `Dev Containers: Show Log` into the command palette will list all the available commands to review and revisit these log files later. ::: -While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about creating and user dev containers, or how to visualize and leverage graphical user interfaces from a headless development environment. +While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about creating and using dev containers, or how to visualize and leverage graphical user interfaces from a headless development environment. - **[](dev_containers_docs/development_guide.md)** - How to develop Nav2 using dev containers and supporting tools From aad97aff31d608f203672443186b35accb943abd Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:04:29 +0200 Subject: [PATCH 068/107] Comment out runArgs unintended side effects or cross talk between containers by default also avoids interfering with vscode's X11 forwarding --- dev_containers/dev_containers_docs/development_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 3e064607b..919e2b89f 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -77,7 +77,7 @@ To speed up subsequent startups, volumes that are mounted to the container store While these volumes are uniquely named, you could rename them locally to further organize or segment works-in-progress. E.g. appending branch names to the volume name to quickly switch between pull requests and cached colcon workspaces. ::: -Additionally, the container is granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) and non-default [Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities), connected using the [host](https://docs.docker.com/network/host/) network mode and [IPC](https://docs.docker.com/engine/reference/run/#ipc-settings---ipc) and [PID](https://docs.docker.com/engine/reference/run/#pid-settings---pid) spaces, with a relaxed [security configuration](https://docs.docker.com/engine/reference/run/#security-configuration) and seccomp confinement for native debugging and external connectivity. This is especially useful for: +Additionally, the container can be granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) and non-default [Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities), connected using the [host](https://docs.docker.com/network/host/) network mode and [IPC](https://docs.docker.com/engine/reference/run/#ipc-settings---ipc) and [PID](https://docs.docker.com/engine/reference/run/#pid-settings---pid) spaces, with a relaxed [security configuration](https://docs.docker.com/engine/reference/run/#security-configuration) and seccomp confinement for native debugging and external connectivity. This is especially useful for: - Hybrid development - Enables connecting ROS nodes external to the container @@ -89,7 +89,7 @@ Additionally, the container is granted [privileged](https://docs.docker.com/engi - Necessary for some GPU drivers and USB devices :::{attention} -These `runArgs` in the `devcontainer.json` config can be further customized, either expanded or or narrowed in scope, to better suit your desired development environment. The current configuration is merely the project default in order to be the most flexible and useful for the widest range of development use cases. +Such `runArgs` in the `devcontainer.json` config can be enabled or customized, either expanded or or narrowed in scope, to better suit your desired development environment. The default configuration merely comments out these parameters, to limit unintended side effects or cross talk between containers, but can be uncommented to accommodate the widest range of development use cases. ::: :::{seealso} From 076596f16d08273091c425e69951fc3f576f8c43 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:05:31 +0200 Subject: [PATCH 069/107] Wording --- dev_containers/dev_containers_docs/development_guide.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/dev_containers/dev_containers_docs/development_guide.md b/dev_containers/dev_containers_docs/development_guide.md index 919e2b89f..ba714b882 100644 --- a/dev_containers/dev_containers_docs/development_guide.md +++ b/dev_containers/dev_containers_docs/development_guide.md @@ -37,7 +37,7 @@ Once the base image from the target stage is built, the supporting tool or servi When first creating Dev Containers, any supporting tool or service will invoke a sequence of commands specified in the chosen `devcontainer.json` config file. This can take a while, but only needs to be done once, or at least not again until the container is rebuilt, triggered by either updating the Dockerfile, base image, or `.devcontainer/` config. -Specifically, for this project, the default `devcontainer.json` config executes the `onCreateCommand` to initially colcon cache, clean, and build the overlay workspace for the project. This ensures the workspace is precompiled and ready to use, while also ensuring any changes to the project's source code are reflected in the container. This is especially useful for: +Specifically, for this project, the default `devcontainer.json` config executes the `onCreateCommand` to initially colcon cache, clean, and build the overlay workspace for the project. This ensures the workspace is precompiled and ready to use, while also ensuring any changes to the project's source code are reflected in the container. This is useful for: - IntelliSense - Enables VS Code extensions to parse auto generated code @@ -48,7 +48,7 @@ Specifically, for this project, the default `devcontainer.json` config executes - Applicable for reducing startup time when spawning new Codespaces - Necessary for limiting costs from CPU and storage usage -While the colcon workspace is being built, VS Code will simultaneously install any specified extensions and settings. Next the `updateContentCommand` is executed, which reruns whenever the container is started or restarted. Specifically, for this project, this command re-cleans and re-builds the same colcon workspace as before, but only for invalidated packages detected by colcon cache using the lockfiles initialized during the `onCreateCommand`. This caching behavior also replicates the project's CI workflow. This is especially useful for: +While the colcon workspace is being built, VS Code will simultaneously install any specified extensions and settings. Next the `updateContentCommand` is executed, which reruns whenever the container is started or restarted. Specifically, for this project, this command re-cleans and re-builds the same colcon workspace as before, but only for invalidated packages detected by colcon cache using the lockfiles initialized during the `onCreateCommand`. This caching behavior also replicates the project's CI workflow. This is useful for: - Branching - Enables caching of workspace artifacts when switching between branches @@ -66,7 +66,7 @@ More documentation about these additional colcon verb extensions can be found he Finally, the `postCreateCommand` is executed, which also reruns whenever the container is started or restarted. Specifically, for this project, this command makes a last few tweaks to the user's environment to improve the development experience. -To speed up subsequent startups, volumes that are mounted to the container store a persistent ccache and colcon workspace, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). These volumes are labeled using the [`devcontainerId`](https://containers.dev/implementors/json_reference/#variables-in-devcontainerjson) variable, which uniquely identify the dev container on a Docker host, allowing us to refer to a common identifier that is unique to the dev container, while remaining stable across rebuilds. This is especially useful for: +To speed up subsequent startups, volumes that are mounted to the container store a persistent ccache and colcon workspace, while the environment is set to enable [ccache](https://ccache.dev/) via [colcon mixins](https://github.com/colcon/colcon-mixin-repository). These volumes are labeled using the [`devcontainerId`](https://containers.dev/implementors/json_reference/#variables-in-devcontainerjson) variable, which uniquely identify the dev container on a Docker host, allowing us to refer to a common identifier that is unique to the dev container, while remaining stable across rebuilds. This is useful for: - Caching - Enables colcon workspaces and ccache to persist between container rebuilds @@ -77,7 +77,7 @@ To speed up subsequent startups, volumes that are mounted to the container store While these volumes are uniquely named, you could rename them locally to further organize or segment works-in-progress. E.g. appending branch names to the volume name to quickly switch between pull requests and cached colcon workspaces. ::: -Additionally, the container can be granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) and non-default [Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities), connected using the [host](https://docs.docker.com/network/host/) network mode and [IPC](https://docs.docker.com/engine/reference/run/#ipc-settings---ipc) and [PID](https://docs.docker.com/engine/reference/run/#pid-settings---pid) spaces, with a relaxed [security configuration](https://docs.docker.com/engine/reference/run/#security-configuration) and seccomp confinement for native debugging and external connectivity. This is especially useful for: +Additionally, the container can be granted [privileged](https://docs.docker.com/engine/reference/commandline/run/#privileged) and non-default [Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities), connected using the [host](https://docs.docker.com/network/host/) network mode and [IPC](https://docs.docker.com/engine/reference/run/#ipc-settings---ipc) and [PID](https://docs.docker.com/engine/reference/run/#pid-settings---pid) spaces, with a relaxed [security configuration](https://docs.docker.com/engine/reference/run/#security-configuration) and seccomp confinement for native debugging and external connectivity. This is useful for: - Hybrid development - Enables connecting ROS nodes external to the container From 7a536f8df6db23abd2d7d563071f322ae6a1ba1f Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:27:51 +0200 Subject: [PATCH 070/107] Update requirements.txt to match docs and CI --- scripts/requirements.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 939919ae8..8d916139d 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -1,5 +1,6 @@ -breathe==4.9.1 -sphinx==1.7.5 +breathe==4.28.0 docutils==0.14 +jinja2==3.0.3 sphinx_rtd_theme +sphinx==3.5.0 sphinxcontrib-plantuml From 57ba3209f4b740b6a202279a2235ae5b1ede81ed Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:28:24 +0200 Subject: [PATCH 071/107] Add new dependencies --- scripts/requirements.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 8d916139d..cef9edc7e 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -1,6 +1,8 @@ breathe==4.28.0 docutils==0.14 jinja2==3.0.3 +myst-parser +sphinx_copybutton sphinx_rtd_theme sphinx==3.5.0 sphinxcontrib-plantuml From 737b7d6979c3d722e55a64f4613d9ba8e1a4fb3b Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:29:03 +0200 Subject: [PATCH 072/107] Fix version for new dependencies --- scripts/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/requirements.txt b/scripts/requirements.txt index cef9edc7e..92b0762a9 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -1,8 +1,8 @@ breathe==4.28.0 docutils==0.14 jinja2==3.0.3 -myst-parser -sphinx_copybutton +myst-parser==1.0.0 +sphinx_copybutton==0.5.2 sphinx_rtd_theme sphinx==3.5.0 sphinxcontrib-plantuml From 2f5dd357bfa0c861625240f9960e80cd188af38f Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:36:06 +0200 Subject: [PATCH 073/107] Move development files to simplify onboarding --- scripts/docker/Dockerfile => Dockerfile | 0 scripts/requirements.txt => requirements.txt | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename scripts/docker/Dockerfile => Dockerfile (100%) rename scripts/requirements.txt => requirements.txt (100%) diff --git a/scripts/docker/Dockerfile b/Dockerfile similarity index 100% rename from scripts/docker/Dockerfile rename to Dockerfile diff --git a/scripts/requirements.txt b/requirements.txt similarity index 100% rename from scripts/requirements.txt rename to requirements.txt From 95ca826f14b5c721e2ec56e899de3f9961f6429b Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:40:49 +0200 Subject: [PATCH 074/107] Update README to simply install using requirements.txt --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index eb608dd33..0a8ee244c 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,11 @@ This folder holds the source and configuration files used to generate the [Navigation2 documentation](https://navigation.ros.org) web site. Dependencies for Build: -* `sudo apt install python3-pip` -* `pip3 install sphinx==3.5.0 breathe==4.28.0 sphinx_rtd_theme sphinxcontrib-plantuml jinja2==3.0.3 myst-parser sphinx_copybutton` + +``` bash +sudo apt install python3-pip +pip3 install -r requirements.txt +``` Build the docs locally with `make html` and you'll find the built docs entry point in `_build/html/index.html`. From 453722b32a2da044b46bfa4849f20836d7ab1883 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:41:09 +0200 Subject: [PATCH 075/107] Update Dockerfile to simply install using requirements.txt --- Dockerfile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index b1d5c13a2..3adb8d41d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -5,7 +5,8 @@ FROM alpine:latest RUN apk --no-cache add python3 RUN python3 -m ensurepip -RUN pip3 install sphinx==1.7.5 docutils==0.14 sphinx_rtd_theme breathe==4.9.1 sphinxcontrib-plantuml sphinx_copybutton +COPY requirements.txt ./ +RUN pip3 install -r requirements.txt RUN apk --no-cache add make RUN apk --no-cache add doxygen RUN apk --no-cache add graphviz From cee4b247fb4d73274728c962499dba702ef09751 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:42:14 +0200 Subject: [PATCH 076/107] Update CI to simply install using requirements.txt --- .circleci/config.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 2cfbcc4b4..fb57be473 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -17,7 +17,7 @@ references: apt install openjdk-8-jre -y apt install bash git openssh-server -y apt install python3-pip python3 -y - pip3 install sphinx==3.5.0 docutils==0.14 sphinx_rtd_theme breathe==4.28.0 sphinxcontrib-plantuml jinja2==3.0.3 myst-parser sphinx_copybutton + pip3 install -r requirements.txt make_docs: &make_docs run: command: | @@ -62,8 +62,8 @@ jobs: docs_build: executor: docs_exec steps: - - install_doc_dependencies - *on_checkout + - install_doc_dependencies - build_docs - persist_to_workspace: root: nav2_docs @@ -72,8 +72,8 @@ jobs: docs_publish: executor: docs_exec steps: - - install_doc_dependencies - *on_checkout + - install_doc_dependencies - *install_deployment_key - attach_workspace: at: nav2_docs From 66f0fdb534e617506851010d599da3bc95edb31b Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:42:48 +0200 Subject: [PATCH 077/107] Simplify Dockerfile --- Dockerfile | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Dockerfile b/Dockerfile index 3adb8d41d..4ac127276 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,14 +1,14 @@ FROM alpine:latest -#ENV http_proxy -#ENV https_proxy +RUN apk --no-cache add \ + doxygen \ + graphviz \ + make \ + openjdk8-jre \ + python3 \ + ttf-dejavu -RUN apk --no-cache add python3 RUN python3 -m ensurepip + COPY requirements.txt ./ RUN pip3 install -r requirements.txt -RUN apk --no-cache add make -RUN apk --no-cache add doxygen -RUN apk --no-cache add graphviz -RUN apk --no-cache add ttf-dejavu -RUN apk --no-cache add openjdk8-jre From d0215956f00866e4ef7e228192a87e1a57ba7369 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:47:12 +0200 Subject: [PATCH 078/107] Simplify apt install --- .circleci/config.yml | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index fb57be473..38eb14f1a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -10,13 +10,14 @@ references: command: | export DEBIAN_FRONTEND=noninteractive apt update - apt install make -y - apt install doxygen -y - apt install graphviz -y - apt install ttf-dejavu -y - apt install openjdk-8-jre -y - apt install bash git openssh-server -y - apt install python3-pip python3 -y + apt install -y \ + make \ + doxygen \ + graphviz \ + ttf-dejavu \ + openjdk-8-jre \ + bash git openssh-server \ + python3-pip python3 pip3 install -r requirements.txt make_docs: &make_docs run: From b6c878bddd0951cfdeae1a2a49da6a523a63eb03 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:48:00 +0200 Subject: [PATCH 079/107] Expand apt install --- .circleci/config.yml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 38eb14f1a..40de17d7d 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -16,8 +16,11 @@ references: graphviz \ ttf-dejavu \ openjdk-8-jre \ - bash git openssh-server \ - python3-pip python3 + bash \ + git \ + openssh-server \ + python3-pip \ + python3 pip3 install -r requirements.txt make_docs: &make_docs run: From f53400de15dd39877d148181a70758ed14fcee03 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:48:29 +0200 Subject: [PATCH 080/107] Sort apt install --- .circleci/config.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 40de17d7d..4a7839d8a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -11,16 +11,16 @@ references: export DEBIAN_FRONTEND=noninteractive apt update apt install -y \ - make \ + bash \ doxygen \ + git \ graphviz \ - ttf-dejavu \ + make \ openjdk-8-jre \ - bash \ - git \ openssh-server \ + python3 \ python3-pip \ - python3 + ttf-dejavu pip3 install -r requirements.txt make_docs: &make_docs run: From 61752b83cb59dfbeefbbd0daf86997d92bc51f4a Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 13:50:13 +0200 Subject: [PATCH 081/107] Prune apt install --- .circleci/config.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 4a7839d8a..fd96a4b76 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -11,14 +11,12 @@ references: export DEBIAN_FRONTEND=noninteractive apt update apt install -y \ - bash \ doxygen \ git \ graphviz \ make \ openjdk-8-jre \ openssh-server \ - python3 \ python3-pip \ ttf-dejavu pip3 install -r requirements.txt From 0126257838f0539e049da2210db953b0a286c8c4 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 14:00:13 +0200 Subject: [PATCH 082/107] Update Dockerfile to match CI for consistency and simplicity --- Dockerfile | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Dockerfile b/Dockerfile index 4ac127276..789258bf3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,14 +1,16 @@ -FROM alpine:latest +FROM ubuntu:focal -RUN apk --no-cache add \ +ARG DEBIAN_FRONTEND=noninteractive +RUN apt update && \ + apt install -y \ doxygen \ + git \ graphviz \ make \ - openjdk8-jre \ - python3 \ + openjdk-8-jre \ + openssh-server \ + python3-pip \ ttf-dejavu -RUN python3 -m ensurepip - COPY requirements.txt ./ RUN pip3 install -r requirements.txt From f0e2f3f8fb98dbaae3ec91d853fe94a421f24eab Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 14:13:09 +0200 Subject: [PATCH 083/107] Fix working_directory to find requirements.txt in PWD --- .circleci/config.yml | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index fd96a4b76..2e10a18d3 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -3,7 +3,7 @@ version: 2.1 references: on_checkout: &on_checkout checkout: - path: nav2_docs + path: . setup_doc_dependencies: &setup_doc_dependencies run: name: Install Doc Dependencies @@ -23,7 +23,6 @@ references: make_docs: &make_docs run: command: | - cd nav2_docs make html store_docs: &store_docs store_artifacts: @@ -32,7 +31,6 @@ references: publish_docs: &publish_docs run: command: | - cd nav2_docs make publish install_deployment_key: &install_deployment_key add_ssh_keys: @@ -58,7 +56,7 @@ executors: docs_exec: docker: - image: ubuntu:focal - working_directory: / + working_directory: /nav2_docs jobs: docs_build: @@ -68,7 +66,7 @@ jobs: - install_doc_dependencies - build_docs - persist_to_workspace: - root: nav2_docs + root: . paths: - _build docs_publish: @@ -78,7 +76,7 @@ jobs: - install_doc_dependencies - *install_deployment_key - attach_workspace: - at: nav2_docs + at: . - publish_docs_to_gh_pages_branch workflows: From 5b503647184b89b4e9e96d6bd66255457d51b87f Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 14:22:54 +0200 Subject: [PATCH 084/107] Update attention to omit remarks about EOL distros --- build_instructions/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index 952994de7..fa0763d56 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -53,7 +53,7 @@ To build Nav2, you'll first need to build or install ROS 2 and related developme Once your environment is setup, clone the repo, install all dependencies, and build the workspace: .. attention:: - For ROS 2 distros prior to Galactic, the branch naming schema was ``-devel``, while the default branch for Rolling is ``main``. + The branch naming schema for Nav2 is organized by ROS distro, while the default branch for Rolling is ``main``. .. code:: bash From dad243134ed4d0481f203c58fd2c5b3db6591bc2 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 14:54:35 +0200 Subject: [PATCH 085/107] Avoid the use of export also helps keep the commands shell agnostic e.g. using another shell such as fish --- build_instructions/index.rst | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index fa0763d56..e4b29488d 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -58,8 +58,7 @@ Once your environment is setup, clone the repo, install all dependencies, and bu .. code:: bash source /opt/ros//setup.bash - export NAV2_WS=~/nav2_ws - mkdir -p $NAV2_WS/src && cd $NAV2_WS + mkdir -p ~/nav2_ws/src && cd ~/nav2_ws git clone https://github.com/ros-planning/navigation2.git --branch $ROS_DISTRO ./src/navigation2 rosdep install -y \ --from-paths ./src \ @@ -67,7 +66,7 @@ Once your environment is setup, clone the repo, install all dependencies, and bu colcon build \ --symlink-install -You can then ``source $NAV2_WS/install/setup.bash`` to get ready for demonstrations! +You can then ``source ~/nav2_ws/install/setup.bash`` to get ready for demonstrations! .. hint:: For more examples on building Nav2 from released distribution binaries, checkout `distro.Dockerfile `_. @@ -92,8 +91,7 @@ Once your environment is setup, clone the repo, import all dependencies, and bui .. code:: bash source /install/setup.bash - export NAV2_WS=~/nav2_ws - mkdir -p $NAV2_WS/src && cd $NAV2_WS + mkdir -p ~/nav2_ws/src && cd ~/nav2_ws git clone https://github.com/ros-planning/navigation2.git --branch $ROS_DISTRO ./src/navigation2 vcs import ./src < ./src/navigation2/tools/underlay.repos rosdep install -y \ @@ -102,7 +100,7 @@ Once your environment is setup, clone the repo, import all dependencies, and bui colcon build \ --symlink-install -You can then ``source $NAV2_WS/install/setup.bash`` to get ready for demonstrations! +You can then ``source ~/nav2_ws/install/setup.bash`` to get ready for demonstrations! .. hint:: For more examples on building Nav2 from rolling development source, checkout `source.Dockerfile `_. From 0cb6fb3a2cd3f849335ffffffa896a17d2836da4 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Tue, 18 Apr 2023 18:14:04 +0200 Subject: [PATCH 086/107] Rewording --- build_instructions/index.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/build_instructions/index.rst b/build_instructions/index.rst index e4b29488d..b87df9a14 100644 --- a/build_instructions/index.rst +++ b/build_instructions/index.rst @@ -24,15 +24,15 @@ There are a few ways to build Nav2 using: * Released Distribution Binaries - * Simply build Nav2 using installable binary dependencies + * Build Nav2 using readily installable binary dependencies * Rolling Development Source - * Intricately build Nav2 using from latest source dependencies + * Build Nav2 using custom or latest source dependencies * Docker Container Images - * Quickly build Nav2 using cached images and templated Dockerfiles + * Build Nav2 using cached images and templated Dockerfiles .. tip:: For a *repeatable*, *reproducible* and *streamlined* development experience, check the Nav2 documentation on using :ref:`dev-containers`! From e6aa6e4300f89d997020088d73f560692a5148c5 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 14:15:13 +0200 Subject: [PATCH 087/107] Refactor build and devcontainer docs --- .../index.rst => development/build_docs/build_instructions.rst | 0 .../build_docs/build_troubleshooting_guide.rst | 0 .../devcontainer_docs/devcontainer_guide.md | 0 .../devcontainer_docs/devcontainer_introduction.md | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename build_instructions/index.rst => development/build_docs/build_instructions.rst (100%) rename {build_instructions => development}/build_docs/build_troubleshooting_guide.rst (100%) rename dev_containers/dev_containers_docs/development_guide.md => development/devcontainer_docs/devcontainer_guide.md (100%) rename dev_containers/index.md => development/devcontainer_docs/devcontainer_introduction.md (100%) diff --git a/build_instructions/index.rst b/development/build_docs/build_instructions.rst similarity index 100% rename from build_instructions/index.rst rename to development/build_docs/build_instructions.rst diff --git a/build_instructions/build_docs/build_troubleshooting_guide.rst b/development/build_docs/build_troubleshooting_guide.rst similarity index 100% rename from build_instructions/build_docs/build_troubleshooting_guide.rst rename to development/build_docs/build_troubleshooting_guide.rst diff --git a/dev_containers/dev_containers_docs/development_guide.md b/development/devcontainer_docs/devcontainer_guide.md similarity index 100% rename from dev_containers/dev_containers_docs/development_guide.md rename to development/devcontainer_docs/devcontainer_guide.md diff --git a/dev_containers/index.md b/development/devcontainer_docs/devcontainer_introduction.md similarity index 100% rename from dev_containers/index.md rename to development/devcontainer_docs/devcontainer_introduction.md From 9888bc73e9a8183fea5f664474489fcc58c700f3 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 19 Apr 2023 15:44:08 +0200 Subject: [PATCH 088/107] Set language to english to fix warnings - https://github.com/sphinx-doc/sphinx/issues/10474 --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index 091f74a30..e37f46d61 100644 --- a/conf.py +++ b/conf.py @@ -84,7 +84,7 @@ # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. -language = None +language = 'en' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. From e385b01fd820094e29cb05018181e317ef5e764b Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 19 Apr 2023 15:46:53 +0200 Subject: [PATCH 089/107] Exclude README file from markdown pattern to fix warnings --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index e37f46d61..14bd0061b 100644 --- a/conf.py +++ b/conf.py @@ -89,7 +89,7 @@ # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path -exclude_patterns = ['_build','_themes','scripts' ] +exclude_patterns = ['_build','_themes','scripts', 'README.md' ] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' From a55e3f92d118eb1202680494e1b05e37093e9920 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 19 Apr 2023 15:50:29 +0200 Subject: [PATCH 090/107] Orphan files not included in any toctree to fix warnings - https://stackoverflow.com/a/40725923/2577586 --- 2021summerOfCode/Summer_2021_Student_Program.rst | 2 ++ 2021summerOfCode/projects/create_configuration_assistant.rst | 2 ++ 2021summerOfCode/projects/create_plugins.rst | 2 ++ 2021summerOfCode/projects/grid_maps.rst | 2 ++ 2021summerOfCode/projects/localization.rst | 2 ++ 2021summerOfCode/projects/navigation_rebranding.rst | 2 ++ 6 files changed, 12 insertions(+) diff --git a/2021summerOfCode/Summer_2021_Student_Program.rst b/2021summerOfCode/Summer_2021_Student_Program.rst index 492fc4255..7e90bad69 100644 --- a/2021summerOfCode/Summer_2021_Student_Program.rst +++ b/2021summerOfCode/Summer_2021_Student_Program.rst @@ -1,3 +1,5 @@ +:orphan: + .. _summer_2021_student_program: diff --git a/2021summerOfCode/projects/create_configuration_assistant.rst b/2021summerOfCode/projects/create_configuration_assistant.rst index cb7597006..02340a782 100644 --- a/2021summerOfCode/projects/create_configuration_assistant.rst +++ b/2021summerOfCode/projects/create_configuration_assistant.rst @@ -1,3 +1,5 @@ +:orphan: + .. _create_moveit_analog: 1. Create a Configuration Assistant (Analog to MoveIt) diff --git a/2021summerOfCode/projects/create_plugins.rst b/2021summerOfCode/projects/create_plugins.rst index d5afb9fea..ae40be3a6 100644 --- a/2021summerOfCode/projects/create_plugins.rst +++ b/2021summerOfCode/projects/create_plugins.rst @@ -1,3 +1,5 @@ +:orphan: + .. _create_plugins: diff --git a/2021summerOfCode/projects/grid_maps.rst b/2021summerOfCode/projects/grid_maps.rst index cf7baa293..1cd1f3011 100644 --- a/2021summerOfCode/projects/grid_maps.rst +++ b/2021summerOfCode/projects/grid_maps.rst @@ -1,3 +1,5 @@ +:orphan: + .. _grid_maps: diff --git a/2021summerOfCode/projects/localization.rst b/2021summerOfCode/projects/localization.rst index 058a55b9e..db29af9d7 100644 --- a/2021summerOfCode/projects/localization.rst +++ b/2021summerOfCode/projects/localization.rst @@ -1,3 +1,5 @@ +:orphan: + .. _localization: diff --git a/2021summerOfCode/projects/navigation_rebranding.rst b/2021summerOfCode/projects/navigation_rebranding.rst index 5660898ab..adf0a59de 100644 --- a/2021summerOfCode/projects/navigation_rebranding.rst +++ b/2021summerOfCode/projects/navigation_rebranding.rst @@ -1,3 +1,5 @@ +:orphan: + .. _rebranding: From 748051331b687ab77afb47a12c73cf14a212d8d7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 14:57:09 +0200 Subject: [PATCH 091/107] Stage renaming links --- .../build_docs/build_instructions.rst | 6 +++--- .../build_docs/build_troubleshooting_guide.rst | 0 .../devcontainer_docs/devcontainer_guide.md | 3 ++- .../devcontainer_docs/devcontainer_introduction.md | 9 ++++----- development_guides/index.rst | 11 +++++++++++ index.rst | 3 +-- 6 files changed, 21 insertions(+), 11 deletions(-) rename {development => development_guides}/build_docs/build_instructions.rst (97%) rename {development => development_guides}/build_docs/build_troubleshooting_guide.rst (100%) rename {development => development_guides}/devcontainer_docs/devcontainer_guide.md (99%) rename {development => development_guides}/devcontainer_docs/devcontainer_introduction.md (97%) create mode 100644 development_guides/index.rst diff --git a/development/build_docs/build_instructions.rst b/development_guides/build_docs/build_instructions.rst similarity index 97% rename from development/build_docs/build_instructions.rst rename to development_guides/build_docs/build_instructions.rst index b87df9a14..0616c8ca3 100644 --- a/development/build_docs/build_instructions.rst +++ b/development_guides/build_docs/build_instructions.rst @@ -35,7 +35,7 @@ There are a few ways to build Nav2 using: * Build Nav2 using cached images and templated Dockerfiles .. tip:: - For a *repeatable*, *reproducible* and *streamlined* development experience, check the Nav2 documentation on using :ref:`dev-containers`! + For a *repeatable*, *reproducible* and *streamlined* development experience, check the Nav2 documentation on using :ref:`devcontainer-introduction`! .. rst-class:: content-collapse @@ -133,7 +133,7 @@ Once your system is setup, you can build the Nav2 Dockerfile from the root of th The `docker build `_ command above creates a tagged image using the `Dockerfile` from the context specified using the path to the repo, where build-time variables are set using additional arguments, e.g. passing a set of `colcon mixins `_ to configure the workspace build. Check the ``ARG`` directives in the `Dockerfile` to discover all build-time variables available. The command also specifies an `external cache source `_ to pull the latest cached image from Nav2's `Container Registry `_ to speed up the build process. .. tip:: - The images cached from above are used for Nav2 CI, but can also be used with Nav2 :ref:`dev-containers`! + The images cached from above are used for Nav2 CI, but can also be used with Nav2 :ref:`devcontainer-introduction`! !!!! @@ -155,4 +155,4 @@ Help .. toctree:: :hidden: - build_docs/build_troubleshooting_guide.rst + build_troubleshooting_guide.rst diff --git a/development/build_docs/build_troubleshooting_guide.rst b/development_guides/build_docs/build_troubleshooting_guide.rst similarity index 100% rename from development/build_docs/build_troubleshooting_guide.rst rename to development_guides/build_docs/build_troubleshooting_guide.rst diff --git a/development/devcontainer_docs/devcontainer_guide.md b/development_guides/devcontainer_docs/devcontainer_guide.md similarity index 99% rename from development/devcontainer_docs/devcontainer_guide.md rename to development_guides/devcontainer_docs/devcontainer_guide.md index ba714b882..dd2403185 100644 --- a/development/devcontainer_docs/devcontainer_guide.md +++ b/development_guides/devcontainer_docs/devcontainer_guide.md @@ -1,4 +1,5 @@ -# Development Guide +(devcontainer-guide)= +# Dev Container Guide In this guide, we'll walk through the process of creating and using dev containers for the project. While included subsections will provide greater detail on the various aspects of the process, complete comprehension of the entire guide is not required to get started, but is recommended for those interested in how dev containers work, or how to customize and optimize them for their own personal workflows. diff --git a/development/devcontainer_docs/devcontainer_introduction.md b/development_guides/devcontainer_docs/devcontainer_introduction.md similarity index 97% rename from development/devcontainer_docs/devcontainer_introduction.md rename to development_guides/devcontainer_docs/devcontainer_introduction.md index 13929fbb8..f31ace286 100644 --- a/development/devcontainer_docs/devcontainer_introduction.md +++ b/development_guides/devcontainer_docs/devcontainer_introduction.md @@ -1,5 +1,5 @@ -(dev-containers)= -# Dev Containers +(devcontainer-introduction)= +# Dev Container Introduction You can use dev containers to build the project if you prefer a streamlined setup experience. This means you can use the same tools and dependencies as the rest of the team, including our Continuous Integration (CI) workflows, without worrying about installing dependencies on your host machine. Additionally, using Dev Containers makes it simple to switch between local or remote development environments, such as GitHub Codespaces. More info on Dev Containers can be found here: @@ -13,8 +13,7 @@ You can use dev containers to build the project if you prefer a streamlined setu ```{toctree} :hidden: -dev_containers_docs/development_guide.md -dev_containers_docs/visualization_guide.md +development_guide.md ``` ## What, Why, How? @@ -80,7 +79,7 @@ Clicking the `Starting Dev Container (show log)` notification in VS Code allows While waiting for the initial setup, feel free to stretch your legs, grab a coffee, or continue to read the following guides to learn more about creating and using dev containers, or how to visualize and leverage graphical user interfaces from a headless development environment. -- **[](dev_containers_docs/development_guide.md)** +- **[](devcontainer_guide.md)** - How to develop Nav2 using dev containers and supporting tools ## Security diff --git a/development_guides/index.rst b/development_guides/index.rst new file mode 100644 index 000000000..52ed88889 --- /dev/null +++ b/development_guides/index.rst @@ -0,0 +1,11 @@ +.. _development-guides: + +Development Guides +########### + +.. toctree:: + :maxdepth: 1 + + build_docs/build_instructions.rst + devcontainer_docs/devcontainer_introduction.md + devcontainer_docs/devcontainer_guide.md diff --git a/index.rst b/index.rst index b1607a88b..87d582c39 100644 --- a/index.rst +++ b/index.rst @@ -136,8 +136,7 @@ Below is an example of the TB3 navigating in a small lounge. :hidden: getting_started/index.rst - build_instructions/index.rst - dev_containers/index.md + development_guides/index.rst concepts/index.rst setup_guides/index.rst tutorials/index.rst From 7c54a15030f96277f447f3716526295194e05d61 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 15:44:01 +0200 Subject: [PATCH 092/107] Rename indexs and headings --- .../build_docs/{build_instructions.rst => index.rst} | 4 ++-- .../{devcontainer_introduction.md => index.md} | 4 ++-- development_guides/index.rst | 5 ++--- 3 files changed, 6 insertions(+), 7 deletions(-) rename development_guides/build_docs/{build_instructions.rst => index.rst} (98%) rename development_guides/devcontainer_docs/{devcontainer_introduction.md => index.md} (99%) diff --git a/development_guides/build_docs/build_instructions.rst b/development_guides/build_docs/index.rst similarity index 98% rename from development_guides/build_docs/build_instructions.rst rename to development_guides/build_docs/index.rst index 0616c8ca3..017756f2c 100644 --- a/development_guides/build_docs/build_instructions.rst +++ b/development_guides/build_docs/index.rst @@ -35,7 +35,7 @@ There are a few ways to build Nav2 using: * Build Nav2 using cached images and templated Dockerfiles .. tip:: - For a *repeatable*, *reproducible* and *streamlined* development experience, check the Nav2 documentation on using :ref:`devcontainer-introduction`! + For a *repeatable*, *reproducible* and *streamlined* development experience, check the Nav2 documentation on using :ref:`devcontainers`! .. rst-class:: content-collapse @@ -133,7 +133,7 @@ Once your system is setup, you can build the Nav2 Dockerfile from the root of th The `docker build `_ command above creates a tagged image using the `Dockerfile` from the context specified using the path to the repo, where build-time variables are set using additional arguments, e.g. passing a set of `colcon mixins `_ to configure the workspace build. Check the ``ARG`` directives in the `Dockerfile` to discover all build-time variables available. The command also specifies an `external cache source `_ to pull the latest cached image from Nav2's `Container Registry `_ to speed up the build process. .. tip:: - The images cached from above are used for Nav2 CI, but can also be used with Nav2 :ref:`devcontainer-introduction`! + The images cached from above are used for Nav2 CI, but can also be used with Nav2 :ref:`devcontainers`! !!!! diff --git a/development_guides/devcontainer_docs/devcontainer_introduction.md b/development_guides/devcontainer_docs/index.md similarity index 99% rename from development_guides/devcontainer_docs/devcontainer_introduction.md rename to development_guides/devcontainer_docs/index.md index f31ace286..638a5edb4 100644 --- a/development_guides/devcontainer_docs/devcontainer_introduction.md +++ b/development_guides/devcontainer_docs/index.md @@ -1,5 +1,5 @@ -(devcontainer-introduction)= -# Dev Container Introduction +(devcontainers)= +# Dev Containers You can use dev containers to build the project if you prefer a streamlined setup experience. This means you can use the same tools and dependencies as the rest of the team, including our Continuous Integration (CI) workflows, without worrying about installing dependencies on your host machine. Additionally, using Dev Containers makes it simple to switch between local or remote development environments, such as GitHub Codespaces. More info on Dev Containers can be found here: diff --git a/development_guides/index.rst b/development_guides/index.rst index 52ed88889..278bd1fbe 100644 --- a/development_guides/index.rst +++ b/development_guides/index.rst @@ -6,6 +6,5 @@ Development Guides .. toctree:: :maxdepth: 1 - build_docs/build_instructions.rst - devcontainer_docs/devcontainer_introduction.md - devcontainer_docs/devcontainer_guide.md + build_docs/index.rst + devcontainer_docs/index.md From 316d4971693c1c65ce3ca4d584d8bd281a08e22c Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 15:44:53 +0200 Subject: [PATCH 093/107] Use globe for hidden toctree --- development_guides/devcontainer_docs/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/development_guides/devcontainer_docs/index.md b/development_guides/devcontainer_docs/index.md index 638a5edb4..00a41317f 100644 --- a/development_guides/devcontainer_docs/index.md +++ b/development_guides/devcontainer_docs/index.md @@ -12,8 +12,9 @@ You can use dev containers to build the project if you prefer a streamlined setu ```{toctree} :hidden: +:glob: -development_guide.md +./* ``` ## What, Why, How? From d221394f9913ccd3c0d6e133b55108f33b76a551 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 15:45:33 +0200 Subject: [PATCH 094/107] Add guide index for discoverability --- development_guides/devcontainer_docs/index.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/development_guides/devcontainer_docs/index.md b/development_guides/devcontainer_docs/index.md index 00a41317f..5759acef0 100644 --- a/development_guides/devcontainer_docs/index.md +++ b/development_guides/devcontainer_docs/index.md @@ -97,3 +97,12 @@ More info on trusting workspaces and extensions in general can be found here: - [Workspace Trust](https://code.visualstudio.com/docs/editor/workspace-trust) - VS Code user guid on trusting and configure workspaces ::: + +## Guide Index + +```{toctree} +:maxdepth: 3 +:glob: + +./* +``` From f9d1804d5a46dfd2be9306eec2e8ebeb51079f7f Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 15:52:57 +0200 Subject: [PATCH 095/107] Reorder caution admonition as a TLDR --- development_guides/devcontainer_docs/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/development_guides/devcontainer_docs/index.md b/development_guides/devcontainer_docs/index.md index 5759acef0..9d0f32751 100644 --- a/development_guides/devcontainer_docs/index.md +++ b/development_guides/devcontainer_docs/index.md @@ -85,12 +85,12 @@ While waiting for the initial setup, feel free to stretch your legs, grab a coff ## Security -A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. - -:::{danger} +:::{caution} Ensure you trust the authors and contents of workspaces before launching derived dev containers. ::: +A word of caution when using dev containers: they are powerful tools, but can be a security concern, as the capability of arbitrary code execution facilitated by IDE extensions to enable such automation and convenience remains inherently dual use. Before launching a dev container, ensure you trust the workspaces and authors. For example, when reviewing a pull request, verify patches remain benign and do not introduce any malicious code. Although such vigilance is merited whenever compiling and running patched code, using containers with either elevated privileges or filesystem access renders this diligence even more prudent. + :::{seealso} More info on trusting workspaces and extensions in general can be found here: From a43c001f7ed617a27dbc37c326907a65bf2500c7 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 16:12:16 +0200 Subject: [PATCH 096/107] Add body text to index --- development_guides/index.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/development_guides/index.rst b/development_guides/index.rst index 278bd1fbe..03f69b05c 100644 --- a/development_guides/index.rst +++ b/development_guides/index.rst @@ -1,7 +1,12 @@ .. _development-guides: Development Guides -########### +################## + +This section contains various guides for developing the project. + +.. note:: + For more general information about contributing to the project, see the :ref:`contribute` section. .. toctree:: :maxdepth: 1 From 265c2cb037f8ca18e273bcdf20d0af66431bd6f8 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 16:37:01 +0200 Subject: [PATCH 097/107] Combine contribute into development guide --- development_guides/index.rst | 11 ++++------- .../involvement_docs}/index.rst | 0 index.rst | 1 - 3 files changed, 4 insertions(+), 8 deletions(-) rename {contribute => development_guides/involvement_docs}/index.rst (100%) diff --git a/development_guides/index.rst b/development_guides/index.rst index 03f69b05c..862a7e8ee 100644 --- a/development_guides/index.rst +++ b/development_guides/index.rst @@ -3,13 +3,10 @@ Development Guides ################## -This section contains various guides for developing the project. - -.. note:: - For more general information about contributing to the project, see the :ref:`contribute` section. +This section includes guides for developing Nav2, e.g. how to build from source, how to use dev containers, and how to get involved. .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :glob: - build_docs/index.rst - devcontainer_docs/index.md + */index diff --git a/contribute/index.rst b/development_guides/involvement_docs/index.rst similarity index 100% rename from contribute/index.rst rename to development_guides/involvement_docs/index.rst diff --git a/index.rst b/index.rst index 87d582c39..18c3469b0 100644 --- a/index.rst +++ b/index.rst @@ -148,6 +148,5 @@ Below is an example of the TB3 navigating in a small lounge. migration/index.rst commander_api/index.rst roadmap/roadmap.rst - contribute/index.rst about/index.rst about/robots.rst From 2f98a97623ebb57970c19bd919a5672c028242fb Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 16:42:26 +0200 Subject: [PATCH 098/107] Add hidden toctree without maxdepth to ensure build_instructions is included in the sidebar --- development_guides/index.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/development_guides/index.rst b/development_guides/index.rst index 862a7e8ee..72a2cd9c8 100644 --- a/development_guides/index.rst +++ b/development_guides/index.rst @@ -5,6 +5,12 @@ Development Guides This section includes guides for developing Nav2, e.g. how to build from source, how to use dev containers, and how to get involved. +.. toctree:: + :hidden: + :glob: + + */index + .. toctree:: :maxdepth: 2 :glob: From 42689a62452e55c4cbf202a1b2bfe3c0717b6bea Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 16:46:35 +0200 Subject: [PATCH 099/107] Use seealso admonition for dev container links --- development_guides/devcontainer_docs/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/development_guides/devcontainer_docs/index.md b/development_guides/devcontainer_docs/index.md index 9d0f32751..059e72f28 100644 --- a/development_guides/devcontainer_docs/index.md +++ b/development_guides/devcontainer_docs/index.md @@ -1,7 +1,10 @@ (devcontainers)= # Dev Containers -You can use dev containers to build the project if you prefer a streamlined setup experience. This means you can use the same tools and dependencies as the rest of the team, including our Continuous Integration (CI) workflows, without worrying about installing dependencies on your host machine. Additionally, using Dev Containers makes it simple to switch between local or remote development environments, such as GitHub Codespaces. More info on Dev Containers can be found here: +You can use dev containers to build the project if you prefer a streamlined setup experience. This means you can use the same tools and dependencies as the rest of the team, including our Continuous Integration (CI) workflows, without worrying about installing dependencies on your host machine. Additionally, using Dev Containers makes it simple to switch between local or remote development environments, such as GitHub Codespaces. + +:::{seealso} +More info on Dev Containers can be found here: - [Development Containers](https://containers.dev/) - An open specification for enriching containers with development specific content and settings @@ -9,6 +12,7 @@ You can use dev containers to build the project if you prefer a streamlined setu - Learn how to use Visual Studio Code to develop inside a Docker container - [GitHub Codespaces overview](https://docs.github.com/en/codespaces/overview) - A development environment hosted in the cloud +::: ```{toctree} :hidden: From ef233b27694d2ea3f703803c799ea4617496ea52 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 18:00:13 +0200 Subject: [PATCH 100/107] Revert change to avoid duplicate toctree entries otherwise the sidebar navigation open duplicate dropdowns --- development_guides/devcontainer_docs/index.md | 11 +---------- development_guides/index.rst | 6 ------ 2 files changed, 1 insertion(+), 16 deletions(-) diff --git a/development_guides/devcontainer_docs/index.md b/development_guides/devcontainer_docs/index.md index 059e72f28..a6d3bdd6a 100644 --- a/development_guides/devcontainer_docs/index.md +++ b/development_guides/devcontainer_docs/index.md @@ -18,7 +18,7 @@ More info on Dev Containers can be found here: :hidden: :glob: -./* +* ``` ## What, Why, How? @@ -101,12 +101,3 @@ More info on trusting workspaces and extensions in general can be found here: - [Workspace Trust](https://code.visualstudio.com/docs/editor/workspace-trust) - VS Code user guid on trusting and configure workspaces ::: - -## Guide Index - -```{toctree} -:maxdepth: 3 -:glob: - -./* -``` diff --git a/development_guides/index.rst b/development_guides/index.rst index 72a2cd9c8..862a7e8ee 100644 --- a/development_guides/index.rst +++ b/development_guides/index.rst @@ -5,12 +5,6 @@ Development Guides This section includes guides for developing Nav2, e.g. how to build from source, how to use dev containers, and how to get involved. -.. toctree:: - :hidden: - :glob: - - */index - .. toctree:: :maxdepth: 2 :glob: From 99a806e4e7d936855fa94a2bf233bfa3d3433c5b Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 18:42:48 +0200 Subject: [PATCH 101/107] Instal sphinx-autobuild --- requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/requirements.txt b/requirements.txt index 92b0762a9..a09f86ad9 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,5 +4,6 @@ jinja2==3.0.3 myst-parser==1.0.0 sphinx_copybutton==0.5.2 sphinx_rtd_theme +sphinx-autobuild sphinx==3.5.0 sphinxcontrib-plantuml From c3a1e43b85540e39029914b5d4360844b2992a6e Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 18:44:18 +0200 Subject: [PATCH 102/107] Add make build task and make it default --- .vscode/tasks.json | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 .vscode/tasks.json diff --git a/.vscode/tasks.json b/.vscode/tasks.json new file mode 100644 index 000000000..d8bd9ca0d --- /dev/null +++ b/.vscode/tasks.json @@ -0,0 +1,21 @@ +{ + // See https://go.microsoft.com/fwlink/?LinkId=733558 + // for the documentation about the tasks.json format + "version": "2.0.0", + "tasks": [ + { + "label": "Build", + "icon": { + "id": "gear" + }, + "type": "shell", + "command": "make html", + "isBackground": false, + "problemMatcher": [], + "group": { + "kind": "build", + "isDefault": true + } + } + ] +} From fbecd889b2ae073fac44498580d5a7c7bb0ab996 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 18:44:39 +0200 Subject: [PATCH 103/107] Add .dockerignore file --- .dockerignore | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 .dockerignore diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 000000000..111b0017b --- /dev/null +++ b/.dockerignore @@ -0,0 +1,13 @@ +################################################################################ +# Repo + +.circleci/ +.devcontainer/ +.dockerignore +.git/ +.github/ +.gitignore +.vscode/ +**.Dockerfile +**Dockerfile +_build/ From 8bc021fe2fde7086b9011c28755d50493534d73d Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 18:45:36 +0200 Subject: [PATCH 104/107] Add auto build option --- .vscode/tasks.json | 14 ++++++++++++++ Makefile | 5 +++++ 2 files changed, 19 insertions(+) diff --git a/.vscode/tasks.json b/.vscode/tasks.json index d8bd9ca0d..d654a782f 100644 --- a/.vscode/tasks.json +++ b/.vscode/tasks.json @@ -16,6 +16,20 @@ "kind": "build", "isDefault": true } + }, + { + "label": "Autobuild", + "icon": { + "id": "debug-start" + }, + "type": "shell", + "command": "make autobuild", + "isBackground": true, + "problemMatcher": [], + "group": { + "kind": "build", + "isDefault": false + } } ] } diff --git a/Makefile b/Makefile index c2de14db2..64c83765e 100644 --- a/Makefile +++ b/Makefile @@ -35,6 +35,11 @@ help: html: $(Q)$(SPHINXBUILD) -t $(DOC_TAG) -b html -d $(BUILDDIR)/doctrees $(SOURCEDIR) $(BUILDDIR)/html $(SPHINXOPTS) $(O) +# Autobuild the docs on changes + +autobuild: + sphinx-autobuild -t $(DOC_TAG) -b html -d $(BUILDDIR)/doctrees $(SOURCEDIR) $(BUILDDIR)/html $(SPHINXOPTS) + # Remove generated content (Sphinx and doxygen) clean: From 33842a83e31e3af73cce137e73924cab3a49ada5 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 18:46:08 +0200 Subject: [PATCH 105/107] Show all warning for better debugging --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 64c83765e..7a87b96a3 100644 --- a/Makefile +++ b/Makefile @@ -8,7 +8,7 @@ else endif # You can set these variables from the command line. -SPHINXOPTS ?= -q +SPHINXOPTS ?= SPHINXBUILD = sphinx-build SPHINXPROJ = "Nav2 Documentation" SOURCEDIR = . From 068a547f51d0a6a1d9738582320f0d3b2d8a4f18 Mon Sep 17 00:00:00 2001 From: ruffsl Date: Thu, 20 Apr 2023 18:47:14 +0200 Subject: [PATCH 106/107] Add devcontainer config --- .devcontainer/devcontainer.json | 22 ++++++++++++++++++++++ .devcontainer/on-create-command.sh | 12 ++++++++++++ .devcontainer/update-content-command.sh | 11 +++++++++++ 3 files changed, 45 insertions(+) create mode 100644 .devcontainer/devcontainer.json create mode 100755 .devcontainer/on-create-command.sh create mode 100755 .devcontainer/update-content-command.sh diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json new file mode 100644 index 000000000..4ef032abb --- /dev/null +++ b/.devcontainer/devcontainer.json @@ -0,0 +1,22 @@ +{ + "name": "Nav2 Docs", + "build": { + "dockerfile": "../Dockerfile", + "context": ".." + }, + "workspaceMount": "source=${localWorkspaceFolder},target=/docs,type=bind", + "workspaceFolder": "/docs", + "onCreateCommand": ".devcontainer/on-create-command.sh", + "updateContentCommand": ".devcontainer/update-content-command.sh", + "customizations": { + "vscode": { + "settings": {}, + "extensions": [ + "eamodio.gitlens", + "esbenp.prettier-vscode", + "GitHub.copilot", + "streetsidesoftware.code-spell-checker" + ] + } + } +} diff --git a/.devcontainer/on-create-command.sh b/.devcontainer/on-create-command.sh new file mode 100755 index 000000000..f86d22bc8 --- /dev/null +++ b/.devcontainer/on-create-command.sh @@ -0,0 +1,12 @@ +#!/bin/bash + +# Immediately catch all errors +set -eo pipefail + +# Uncomment for debugging +# set -x +# env + +git config --global --add safe.directory "*" + +.devcontainer/update-content-command.sh diff --git a/.devcontainer/update-content-command.sh b/.devcontainer/update-content-command.sh new file mode 100755 index 000000000..b7f6536ee --- /dev/null +++ b/.devcontainer/update-content-command.sh @@ -0,0 +1,11 @@ +#!/bin/bash + +# Immediately catch all errors +set -eo pipefail + +# Uncomment for debugging +# set -x +# env + +# make clean +# make html From c95f1171048e35c8916dcee530533b96d4b1344a Mon Sep 17 00:00:00 2001 From: ruffsl Date: Wed, 26 Apr 2023 22:34:33 +0200 Subject: [PATCH 107/107] Fix artifact path with respect to working_directory in docs_exec --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 2e10a18d3..c3b33c341 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -26,7 +26,7 @@ references: make html store_docs: &store_docs store_artifacts: - path: nav2_docs/_build/html + path: _build/html destination: html publish_docs: &publish_docs run: