From 2c8a9e44a474c37c4d53b2e504cff67c33aabfe6 Mon Sep 17 00:00:00 2001 From: LachlanCourt Date: Wed, 13 Jul 2022 00:42:13 +1000 Subject: [PATCH 1/6] Add opencl, fake camera, and image compressor information to maintaining subsystems --- .../02-maintaining-subsystems.mdx | 75 +++++++++++++++++-- 1 file changed, 70 insertions(+), 5 deletions(-) diff --git a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx index 63820e5e0..21f647df7 100644 --- a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx +++ b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx @@ -48,12 +48,49 @@ The resulting network should be exported to a [yaml](https://yaml.org/) file and 3. Add this configuration file to the NUbots repository in the [VisualMesh module](https://github.com/NUbots/NUbots/tree/main/module/vision/VisualMesh/data/config/networks). Replace or add a configuration file depending on the use case of the Mesh - `RobocupNetwork.yaml` is for soccer playing on the real robot and `WebotsNetwork` is for soccer playing in the Webots simulator. View the [Git Guide](/guides/general/git) for information on using Git and submitting this change in a pull request. +#### Running OpenCL not on the robot + +1. Install the OpenCL SDK for your device from the [Intel website](https://www.intel.com/content/www/us/en/developer/tools/opencl-sdk/choose-download.html) + +2. Create a temporary directory and install the pre-built OpenCL drivers + +```bash +mkdir temp +cd temp +wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-gmmlib_19.1.1_amd64.deb +wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-igc-core_19.11.1622_amd64.deb +wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-igc-opencl_19.11.1622_amd64.deb +wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-opencl_19.14.12751_amd64.deb +wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-ocloc_19.14.12751_amd64.deb +sudo apt install ./*deb +``` + +3. Change the `VisualMesh.yaml` configuration "engine" node to `opencl` + +4. Build the binary and then run with the following command to bind the PC drivers to the docker image + +```bash +./b build +./b run --mount type=bind,source="/dev/dri",target="/dev/dri" +``` + ### Camera Calibration The vision system cannot work optimally if the cameras are not calibrated correctly. [The input page](/system/subsystems/input) describes the camera parameters that can be calibrated. An automatic camera calibration tool is available in the NUbots repository. See the [camera calibration guide](/guides/main/camera-calibration) to find out how to use this tool. + +### Fake Camera + +To run the fake camera module, an images directory needs to exist inside the `data` directory of the module. + +1. Create an `images` directory and add several images with filenames matching `imageXXXXXX.jpg` where "XXXXXX" refers to any numerical digits + +2. Change the `image_folder` node to the location that the data is stored in the docker image: `/home/nubots/NUbots/module/input/FakeCamera/data/images` + +3. Build and run a role with the fake camera in the position in the list that the regular camera module normally installs + ### Testing After updating the Visual Mesh in the NUbots repository, it should be tested before merging. Refer to the [Getting Started guide](/guides/main/getting-started) for assistance for the following steps. @@ -85,9 +122,37 @@ After updating the Visual Mesh in the NUbots repository, it should be tested bef If you would like to see the Visual Mesh output in NUsight, you will need to log the data and run it back in NUsight using DataPlayback, since the data is too large to send over a network. Use the steps in the [DataLogging and DataPlayback guide](/guides/main/data-recording-playback) to record and playback data. Adjust the instructions for our purpose using the following hints: - In step 1 of Recording Data, use the [`visualmesh`](https://github.com/NUbots/NUbots/blob/main/roles/visualmesh.role) role to record the data. -- In step 2 of Recording Data and step 4 of Playing Back Data, set `message.output.CompressedImage` to `true` and add `message.vision.VisualMesh: true` in both [`DataLogging.yaml](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataLogging/data/config/DataLogging.yaml) and [`DataPlayback.yaml](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataPlayback/data/config/DataPlayback.yaml). +- In step 2 of Recording Data and step 4 of Playing Back Data, set `message.output.CompressedImage` to `true` and add `message.vision.VisualMesh: true` in both [`DataLogging.yaml`](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataLogging/data/config/DataLogging.yaml) and [`DataPlayback.yaml`](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataPlayback/data/config/DataPlayback.yaml). - In steps 1, 2 and 5 of Playing Back Data, use the [`playback`](https://github.com/NUbots/NUbots/blob/main/roles/playback.role) role to playback the data, without changes. +### Image Compressors + +In order to send data to NUsight via the network, images need to be compressed. Depending on the environment different compressors can be used. [TurboJPEG](https://github.com/libjpeg-turbo/libjpeg-turbo) is the compressor that can be used on all platforms, whilst [vaapi](https://01.org/linuxmedia/vaapi) is the preferred compressor for devices running on Intel CPUs + +The configuration for either of these belongs in `ImageCompressor.yaml` and is as follows + +#### TurboJPEG + +```bash +compressor: + name: turbojpeg + concurrent: 2 + quality: 90 +``` + +#### Vaapi + +Only computers with an Intel CPU will be able to use vaapi + +```bash +compressor: + name: vaapi + concurrent: 2 + quality: 90 + device: /dev/dri/renderD128 + driver: iHD +``` + ### Tuning Detectors Potentially, the Visual Mesh had positive results after training, but when used on a robot it performed poorly. In this case, the detectors may need tuning. @@ -98,11 +163,11 @@ Potentially, the Visual Mesh had positive results after training, but when used 2. SSH onto the robot. 3. Enable NUsight messages on the robot by running - ```bash - nano config/NetworkForwarder.yaml - ``` +```bash +nano config/NetworkForwarder.yaml +``` - and set `message.vision.Balls` and `message.vision.Goals` to `true`. +and set `message.vision.Balls` and `message.vision.Goals` to `true`. 4. Run NUsight using `yarn prod` on a computer. Set up NUsight using the [Getting Started page](/guides/main/getting-started) if necessary. 5. Run `./visualmesh` on the robot. From 28f794cab889f76e18e417e6dd0f2631dc189ebf Mon Sep 17 00:00:00 2001 From: LachlanCourt Date: Tue, 19 Jul 2022 10:19:34 +1000 Subject: [PATCH 2/6] Tidy up content --- .../01-main-codebase/02-maintaining-subsystems.mdx | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx index 21f647df7..12d154072 100644 --- a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx +++ b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx @@ -50,6 +50,8 @@ The resulting network should be exported to a [yaml](https://yaml.org/) file and #### Running OpenCL not on the robot +The Visual Mesh works best running the OpenCL framework, however most operating systems are not set up to run this by default. Whilst running the Visual Mesh on CPU is possible, sometimes it is best to replicate an environment as close to what is running on the robot as possible. Follow the steps below to set up OpenCL on a device other than a robot. + 1. Install the OpenCL SDK for your device from the [Intel website](https://www.intel.com/content/www/us/en/developer/tools/opencl-sdk/choose-download.html) 2. Create a temporary directory and install the pre-built OpenCL drivers @@ -67,7 +69,7 @@ sudo apt install ./*deb 3. Change the `VisualMesh.yaml` configuration "engine" node to `opencl` -4. Build the binary and then run with the following command to bind the PC drivers to the docker image +4. Build the binary and then run with the following command to bind the PC drivers to the docker container ```bash ./b build @@ -80,16 +82,15 @@ The vision system cannot work optimally if the cameras are not calibrated correc An automatic camera calibration tool is available in the NUbots repository. See the [camera calibration guide](/guides/main/camera-calibration) to find out how to use this tool. - ### Fake Camera To run the fake camera module, an images directory needs to exist inside the `data` directory of the module. 1. Create an `images` directory and add several images with filenames matching `imageXXXXXX.jpg` where "XXXXXX" refers to any numerical digits -2. Change the `image_folder` node to the location that the data is stored in the docker image: `/home/nubots/NUbots/module/input/FakeCamera/data/images` +2. Change the `FakeCamera.yaml` configuration "image_folder" node to the location that the data is stored in the docker image, which should be something like this: `/home/nubots/NUbots/module/input/FakeCamera/data/images` -3. Build and run a role with the fake camera in the position in the list that the regular camera module normally installs +3. Build and run a role with the fake camera module in the heirarchical position in the list that the regular camera module would normally be ### Testing @@ -127,7 +128,7 @@ If you would like to see the Visual Mesh output in NUsight, you will need to log ### Image Compressors -In order to send data to NUsight via the network, images need to be compressed. Depending on the environment different compressors can be used. [TurboJPEG](https://github.com/libjpeg-turbo/libjpeg-turbo) is the compressor that can be used on all platforms, whilst [vaapi](https://01.org/linuxmedia/vaapi) is the preferred compressor for devices running on Intel CPUs +In order to send data to NUsight via the network, images need to be compressed. Depending on the environment different compressors can be used. [TurboJPEG](https://github.com/libjpeg-turbo/libjpeg-turbo) is the compressor that can be used on all platforms and is used primarily when submitting docker images for virtual competitions as the processor type cannot be guaranteed. [VAAPI](https://01.org/linuxmedia/vaapi) is the preferred compressor for devices running on Intel CPUs, which support the framework. The configuration for either of these belongs in `ImageCompressor.yaml` and is as follows @@ -142,7 +143,7 @@ compressor: #### Vaapi -Only computers with an Intel CPU will be able to use vaapi +Only computers with an Intel CPU will be able to use VAAPI ```bash compressor: From 5ac2cf41afb8afc05082bac3cf9d74e0d53e4cbc Mon Sep 17 00:00:00 2001 From: LachlanCourt Date: Tue, 19 Jul 2022 10:23:58 +1000 Subject: [PATCH 3/6] Add myself as an author :) --- .../03-guides/01-main-codebase/02-maintaining-subsystems.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx index 12d154072..5bdb61b1e 100644 --- a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx +++ b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx @@ -6,6 +6,7 @@ description: How to maintain subsystems within the main codebase. slug: /guides/main/maintaining-subsystems authors: - Ysobel Sims (@ysims) + - Lachlan Court (@LachlanCourt) --- This page details how to maintain various subsystems within the main codebase. From 3242b1c80eee26b6d6e25688b9bce105a36b7b71 Mon Sep 17 00:00:00 2001 From: LachlanCourt <80299053+LachlanCourt@users.noreply.github.com> Date: Tue, 19 Jul 2022 10:43:33 +1000 Subject: [PATCH 4/6] Change code block language to yaml from bash --- .../03-guides/01-main-codebase/02-maintaining-subsystems.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx index 5bdb61b1e..7380600e5 100644 --- a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx +++ b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx @@ -135,7 +135,7 @@ The configuration for either of these belongs in `ImageCompressor.yaml` and is a #### TurboJPEG -```bash +```yaml compressor: name: turbojpeg concurrent: 2 @@ -146,7 +146,7 @@ compressor: Only computers with an Intel CPU will be able to use VAAPI -```bash +```yaml compressor: name: vaapi concurrent: 2 From 72ff32c84cf52aa49a9769d5c0a8cabae97d0233 Mon Sep 17 00:00:00 2001 From: LachlanCourt Date: Tue, 19 Jul 2022 10:48:16 +1000 Subject: [PATCH 5/6] Formatting --- .../02-maintaining-subsystems.mdx | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx index 7380600e5..24cd553e1 100644 --- a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx +++ b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx @@ -137,9 +137,9 @@ The configuration for either of these belongs in `ImageCompressor.yaml` and is a ```yaml compressor: - name: turbojpeg - concurrent: 2 - quality: 90 + - name: turbojpeg + concurrent: 2 + quality: 90 ``` #### Vaapi @@ -148,11 +148,11 @@ compressor: ```yaml compressor: - name: vaapi - concurrent: 2 - quality: 90 - device: /dev/dri/renderD128 - driver: iHD + - name: vaapi + concurrent: 2 + quality: 90 + device: /dev/dri/renderD128 + driver: iHD ``` ### Tuning Detectors From 4e588e34b345a0c4fa19b604fb556f9e6a53e2f2 Mon Sep 17 00:00:00 2001 From: LachlanCourt Date: Fri, 22 Jul 2022 15:52:48 +1000 Subject: [PATCH 6/6] Fix formatting error --- .../01-main-codebase/02-maintaining-subsystems.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx index 24cd553e1..f22ee7235 100644 --- a/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx +++ b/src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx @@ -165,11 +165,11 @@ Potentially, the Visual Mesh had positive results after training, but when used 2. SSH onto the robot. 3. Enable NUsight messages on the robot by running -```bash -nano config/NetworkForwarder.yaml -``` + ```bash + nano config/NetworkForwarder.yaml + ``` -and set `message.vision.Balls` and `message.vision.Goals` to `true`. + and set `message.vision.Balls` and `message.vision.Goals` to `true`. 4. Run NUsight using `yarn prod` on a computer. Set up NUsight using the [Getting Started page](/guides/main/getting-started) if necessary. 5. Run `./visualmesh` on the robot.