Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add content regarding Image Compressor, Fake Camera, and OpenCL #235

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 72 additions & 5 deletions src/book/03-guides/01-main-codebase/02-maintaining-subsystems.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ description: How to maintain subsystems within the main codebase.
slug: /guides/main/maintaining-subsystems
authors:
- Ysobel Sims (@ysims)
- Lachlan Court (@LachlanCourt)
---

This page details how to maintain various subsystems within the main codebase.
Expand Down Expand Up @@ -48,12 +49,50 @@ The resulting network should be exported to a [yaml](https://yaml.org/) file and

3. Add this configuration file to the NUbots repository in the [VisualMesh module](https://github.com/NUbots/NUbots/tree/main/module/vision/VisualMesh/data/config/networks). Replace or add a configuration file depending on the use case of the Mesh - `RobocupNetwork.yaml` is for soccer playing on the real robot and `WebotsNetwork` is for soccer playing in the Webots simulator. View the [Git Guide](/guides/general/git) for information on using Git and submitting this change in a pull request.

#### Running OpenCL not on the robot

The Visual Mesh works best running the OpenCL framework, however most operating systems are not set up to run this by default. Whilst running the Visual Mesh on CPU is possible, sometimes it is best to replicate an environment as close to what is running on the robot as possible. Follow the steps below to set up OpenCL on a device other than a robot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to confirm that these steps are correct with a fresh install so general review is fine but these steps may change

1. Install the OpenCL SDK for your device from the [Intel website](https://www.intel.com/content/www/us/en/developer/tools/opencl-sdk/choose-download.html)

2. Create a temporary directory and install the pre-built OpenCL drivers

```bash
mkdir temp
cd temp
wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-gmmlib_19.1.1_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-igc-core_19.11.1622_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-igc-opencl_19.11.1622_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-opencl_19.14.12751_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/19.14.12751/intel-ocloc_19.14.12751_amd64.deb
sudo apt install ./*deb
```

3. Change the `VisualMesh.yaml` configuration "engine" node to `opencl`

4. Build the binary and then run with the following command to bind the PC drivers to the docker container

```bash
./b build
./b run --mount type=bind,source="/dev/dri",target="/dev/dri" <role_name>
```
Comment on lines +77 to +78
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably mention what <role_name> should be replaced with


### Camera Calibration

The vision system cannot work optimally if the cameras are not calibrated correctly. [The input page](/system/subsystems/input) describes the camera parameters that can be calibrated.

An automatic camera calibration tool is available in the NUbots repository. See the [camera calibration guide](/guides/main/camera-calibration) to find out how to use this tool.

### Fake Camera

To run the fake camera module, an images directory needs to exist inside the `data` directory of the module.

1. Create an `images` directory and add several images with filenames matching `imageXXXXXX.jpg` where "XXXXXX" refers to any numerical digits
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it have to be 6 digits? If so this would help clarify:

Suggested change
1. Create an `images` directory and add several images with filenames matching `imageXXXXXX.jpg` where "XXXXXX" refers to any numerical digits
1. Create an `images` directory and add several images with filenames matching `imageXXXXXX.jpg`, where "XXXXXX" is any 6-digit number, e.g. `000001`, `000002`, etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't have to be 6 digits, I believe it just needs to be enumerated somehow


2. Change the `FakeCamera.yaml` configuration "image_folder" node to the location that the data is stored in the docker image, which should be something like this: `/home/nubots/NUbots/module/input/FakeCamera/data/images`

3. Build and run a role with the fake camera module in the heirarchical position in the list that the regular camera module would normally be

### Testing

After updating the Visual Mesh in the NUbots repository, it should be tested before merging. Refer to the [Getting Started guide](/guides/main/getting-started) for assistance for the following steps.
Expand Down Expand Up @@ -85,9 +124,37 @@ After updating the Visual Mesh in the NUbots repository, it should be tested bef
If you would like to see the Visual Mesh output in NUsight, you will need to log the data and run it back in NUsight using DataPlayback, since the data is too large to send over a network. Use the steps in the [DataLogging and DataPlayback guide](/guides/main/data-recording-playback) to record and playback data. Adjust the instructions for our purpose using the following hints:

- In step 1 of Recording Data, use the [`visualmesh`](https://github.com/NUbots/NUbots/blob/main/roles/visualmesh.role) role to record the data.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- In step 1 of Recording Data, use the [`visualmesh`](https://github.com/NUbots/NUbots/blob/main/roles/visualmesh.role) role to record the data.
- In step 1 of [Recording Data](/guides/main/data-recording-playback#recording-data), use the [`visualmesh`](https://github.com/NUbots/NUbots/blob/main/roles/visualmesh.role) role to record the data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add this to this PR 👍

- In step 2 of Recording Data and step 4 of Playing Back Data, set `message.output.CompressedImage` to `true` and add `message.vision.VisualMesh: true` in both [`DataLogging.yaml](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataLogging/data/config/DataLogging.yaml) and [`DataPlayback.yaml](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataPlayback/data/config/DataPlayback.yaml).
- In step 2 of Recording Data and step 4 of Playing Back Data, set `message.output.CompressedImage` to `true` and add `message.vision.VisualMesh: true` in both [`DataLogging.yaml`](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataLogging/data/config/DataLogging.yaml) and [`DataPlayback.yaml`](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataPlayback/data/config/DataPlayback.yaml).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- In step 2 of Recording Data and step 4 of Playing Back Data, set `message.output.CompressedImage` to `true` and add `message.vision.VisualMesh: true` in both [`DataLogging.yaml`](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataLogging/data/config/DataLogging.yaml) and [`DataPlayback.yaml`](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataPlayback/data/config/DataPlayback.yaml).
- In step 2 of [Recording Data](/guides/main/data-recording-playback#recording-data) and step 4 of [Playing Back Data](/guides/main/data-recording-playback#playing-back-data), set `message.output.CompressedImage` to `true` and add `message.vision.VisualMesh: true` in both [`DataLogging.yaml`](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataLogging/data/config/DataLogging.yaml) and [`DataPlayback.yaml`](https://github.com/NUbots/NUbots/blob/main/module/support/logging/DataPlayback/data/config/DataPlayback.yaml).

- In steps 1, 2 and 5 of Playing Back Data, use the [`playback`](https://github.com/NUbots/NUbots/blob/main/roles/playback.role) role to playback the data, without changes.

### Image Compressors

In order to send data to NUsight via the network, images need to be compressed. Depending on the environment different compressors can be used. [TurboJPEG](https://github.com/libjpeg-turbo/libjpeg-turbo) is the compressor that can be used on all platforms and is used primarily when submitting docker images for virtual competitions as the processor type cannot be guaranteed. [VAAPI](https://01.org/linuxmedia/vaapi) is the preferred compressor for devices running on Intel CPUs, which support the framework.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In order to send data to NUsight via the network, images need to be compressed. Depending on the environment different compressors can be used. [TurboJPEG](https://github.com/libjpeg-turbo/libjpeg-turbo) is the compressor that can be used on all platforms and is used primarily when submitting docker images for virtual competitions as the processor type cannot be guaranteed. [VAAPI](https://01.org/linuxmedia/vaapi) is the preferred compressor for devices running on Intel CPUs, which support the framework.
In order to send data to NUsight via the network, images need to be compressed. Depending on the environment different compressors can be used. [TurboJPEG](https://github.com/libjpeg-turbo/libjpeg-turbo) is the compressor that can be used on all platforms and is used primarily when submitting docker images for virtual competitions as the processor type cannot be guaranteed. [VAAPI](https://01.org/linuxmedia/vaapi) is the preferred compressor for devices running on Intel CPUs which support the framework, like the [CPUs in the robots](/system/hardware/overview#computing).


The configuration for either of these belongs in `ImageCompressor.yaml` and is as follows

#### TurboJPEG

```yaml
compressor:
- name: turbojpeg
concurrent: 2
quality: 90
```

#### Vaapi

<Alert>Only computers with an Intel CPU will be able to use VAAPI</Alert>

```yaml
compressor:
- name: vaapi
concurrent: 2
quality: 90
device: /dev/dri/renderD128
driver: iHD
```

### Tuning Detectors

Potentially, the Visual Mesh had positive results after training, but when used on a robot it performed poorly. In this case, the detectors may need tuning.
Expand All @@ -98,11 +165,11 @@ Potentially, the Visual Mesh had positive results after training, but when used
2. SSH onto the robot.
3. Enable NUsight messages on the robot by running

```bash
nano config/NetworkForwarder.yaml
```
```bash
nano config/NetworkForwarder.yaml
```

and set `message.vision.Balls` and `message.vision.Goals` to `true`.
and set `message.vision.Balls` and `message.vision.Goals` to `true`.
LachlanCourt marked this conversation as resolved.
Show resolved Hide resolved

4. Run NUsight using `yarn prod` on a computer. Set up NUsight using the [Getting Started page](/guides/main/getting-started) if necessary.
5. Run `./visualmesh` on the robot.
Expand Down