[AE-79] Need robust alternative to Processing for viewing camera feed

[aliphys]

Describe the problem

The Arduino + Processing sketch solution for streaming video from the Portenta Vision Shield, Nicla Vision and GIGA board is very unreliable. Problems include:

• [AE-49] Colour mode mismatch between CameraCaptureRawBytes.ino and CameraRawBytesVisualizer.pde #662
• [AE-49] No video output for Portenta Vision Shield #661
• [AE-49] Chopped up video output from the Nicla Vision #660

See @jacobhylen 's #673 (comment)

To reproduce

1. Upload the CameraCaptureRawBytes.ino sketch.
2. Using either Windows, Linux or macOSX run the CameraRawBytesVisualizer.pde Processing sketch
3. If you are lucky enough, you will see a live feed of the camera. Using Windows will reduce your luck, due to how it handles I/O Portenta Vision Shield Camera demo not running #112 (comment).

Expected behaviour

Regardless of what Arduino board is connected, you should see a stable video feed.

• Without flickering [AE-49] Chopped up video output from the Nicla Vision #660 [Nicla Vision] Camera example not working properly #559 (comment).
• Without seeing a grey screen [AE-49] No video output for Portenta Vision Shield #661 (comment).
• Without a 'Connection timed out' message in Processing [AE-79] Switch useGreyScale value in Processing sketch without changing code #673 (comment).
• Without having to change the useGreyScale boolean variable [AE-49] Colour mode mismatch between CameraCaptureRawBytes.ino and CameraRawBytesVisualizer.pde #662.
• Without having to switch to Linux, or use an external OLED screen [AE-49] No video output for Portenta Vision Shield #661 (comment)
• And without buffer sync issues Fix buffer sync issue in camera raw bytes example #143 (comment).

Proposed Solutions

The current Arduino + Processing sketch is not fit for purpose. We need an alternative solution with the following four features:

• Cross-platform
• Resilient to errors
• Simple to use, with minimal configuration by User
• Light on CPU

Here are two different - yet complimentary - approaches to meet these needs

Approach 1: Barebones Arduino-Python script

The sketch will communicate at 9600 bps. This is very slow, and will have minimal demand on CPU resources ensuring that data is read in a timely manner from the serial port (arduino/arduino-ide#79). Of course, the resolution and frame per second will drop but as a test sketch this is fine to ensure maximum compatibility. The Serial.begin() method includes an option for defining a parity (based on the underlying mbed function), which should be utilised.

ArduinoCore-mbed/cores/arduino/Serial.cpp

Lines 91 to 101 in 8031008

	switch (config & SERIAL_PARITY_MASK) {
	case SERIAL_PARITY_EVEN:
	parity = mbed::SerialBase::Even;
	break;
	case SERIAL_PARITY_ODD:
	parity = mbed::SerialBase::Odd;
	break;
	case SERIAL_PARITY_NONE:
	parity = mbed::SerialBase::None;
	break;
	}

The packets from the Arduino board are then parsed by a Python script. This would make use of cross-compatible libraries for I/O.
The packet should also contain data specifying if the transmission is greyscale or RGB, so that the Python script can identify this. No OS specific routines should be avoided.

Approach 2: WebBLE Camera Dashboard

An alternative approach to send data from the camera to the PC (or any other compatible device) would be over WebBLE. We already have a successful implementation of this for the Nicla Sense ME (see @marqdevx 's tutorial here as well as @sebromero battery status indicator here). A single image would need to be split into smaller chunks, which are then reconstructed on the browser side. While support for WebBLE is limited mainly to Chrome, it is expected to grow. And it works across Windows, Linux and OSX without the need to defined a serial port. Device detection is intuitively done in the browser, where a human-readable name is assigned to the Arduino device.

(M)JPEG compression can allow for larger resolution/FPS together with some level of resilience to error. Native JPEG decompression algorithms is taken for granted in modern browsers. Note that WebBLE support is still experimental.

[aliphys]

Approach 3: WebSerial Dashboard

@facchinm mentioned WebSerial to transfer data from the Arduino Camera to the computer. WebSerial can be over USB or Bluetooth, so is communication-method agnostic. This overcomes the packet size limitations of WebBLE and can use native features in the browser to display the image data. A Github io page could host the page.

[aliphys]

A robust alternative was developed by @sebromero in #822 which meets the requirements specified in this issue 🎉 :

• Cross-platform -> Build upon WebSerial JS (Approach 3) and can run in Google Chrome in Windows, Linux and Mac.
• Resilient to errors -> Instead of continuous data transfer (which is problematic in Windows, see: Portenta Vision Shield Camera demo not running #112 (comment)) defines sendChunk() and sendFrame() methods. Additionally, sendCameraConfig() sends the IMAGE_MODE based on the specified platform.
• Simple to use, with minimal configuration by User -> Compiler directives are used to identify the platform. Manual configuration needs to be made by (un)commenting lines since single platforms may have different cameras. This is currently the best workaround, since automatic detection of the camera is not possible.
• Light on CPU -> Runs stable on Mac and OK on Linux. Some stability issues on Windows, but more a limitation of the OS. Runs continuously on the Nicla Vision without crashing.