Serial Monitor only allows ASCII characters

[WebDust21]

What kind of issue is this?

• PlatformIO Core.
  If you’ve found a bug, please provide an information below.

---

Configuration

Operating system: Ubuntu 22.04 LTS x64

PlatformIO Version (platformio --version): PlatformIO Core, version 6.1.11a2

Description of problem

Characters >127 cannot be used with the Serial Monitor. This is a full-stop problem for any sort of custom filters requiring use of the full byte range (0-255) of characters.

After banging my head into this issue for several hours, I have concluded that the issue is upstream of the filter extensibility (i.e. in DeviceMonitorFilterBase), as the actual raw serial characters >127 are all replaced with U+FFFD (Unicode Replacement Character), making it totally impossible to write a custom filter for these usecases.

Steps to Reproduce

1. Use the Hexlify filter ("monitor_filters = hexlify") in "platformio.ini"
2. Program an MCU to send characters in the full 0-255 byte range via the serial port
   In my case, I used this effective pseudocode in the MCU:

char value = 0;
void timer() {
    sendByte(value++);
}

Actual Results

Only characters 0-127 can be used.

--- Terminal on /dev/ttyUSB0 | 19200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H

0000 | 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
0010 | 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F
0020 | 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F
0030 | 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F
0040 | 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F
0050 | 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F
0060 | 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F
0070 | 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F
0080 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
0090 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00A0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00B0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00C0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00D0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00E0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00F0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 

Expected Results

The complete serial 0-255 byte range is usable.

--- Terminal on /dev/ttyUSB0 | 19200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H

0000 | 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
0010 | 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F
0020 | 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F
0030 | 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F
0040 | 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F
0050 | 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F
0060 | 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F
0070 | 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F
0080 | 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 
0090 | 90 91 92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F 
00A0 | A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF 
00B0 | B0 B1 B2 B3 B4 B5 B6 B7 B8 B9 BA BB BC BD BE BF 
00C0 | C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 CA CB CC CD CE CF 
00D0 | D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 DA DB DC DD DE DF 
00E0 | E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 EA EB EC ED EE EF 
00F0 | F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FB FC FD FE FF

Additional info

Only took about 4 hours of time + help from an accomplished buddy to eventually figure out how to get the desired result: If I specify "monitor-encoding: latin1", I can for the first time in my life actually get unaltered serial data into the filter.
Worth noting that "monitor-encoding: hexlify" is worse than "monitor-encoding: utf8", as the "invalid" characters get discarded before getting to the filter in the first place.

In view of not breaking existing filters, I would suggest adding "rx_byte" and "tx_byte" extensibilities to DeviceMonitorFilterBase , where data can be passed to a custom filter via bytearray (without any encoding nonsense), instead of utilizing the Python string type. This would completely eliminate all of the headbanging around trying to simply get the monitor to pass the actual serial data to the custom filter, it would also not affect any existing filters, and it would also make writing custom filters much easier.

[ivankravets]

Does it work with pio device monitor --raw?

[WebDust21]

Does it work with pio device monitor --raw?

No. I get the following error:

Warning! The `monitor_raw` and `monitor_filters` options cannot be used simultaneously for the `_[MCS51]_` environment in the `platformio.ini` file. The `monitor_filters` option will be disabled to avoid conflicts.

[ivankravets]

Ahh.. You the issue related with hexlify. So, please forward this issue to the original package which PlatformIO uses for device monitor => https://github.com/pyserial/pyserial/issues

You can also provide a pull request, see code https://github.com/pyserial/pyserial/blob/master/serial/tools/miniterm.py

[WebDust21]

Hexlify is not the problem, it's simply a stock PIO tool that very easily demonstrates the problem originating within the PlatformIO DeviceMonitorFilterBase extension.

[kazetsukaimiko]

@ivankravets Full disclosure I'm the "accomplished buddy" mentioned above-
Is this really appropriate to close as "done"? Some improvements could be made PIO side:

1. Alternative interfaces could be provided with byte arrays as arguments instead of parsed strings. I'm not sure why string decoding by default was viewed as appropriate in this case, it doesn't seem to make sense. Decoding the stream is going to mangle it.
2. Documentation could be updated so that people who do want to parse byte arrays have an option. Additionally latin1 could be either aliased as raw or an alternative raw could be provided that makes it clear what to use for this use case.

I also struggle to understand what issue it is you think we should be passing on to pyserial / miniterm, and even if there's a case there, why this issue can be closed as "done" until that upstream work is finished. This manifests as behavior in PIO regardless of where the source is- I would normally assume an issue like this stays open and gets linked to the pyserial issue, tracking it until patched and your version of pyserial is updated.

[ivankravets]

Sorry, I've just reopened this issue. If you have any ideas on how to fix it, please provide PR to this repository or https://github.com/platformio/platformio-docs

[kazetsukaimiko]

First, the documentation PR:
platformio/platformio-docs#317

What I'm unsure of is why latin1 worked for us when parsing byte-by-byte. Other environments, this may vary.

[kazetsukaimiko]

I think latin1 happens to work because it is explicitly 8-bit, versus something multi-byte like utf-8.
https://en.wikipedia.org/wiki/ISO/IEC_8859-1

[ivankravets]

Indeed, the hexlify is implemented on the PlatformIO Core side. Sorry, I forgot about it. See https://github.com/platformio/platformio-core/blob/develop/platformio/device/monitor/filters/hexlify.py

Could try pio device monitor -f direct -f hexlify?

[WebDust21]

Could try pio device monitor -f direct -f hexlify?

I commented out all "monitor" configs in the "platformio.ini" file with the exception of "monitor_speed = 19200", and ran the above command in the PIO terminal:

pio device monitor -f direct -f hexlify
--- Terminal on /dev/ttyUSB0 | 19200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H

0000 | ?? 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 
0010 | 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 
0020 | 1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 
0030 | 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 
0040 | 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 
0050 | 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 
0060 | 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 
0070 | 6F 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 
0080 | 7F ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
0090 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00A0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00B0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00C0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00D0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00E0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
00F0 | ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
0100 | ?? 00 01 02 03 04 05 06 

[ivankravets]

The issue is linked to PySerial module which we use for the device monitor. It always decodes incoming bytes using configured --encoding. See https://github.com/pyserial/pyserial/blob/master/serial/tools/miniterm.py#L514

Nevertheless, we extended our API, and DeviceMonitorFilterBase receives set_running_terminal(terminal) now. You have access to the running terminal now and you can override "encoding" and other things in the runtime. See example of "hexllify" filter https://github.com/platformio/platformio-core/blob/develop/platformio/device/monitor/filters/hexlify.py

[kazetsukaimiko]

Thanks @ivankravets . Is platformio/platformio-docs#317 no longer necessary / obsolete with this change?

[ivankravets]

I updated that docs page (fixed URL). Thanks!