Stream rm2 output in color

[gregorbg]

Hello,

during the implementation of reStream for reMarkable2 one person posted a comment that apparently you can stream the display in full color: #28 (comment)

Would it be possible to integrate this feature into the library, so that for example the "Highlighter" pen actually appears yellow instead of gray (like it does when the PDF is exported)?

Thanks and kudos for this awesome project!

[rien]

Hi there, we plan on supporting rm2fb, so in the future this could indeed be possible.

[gregorbg]

Thanks for the prompt feedback! This may be a noob question, but what is the relation between rm2fb and this project?

Most notably, reStream allows me to read from the device/display while rm2fb claims in its README that it is "code for drawing to the [device]". That sounds like two diametrically opposed purposes in terms of software.

[LinusCDE]

Not sure if this is even possible. Seems that they changed the pixel format from rgb565le (basicially rgb in 2 bytes instead of 3) to gray8. That would make the framebuffer never hold any color information at all.

I personally like that approach but can see the need for the highlighter pen as well. Since basically all other apps use rm2fb and it usually translates rgb565le to gray8, there would be a possibility of preserving color there somehow. Though this seems like unnecessary work (there are other more urgent improvements imo) and I'm not sure whether we could get xochitl to even use it (maybe replace some QT lib?).

[LinusCDE]

Most notably, reStream allows me to read from the device/display while rm2fb claims in its README that it is "code for drawing to the [device]". That sounds like two diametrically opposed purposes in terms of software.

The primary purpose of rm2fb is the drawing indeed. In the progress it claims the ownership of the, now userspace, framebuffer to drive the eink. So restream is accessing rm2fbs framebuffer to display it but does not any drawing itself.

[rien]

Can you check if this issue is solved with the latest release?

Thanks for the prompt feedback! This may be a noob question, but what is the relation between rm2fb and this project?

The goal of restream is simply to stream reMarkable's screen to a host computer. rm2fb is just one way to do that (because as Linus said, it is also possible to just read from that framebuffer).

I only own an rM1 so I'm not in any way involved in the development of rm2fb.

[gregorbg]

Hey, great to see that you tried to implement this so quickly! Unfortunately, the new release has several issues.

1. The installation via opkg conflicts with rm2fb.

reMarkable: ~/ opkg install restream
Installing restream (0.0.0-1) to root...
Collected errors:
 * check_conflicts_for: The following packages conflict with restream:
 * check_conflicts_for: 	rm2fb *
 * opkg_install_cmd: Cannot install package restream.

2. When manually scping the binary from this repository after installing rm2fb, I can start reStream.sh on my computer but it does not open the streaming window. Here's the console output:

$ ./reStream.sh
[SSH] true
[SSH] cat /sys/devices/soc0/machine
[SSH] [ ! -f ~/restream ]
[SSH] ./restream
ffplay version n4.3.1 Copyright (c) 2003-2020 the FFmpeg developers
  built with gcc 10.2.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-shared --enable-version3
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Error: No line containing /dev/fb0 in /proc/2788/maps fileB f=0/0
Input #0, rawvideo, from 'pipe:':
  Duration: N/A, bitrate: 525657 kb/s
    Stream #0:0: Video: rawvideo (Y800 / 0x30303859), gray, 1872x1404, 525657 kb/s, 25 tbr, 25 tbn, 25 tbc
    nan M-V:    nan fd=   0 aq=    0KB vq=    0KB sq=    0B f=0/0

If you need more output or system stats, I'll be glad to help!

[LinusCDE]

@suushiemaniac

The installation via opkg conflicts with rm2fb.

The conflict was initially because restream didn't support streaming the framebuffer from rm2fb (only xochilt directly). The current package in testing doesn't have that conflict anymore, so that will be solved in about a week.

When manually scping the binary from this repository after installing rm2fb, I can start reStream.sh on my computer but it does not open the streaming window. Here's the console output:

I think much has changed in the reStream binary. So the stream not starting might be a result of that.

[gregorbg]

Thank you for the reply! Regarding your second remark:

I think much has changed in the reStream binary. So the stream not starting might be a result of that.

Is this something I'm doing wrong or something on your side that has to be fixed?

[LinusCDE]

Is this something I'm doing wrong or something on your side that has to be fixed?

It's just figuring out which script to use. At the moment, the binaries has gotten some breaking changes making it not work with older restream.sh scripts since it now expects the settings be given by the script instead of figuring it out again.

Best is to make sure the binary on your device is the same binary that is in the repo you download/cloned so the script uses the right binary on the reMarkable.

[gregorbg]

Hello! Sorry for the radio silence, I've just now gotten back to plaing around with this again.

It seems that by using the opkg installation I can now combine rm2fb and restream. Awesome! In particular, the start of the ./reStream.sh on my host yields the following output:

[SSH] true
[SSH] cat /sys/devices/soc0/machine
[SSH] [ -f /dev/shm/swtfb.01 ]
[SSH] [ ! -f ~/restream ] && [ ! -f /opt/bin/restream ]
[SSH] PATH="$PATH:/opt/bin/:." restream -h 1872 -w 1404 -b 2 -f /dev/shm/swtfb.01

The part about using the /dev/shm/swtfb.01 buffer leads me to suspect that the streaming is indeed using rm2fb as per https://github.com/rien/reStream/blob/main/reStream.sh#L109. However, even then my stream is in grayscale :( I tried using the highlighter pen on a quick sheet, and it shows as gray. PDFs that have colorful images inside them
also display as grayscale.

For reference, here is the exported PDF with my handwritten remark which clearly retains the color even after exporting.

Is there any other option to enable color streaming?

[LinusCDE]

Not sure either. I have seen some interesting behaviour that apps using the old framebuffer with the rm2fb shim, could read that color back as well. This means that rm2fb likely has multiple kinds of framebuffer with varying information. Though I know enough to know that I don't understand that topic well at all.

I think @raisjn had talked about how to force the pdf rendering engine of xochitl to output color and that this is somehow supported by the gray8 format?!? I'm really confused there as well, but raisjn knows probably a lot more on that matter.

I think his screenshot tool bufshot aims to also extract the coloured output of the framebuffer. Not sure how far this project is atm but when finished, it would sure make for a good reference regarding reading color output or getting xochitl to provide this info at all.

[raisjn]

cc @ddvk

in terms of formats:

• rM1: rgb565le (5 bits for red, little endian)

• rM2: SWTCON internal buffer: Y8 (grayscale 8) (EDIT: unsure if used by restream/rmview)

• rM2: rm2fb's swtfb: rgb565le (same as rM1) - this is actually a QImage from xochitl

• rM2: rm2fb's xofb: rgb565 (same as rM1), but only exposes xochitl's FB:

the color output was a special one off where bokluk gave them a patch to run against xochitl that changed the parameter to pdfium that indicates whether to render in grayscale or not. they then used xofb to expose xochitl's color memory (before it goes to SWTCON) and connected that to reStream for streaming.

[ddvk]

also, iirc the highlighter is not yellow but some gray color, that got rendered transparent.
the rgb patch was only about the pdf rendering

[gregorbg]

Hello again! I have tried activating the patch at https://github.com/ddvk/remarkable-hacks/blob/master/tools/other/rgb_25027_rm2.py and the Python script runs without error, but unfortunately the reStream output is still in grayscale.

What information can I provide to help you debug this issue?

Also, I understand now that the patch only affects the PDF itself, not the highlighter color. But being able to stream color PDFs in color would already be more than sufficient (and quite awesome!) for my purposes.

[ddvk]

Hello again! I have tried activating the patch at https://github.com/ddvk/remarkable-hacks/blob/master/tools/other/rgb_25027_rm2.py and the Python script runs without error, but unfortunately the reStream output is still in grayscale.

What information can I provide to help you debug this issue?

Also, I understand now that the patch only affects the PDF itself, not the highlighter color. But being able to stream color PDFs in color would already be more than sufficient (and quite awesome!) for my purposes.

it worked on my machine. i used xofb to export the framebuffer and then restream as the exported /dev/shm/xofb
also, you sure you ran the patched xochitl?

[gregorbg]

I have just tried again on a clean 2.6.2.75 installation. Here are the steps I followed after updating:

1. scp remarkable:/usr/bin/xochitl ~/rgb/xochitl.26275
2. remarkable-hacks/tools/other/rgb_26272_rm2.py ~/rgb/xochitl.26275 (ran without producing any output messages)
3. scp ~/rgb/xochitl.26275 remarkable:/home/root/xochitl.rgb
4. ssh remarkable (all commands from here are being executed on my remarkable tablet)
5. cp /usr/bin/xochitl /home/root/xochitl.backup
6. systemctl stop xochitl
7. killall xochitl
8. cp /home/root/xochitl.rgb /usr/bin/xochitl
9. systemctl start xochitl

Running reStream afterwards still produces grayscale output. What am I doing wrong?

[ddvk]

you have to export the xochitls qmage buffer, which is done by libxofb and use that in restream (/dev/shm/xofb)

[tspiteri]

I went another route to get color: I filter the buffer before passing it to ffplay. The red color is output as byte 0x55, blue as 0x99, yellow highlight as 0xBB and green highlight as 0xDD, so in my filter I change those 8-bit values to a matching color, and all other 8-bit values to the corresponding gray color.

I also took the opportunity to skip frames when there is no change from a previous frame; this leads to much lower cpu usage by ffplay while not drawing.

Then I changed the reStream.sh script to pipe through my filter before ffplay, and to use pixel format rgb24 instead of gray8.

For my device with 1404×1872 pixels and gray8 format, I used this code for the filter:

use std::io;
use std::io::{Read, Write};
use std::mem;

fn filter(inp: u8, out: &mut [u8; 3]) {
    match inp {
        0x55 => {
            // red pen
            out[0] = 0xd5;
            out[1] = 0x5e;
            out[2] = 0x00;
        }
        0x99 => {
            // blue pen or pink highlighter, use blue
            out[0] = 0x00;
            out[1] = 0x72;
            out[2] = 0xb2;
        }
        0xbb => {
            // green highlighter
            out[0] = 0x00;
            out[1] = 0x9e;
            out[2] = 0x73;
        }
        0xdd => {
            // yellow highlighter
            out[0] = 0xf0;
            out[1] = 0xe4;
            out[2] = 0x42;
        }
        _ => {
            // gray
            out[0] = inp;
            out[1] = inp;
            out[2] = inp;
        }
    }
}

fn main() {
    let frame_pixels = 1404 * 1872;

    let mut stdin = io::stdin();
    let mut stdout = io::stdout();
    let mut in_buffer = vec![0u8; 2 * frame_pixels];
    let (mut current, mut previous) = in_buffer.split_at_mut(frame_pixels);
    let mut out_buffer = vec![[0u8; 3]; frame_pixels];
    while stdin.read_exact(current).is_ok() {
        if current == previous {
            continue;
        }
        for (inp, out) in current.iter().zip(out_buffer.iter_mut()) {
            filter(*inp, out);
        }
        let bytes: &[u8] = bytemuck::cast_slice(&out_buffer);
        if stdout.write_all(bytes).is_err() {
            break;
        }
        mem::swap(&mut current, &mut previous);
    }
}

[dkoenigsberg]

I don't know if this has been discovered already. But I found that if I start reStream, and then on the rm2, initiate screen sharing but don't accept on the desktop app, reStream documents and annotations show up in color!

[mahrud]

For what it's worth, on my 3.6.0.1806 rm2 this is what setting

            bytes_per_pixel=2
            pixel_format="bayer_rggb16be"

looks like:

It's kinda weird, and ffplay seems to consume twice as much cpu decoding it, so I'm sticking to gray16be.

[Lewiscowles1986]

I have a feeling that greyscale can optimize for a literal bit-map of 1 being on and 0 being off or inverse bitmap, which would have 8x less bandwidth requirement than byte encoded 8-bit color, or higher channel multi-byte color. So even palletised transfer would consume more bandwidth than single bit on / off with off-device anti-aliasing