META - UTF-8 and BiDi support for the various languages

[XVilka]

• Support in the comments (including the ascii graph mode ascii art graphs doesnt display correctly when oob rendering radare2#1098, Feature-request: utf8 for VV-graphs radare2#1675, Graphs-related issues radare2#1566)
• Store the flags space and flag names in the SDB Store the flags and flagspace in the sdb using base64 radare2#2389
• Fix glitches in visual mode Wrong handling of the Chinese comments in Visual mode radare2#2359
• Printing Unicode strings (including another formats (UTF-16, UTF-32, etc) Printing Unicode Strings radare2#926
• Support in the panels https://asciinema.org/a/17305
• V! panels and Unicode (UTF-8)
• Unicode compatibility on Windows Unicode compatibility on Windows radare2#7235
• p= commands and Unicode (UTF-8) Support scr.utf8=true in charts output (p=) radare2#14351
• p- commands and Unicode (UTF-8) Support Table API and scr.utf8=true in p-h command radare2#14352
• Support in the strings operation (including the search operations)
• String references with emojis shown in disasm
• Support for the various code pages esp. code page 437, windows-1252, iso 8859-5?
• Support for right-to-left scripts e.g. Arabic, Hebrew, Persian ... with/without console support
• Support for character shaping e.g. for Arabic, Persian ... with/without console support
• Support for combining characters e.g. diacritics esp. in graph mode
• Graceful fallback for error cases esp. in graph mode

[XVilka]

For the RTL part - it supports Arabic filenames and comments but fails only in direction:

[schrotthaufen]

#include <stdio.h>

int main(void) {
        printf("ÄÖæxßf®\n");
        return 0;
}

Actual result:
[0x00400410]> fs strings
[0x00400410]> f
0x004005a4 13 str.____x__f

Expected result:
[0x00400410]> fs strings
[0x00400410]> f
0x004005a4 13 str.ÄÖæxßf®\n

[radare]

flags cant contain strange chars, and by strange i mean non-7bit-ascii

thats how the r_name_filter() works

not going to fix/change this before the release. flags needs to be rewritten to use sdb

On 16 Apr 2015, at 17:52, schrotthaufen [email protected] wrote:

#include <stdio.h>

int main(void) {
printf("ÄÖæxßf®\n");
return 0;
}
Actual result:
[0x00400410]> fs strings
[0x00400410]> f
0x004005a4 13 str.____x__f

Expected result:
[0x00400410]> fs strings
[0x00400410]> f
0x004005a4 13 str.ÄÖæxßf®\n

—
Reply to this email directly or view it on GitHub https://github.com/radare/radare2/issues/2032#issuecomment-93769332.

[XVilka]

@radare updated the bug, thanks!

[XVilka]

Btw, Arabic comments are working in the mlterm:

[holdsworth]

In Hebrew, using XTERM terminal - the comments are being represented, however they are written the other around as a problem in XTERM in general that the first character is in the left and the next one is afterwards, ltr instead of rtl(reverts the text).

Example given:

1. proper:
   זאת הערה
2. how it is actually represented:
   הרעה תאז

As written in xterm/README(https://github.com/joejulian/xterm/blob/master/README.i18n) 👍
"Xterm does not support bi-directional or RTL languages such as Hebrew
and Arab."

From a little bit of research I am taking a look into another console which might offer the support for rtl languages and it will be more correct to operate r2 under that console

[XVilka]

@holdsworth try mlterm

[holdsworth]

@XVilka mlterm doesn't represent Hebrew at all, with Konsole it works perfectly fine for some reason.

[radare]

please update the checkboxes

[holdsworth]

please be more precise

[radare]

cc @kazarmy

[Maijin]

CC @queenp

[kazarmy]

I've run r2 in a (Linux) Emacs shell and it works fine for RTL and Arabic shaping (visual mode is unusable though). For consoles that don't support RTL, implementing a full-fledged Unicode Bidirectional Algorithm in r2 appears to require humongous and probably-not-worth-it effort but a simple algorithm that reverses Arabic and Hebrew character sequences shouldn't be hard to do. An Arabic shaping algorithm shouldn't be hard to do either.

[XVilka]

May be have an option to work with FriBiDi somehow for bidirectional texts. But first, all visual modes should be fixed to properly work with Unicode and RTL chars.

[kazarmy]

May be have an option to work with FriBiDi somehow for bidirectional texts.

Yep. Output of:

r2 -c 'iz~Arabic:2' -q bins/pe/testapp-msvc64.exe | fribidi --nobreak

appears promising (testapp-msvc64.exe is an r2r binary).

[XVilka]

We need better page for BiDirectional text support across terminals, like we did for TrueColors. So it will be easier to push terminal developers and easier track the progress.

[XVilka]

radareorg/radare2#9608 is also related

[XVilka]

@Vane11ope can you please try to enable the unicode reflines and stuff in the visual panels code?

[XVilka]

@cyanpencil @Vane11ope @kazarmy please help to review the current state, and what is needed to be done.

[cyanpencil]

@kazarmy do you for any chance have a binary on which we can test the things you just posted?

Or did you test them by inserting a comment in the disasm?

[kazarmy]

Try looking at the r2r bins bins/pe/testapp-msvc64.exe, bins/elf/strenc and bins/elf/strenc-guess-utf32le. They don't cover all the things posted though and could be more comprehensive.

The test binaries were produced using string literals (C / C++).

Btw, there are some UTF utility functions in libr/util/utf8.c / utf16.c / utf32.c. Better not to have duplicate code.

I see you did:

• Support for fullwidth characters in graph mode

Thanks!

[Maijin]

merged @kazarmy checkbox with main post

[cyanpencil]

Btw, there are some UTF utility functions in libr/util/utf8.c / utf16.c / utf32.c. Better not to have duplicate code.

Unfortunately I knew the existence of that file, but did not read it throughly and re-implemented function r_utf8_decode... will fix it in the next pr

Thanks for the heads up!

[XVilka]

From what I know shaping and diacritics will require dependency on ICU, no less. Those are very complex algorithms and depend highly on the language.

[kazarmy]

Actually at least for Arabic, shaping is not that complex. Remember that in real life shaping has to be done manually by kids.

[XVilka]

For a future reference - mintty/mintty#837

[XVilka]

This library seems also interesting https://github.com/JuliaStrings/utf8proc

[XVilka]

See also radareorg/radare2#12629

[XVilka]

See https://terminal-wg.pages.freedesktop.org/bidi/ for details/proposal about BiDi

[XVilka]

So it started: now implemented in VTE: https://terminal-wg.pages.freedesktop.org/bidi/implementations.html#vte

See also https://gist.github.com/XVilka/a0e49e1c65370ba11c17

[XVilka]

Note, that with the release of GNOME 3.34 the support of BiDi is available in Gnome Terminal out of the box, which makes testing/implementing it in the other programs, such as radare2 way easier. FYI @deepakchethan

[ret2libc]

This issue has been moved from radareorg/radare2 to radareorg/ideas as we are trying to clean our backlog and this issue has probably been created a long while ago. This is an effort to help contributors understand what are the actionable items they can work on, prioritize issues better and help users find active/duplicated issues more easily. If this is not an enhancement/improvement/general idea but a bug, feel free to ask for re-transfer to main repo. Thanks for your understanding and contribution with this issue.