Text encoding detection

[vranki]

Traditional IRC networks don't specify which text encoding is used. This has led to situation where some users use UTF-8, but some still use Latin-1 or other encodings. This is pain for non-english speaking users as clients must be able to detect which encoding is used by other users.

I suggest:

• Add option to enable automatic recode of character sets (most importantly Latin1, others as needed) to UTF-8. This could be enabled by default on selected networks.
• Add option to set outgoing character set to something else than UTF-8 per channel.

For example Irssi does this really well.

[vranki]

For implementation there are 2 approaches:

• KISS lookup table to replace invalid UTF-8 chars with UTF-8 counterparts.
• Use a library such as iconv-lite to convert invalid UTF-8 strings to UTF-8.

I believe one fallback (ISO-8859-15) would be enough for almost everyone but for enhanced version user could define the fallback charset per-channel.

[kaiyou]

We implemented this using the node-irc encoding option: TeDomum@6f556eb

It is far from perfect and the heuristics sometimes backfire, but it does 99% of the job.

[vranki]

If I understand correctly, it sets encoding for server connection. It does not prevent clients from using any other encoding, unless the server has some logic for it.

[kaiyou]

The option is weirdly named in node-irc, but it does enable heuristic detection of other clients encoding and automatic transcription to utf8 (default encoding for JS strings).

[vranki]

Just checked how this could be done client-side. Looks like it's not possible as all invalid chars come as 65535's and it's not possible to distinguish between ä's, ö's and other problem chars.

[vranki]

This has now been implemented and works fine. Thanks!

[zouppen]

It has difficulties on IRC messages containing "mIRC" colour codes leading to double encoding issue (e.g. ä translates to À)

See the following Matrix message ($1591180123573904vFEHU:irc.snt.utwente.nl). It contains doubly encoded UTF-8 because the message contains some colour codes.

{
  "content": {
    "body": "/// Nytsoi pÀivitetty: Kaaosradio 24h Ke - Techno/Electro",
    "format": "org.matrix.custom.html",
    "formatted_body": "<font color=\"#7F0000\">///</font> Nytsoi pÀivitetty: <b>Kaaosradio 24h Ke - Techno/Electro</b>",
    "msgtype": "m.text"
  },
  "event_id": "$1591180123573904vFEHU:irc.snt.utwente.nl",
  "origin_server_ts": 1591180123623,
  "sender": "@_ircnet_kaaosradio:irc.snt.utwente.nl",
  "type": "m.room.message",
  "unsigned": {
    "age": 122
  },
  "room_id": "!rPDKWxyLDIMEdLFtXF:irc.snt.utwente.nl"
}