Add support for different encodings

[Raudius]

• Resolves: Can't show Chinese word corrlecty, Maybe it's because GB2312 and GBk encoding is not supported #405
  • Android client stuck on loading Ansi Latin 1 and UTF16 .md and .txt files with German umlauts #824
  • shift-jis(shift_jis, cp932, sjis-win) text file garbled when view web browser direct. #1633
• Target version: master

Summary

Previously we would just assume files would use UTF-8. Now we make sure to convert files to UTF-8 when we open them. Without configuration just a small set of encodings are supported:

 'UTF-8', 'GB2312', 'GBK', 'BIG-5', 'SJIS-win', 'EUC-JP', 'Windows-1252', 'ISO-8859-15', 'ISO-8859-1', 'ASCII'

More encodings can be configured via the PHP config value: mbstring.detect_order

[vinicius73]

As we are using mbstring functions, it is important to define it in composer.json
Although it is already required by server.

https://github.com/nextcloud/server/blob/9e1d4d17ec792d8abcec2f9262472f5bb7a43acf/composer.json#L21-L29

--- a/composer.json
+++ b/composer.json
@@ -17,7 +17,8 @@
         }
     ],
     "require": {
-      "php": "^7.3|^8.0"
+      "php": "^7.3|^8.0",
+      "ext-mbstring": "*"
     },
     "scripts": {

What do you think?

[juliusknorr]

@Raudius Could you do a rebase to update the branch rather then merge? That would make the commit history linear and a bit cleaner :)

[juliusknorr]

Found two more reportings on other encoding issues when doing a bit of triage:
#1633
#824

[juliusknorr]

Small nitpick on the code style 😉 I guess we currently don't run php-cs-fixer on the tests directory.

Suggested change

	$detected_encoding = $this->encodingService->detectEncoding(file_get_contents($file));
	if ($this->isSupportedEncoding($encoding)) {
	$this->assertNotNull($detected_encoding);
	}
	$original_order = mb_detect_order();
	$this->assertNotFalse(mb_detect_order($encoding));
	$detected_encoding = $this->encodingService->detectEncoding(file_get_contents($file));
	$this->assertEquals($encoding, $detected_encoding);
	mb_detect_order($original_order);
	$detectedEncoding = $this->encodingService->detectEncoding(file_get_contents($file));
	if ($this->isSupportedEncoding($encoding)) {
	$this->assertNotNull($detectedEncoding);
	}
	$originalOrder = mb_detect_order();
	$this->assertNotFalse(mb_detect_order($encoding));
	$detectedEncoding = $this->encodingService->detectEncoding(file_get_contents($file));
	$this->assertEquals($encoding, $detectedEncoding);
	mb_detect_order($originalOrder);

[juliusknorr]

Just a minor code style comment left on the tests, otherwise LGTM

[juliusknorr]

/backport to stable24

[juliusknorr]

/backport to stable23

[juliusknorr]

/backport to stable22