fix #5294: 1. reopen file with another encoding; 2. set default encod…

[a1994846931931]

…ing;

Signed-off-by: Cai Xuye [email protected]

• Register CQ

What it does

1. Add "editor.defaultEncoding" configuration, so that one can specify a proper default encoding for his project;
2. Add encoding item to the status bar, which shows the current encoding used;
3. Pump out a list of encoding for one to choose from when clicking on the encoding status item, or command 'textEditor.change.encoding' is executed. In addition, a guessed encoding will be shown on top of the list if possible.

Added dependencies:

1. "iconv-lite" of version 0.4.23 is used to encode/decode files.
2. The same version of ""jschardet" (1.6.0) as the one on VSCode is used to guess the encoding of a file based on its content.

How to test

Test File

Encoding of each provided file is used as its filename

• GB2312.txt
• Shift-JIS.txt
• Or create your own files

TEST1

Make sure that the default encoding works fine.

Preparation:

• Set files.encoding to utf8
• Prepare some files that are encoded in UTF-8

Process:

1. Open a file that's encoded in UTF-8 and check its content;
2. Create a file, edit it, save it, close it, and reopen it. Just make some modification as you like, and check if the editor is working properly;
3. Also, the encoding item on the status bar should show "UFT-8";
4. Use another test file and repeat.

TEST2

Make sure that the "Reopen with Encoding" action functions well.

Preparation:

• Set files.encoding to utf8
• Prepare some files that are encoded in something different from UTF-8

Process:

1. Open a test file, for example, GB2312.txt, and check it's content. There should be some weird chars like � in the editor, which shows that the file is not decoded correctly;
2. Run "Editor: Change File Encoding" command or click the block on the status bar that shows "UTF-8", and choose "Reopen with Encoding" action;
3. In the selection list, the first item should be a guessed encoding item. The encoding is guessed from the file's content;
4. If you open files that are provided above, the guessed encoding should be correct (the correct encodings are the same as their filenames). If other files are used, the guessed encoding could be either right or wrong, which is normal. But basically, guessed encoding have a better chance to win if more content is put into the file;
5. Choose the right encoding, either from the guessed item or from your own select, and check if it's reopened correctly;
6. Use another test file and repeat.

TEST3

Make sure that the "Save with Encoding" action functions well.

Preparation:

• Set files.encoding to utf8
• Prepare some files that are encoded in something different from UTF-8

Process:

1. Follow 1-5 of TEST2, so that the test file is opened in a correct encoding;
2. Run "Editor: Change File Encoding" command or click the block on the status bar that shows "UTF-8", and choose "Save with Encoding" action;
3. Then choose "UTF-8" or any other encoding you want to test, and check if the encoding of the file is changed correctly. You can do the check by reopening it, opening it with other software, or executing file -i ${FILENAME} command on linux (though it could be wrong);
4. Also, you can make some modification, and check if "Save with Encoding" action could handle dirty file correctly (apply content changes and encoding change as well)
5. Use another test file and repeat.

TEST4

Make sure that files.encoding configuration works.

Preparation:

• Prepare some files that are encoded in something different from UTF-8

Process:

1. Change files.encoding (Files -> encoding) to the encoding of your test file;
2. Open the test file, do some editing, and check if it works fine;
3. Use another test file and repeat.

NOTE
Currently, I have only tested it on Ubuntu.

Review checklist

• as an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• as a reviewer, I agree to behave in accordance with the review guidelines

[a1994846931931]

Demos are shown below:

[akosyakov]

This PR introduces implicit dependencies between editor and filesystem extensions. It should be refactored to remove coupling. These extensions cannot depend each other. There should be some sort of intermediate API to make it work.

[a1994846931931]

@akosyakov Thanks for your advice! I removed the coupling between editor and filesystem extensions by introducing 'Resource'. Codes are updated. Could you please make a further review?

This PR introduces implicit dependencies between editor and filesystem extensions. It should be refactored to remove coupling. These extensions cannot depend each other. There should be some sort of intermediate API to make it work.

[a1994846931931]

All commits merged into one and rebased to solve the conflict.

[a1994846931931]

@akosyakov Hi, I fixed all but the 'IEncodingSupport' and 'overwriteEncoding' (please see comments above). Thanks for your precious advice. And please let me know if further modifications are ought to be made.

[a1994846931931]

@akosyakov Could you please take a further look at this? The PR has been up here for over two months, so I want to focus on it in the following few days and hopefully meet your requirement soon.

[akosyakov]

Could you please take a further look at this? The PR has been up here for over two months, so I want to focus on it in the following few days and hopefully meet your requirement soon.

We have many PRs and work sorry. PRs addressing VS Code API compatibility issues gets highest priority right now on my list. I will have a look when i have time.

[akosyakov]

Could someone test against similar behaviour in VS Code? We also need to look out for race conditions with saving, i.e. changing encoding and starting typing immediately or with delay and see whether it leads to any troubles.

[akosyakov]

Code-wise it looks good to me now. But I don't have time to verify according to #5371 (comment).

[akosyakov]

@a1994846931931 Could you define tests for this PR? We will test that changes don't break anything without changing encoding and then use your tests to test encoding/decoding plus check whether expectations of tests are aligned with behaviour of VS Code. If everything is fine we land it.

Also just FYI: https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#checklist-build-and-test

[a1994846931931]

@a1994846931931 Could you define tests for this PR? We will test that changes don't break anything without changing encoding and then use your tests to test encoding/decoding plus check whether expectations of tests are aligned with behaviour of VS Code. If everything is fine we land it.

Also just FYI: https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#checklist-build-and-test

@akosyakov I have rewritten the PR message, based on the new PR template. So How to test is updated, too. One problem, however, is that I have only tested it on Ubuntu. Maybe someone could have it tested on other platforms, following my tests list?

[akosyakov]

@westbury @kittaakos If you can try tests from How to test section on windows it would help. I will check them on linux.

@westbury @kittaakos maybe we can have os-windows team? 🤔

[akosyakov]

Code-wise it looks good, I've tested according How to test and it seems to behave well and does not break existing behaviour.

@vince-fugnitto please can you clarify your suspicious about CQs? We cannot merge without it.

[vince-fugnitto]

@vince-fugnitto please can you clarify your suspicious about CQs? We cannot merge without it.

I think it's alright, the original CQ covered both the encoding.ts and textfile.ts files.