Code intel. sometimes not supported on non-UTF8 files

[beyang]

New issue description by @slimsag:

Most language servers only support UTF8-encoded files, so code intelligence does not work on such files generally. It would be nice if we could have e.g. some intermediate layer convert all text files to UTF8 so that language servers only need to deal with UTF8 and things 'just work' for all non-UTF8 file encodings.

---

Old issue description:

"Just tried this out on some of our code bases. It appears to fail to generate snippets/highlights for all files that contain non-Unicode text, e.g. "Müller" in ISO-8859-1. Known issue?"

From https://news.ycombinator.com/reply?id=16124304

[slimsag]

It's not clear to me if this user was experiencing issues with viewing non-UTF8 files, or interacting with them.

• If it was the former (viewing), then this issue is fixed.
• If it was the latter (interacting), then the problem is much harder to solve. For example, language servers may need to convert text to UTF8 before interacting with it.

I will reword the issue description and tag this as a feature request to reflect point 2 above.

[felixfbecker]

I don't think LSP has support for non-Unicode files at all - since it's JSON-based and text documents are transferred as JSON strings, doesn't it require all documents to be in Unicode? Afaik JSON strings are always Unicode (it may support non-UTF8 like UTF16 fine though, but not ISO-8859-1 as in the OP)

[slimsag]

LSP is actually a bit funny about this, it transfers text as UTF8 but asks that ranges be measured in UTF16 codepoints: microsoft/language-server-protocol#376 (also in the spec if you search for utf).

But regardless, I certainly think we would not handle this at the LSP or langserver level. If anything, we could convert non-UTF8 documents to UTF8 prior to sending them to langservers --- but that would add a lot of complexity I am unsure we would even want.

I do not think this is worth looking into right now.

[stale]

Please post an update or close this with an explanation if it is no longer relevant.
This will be closed automatically if there is no more activity.

[stale]

Stalebot is closing this due to inactivity.