Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework utf8 support #856

Open
cor3ntin opened this issue Mar 9, 2018 · 3 comments
Open

Rework utf8 support #856

cor3ntin opened this issue Mar 9, 2018 · 3 comments
Labels

Comments

@cor3ntin
Copy link
Contributor

cor3ntin commented Mar 9, 2018

I'm making this issue to keep track of unicode related issue, but it's a low priority work

  • Remove all wstring except at windows API boundaries.
  • Enforce utf8 internally
  • Proper support for unicode variables names in the compiler and everywhere else we use variables names

That implies

  • normalizing the identifiers to a composed form
  • use XID_Start / XID_Continue properties to determine whether a given identifier is valid
  • Probably best to have a strong "identifier" type, rather than passing strings around

The issue is that most libs to do that are based on icu - not too unreasonable since Qt depends already on it.
However, we may want to be something more suitable if the compiler is to be compiled to wasm or embedded directly on a device

@cor3ntin
Copy link
Contributor Author

cor3ntin commented Mar 9, 2018

#746
#609

@marvelous
Copy link
Collaborator

I can't recommend this article about unicode identifiers enough: http://perl11.org/blog/unicode-identifiers.html
One of the points explained in the article that you might want to address is the issue of mixed scripts where for example you might want to disallow both greek and cyrillic in the same program.

@cor3ntin
Copy link
Contributor Author

cor3ntin commented Mar 9, 2018

@marvelous this article makes very good points. It's something to consider once we get the basis working - Which make take some time given how wstring is currently (miss) used everywhere !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants