Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiler regression on 1.14 when using unicode as variables names #11750

Closed
jared-mackey opened this issue Apr 8, 2022 · 5 comments
Closed

Comments

@jared-mackey
Copy link
Contributor

jared-mackey commented Apr 8, 2022

Environment

  • Elixir & Erlang/OTP versions (elixir --version):
Erlang/OTP 24 [erts-12.3] [source] [64-bit] [smp:32:32] [ds:32:32:10] [async-threads:1] [jit]

Elixir 1.14.0-dev (700e0b7) (compiled with Erlang/OTP 24)
  • Operating system:
Linux 5.17.1-arch1-1 x86_64

Current behavior

Errors when compiling the current tesla release from hex (1.4.4).

== Compilation error in file lib/tesla/middleware/telemetry.ex ==
** (SyntaxError) lib/tesla/middleware/telemetry.ex:150:18: unexpected token: "µ" (column 18, code point U+00B5)
     |
 150 |         duration_µs = System.convert_time_unit(duration, :native, :microsecond)
     |                  ^
    (elixir 1.14.0-dev) lib/kernel/parallel_compiler.ex:346: anonymous fn/5 in Kernel.ParallelCompiler.spawn_workers/7
could not compile dependency :tesla, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile tesla", update it with "mix deps.update tesla" or clean it with "mix deps.clean tesla"

This code has since been removed from tesla but they haven't made a new release yet. It was addressed in this PR and mentioned in this issue.

Expected behavior

This code works on 1.13 and should continue working on 1.14.

@josevalim
Copy link
Member

josevalim commented Apr 9, 2022

Hi @mrluc! This is a regression based on Unicode Security. The µ character is no longer allowed, because it has IdentifierType of "NOT_NKFC". The micro sign actually normalizes to Greek's mu. We only required NFC before, not NKFC.

Another interesting aspect is that, µ is considered to be "Common", probably because its scientific usage applies to many scripts. Therefore, if we simply allowed µ as is, it could be mixed with the Greek version, and cause confusable issues.

However, it is still weird that it is forbidden because of its Greek counterpart. It would be similar if we forbade Latin's w because of Greeks ω. So I would like to discuss a mechanism to include it without marking it as confusable. One idea is to give it all scriptsets EXCEPT greek.

A similar issue happens with all mathematical characters. They all translate either to latin or greek, here is the full list:

# Format: {math_codepoint, scriptset, printed, normalized_codepoint, scriptset, printed}
# 1 is latin scriptset
# 4398046511104 is greek scriptset
{"1D400", nil, "𝐀", "41", nil, "A"}
{"1D401", nil, "𝐁", "42", nil, "B"}
{"1D402", nil, "𝐂", "43", nil, "C"}
{"1D403", nil, "𝐃", "44", nil, "D"}
{"1D404", nil, "𝐄", "45", nil, "E"}
{"1D405", nil, "𝐅", "46", nil, "F"}
{"1D406", nil, "𝐆", "47", nil, "G"}
{"1D407", nil, "𝐇", "48", nil, "H"}
{"1D408", nil, "𝐈", "49", nil, "I"}
{"1D409", nil, "𝐉", "4A", nil, "J"}
{"1D40A", nil, "𝐊", "4B", nil, "K"}
{"1D40B", nil, "𝐋", "4C", nil, "L"}
{"1D40C", nil, "𝐌", "4D", nil, "M"}
{"1D40D", nil, "𝐍", "4E", nil, "N"}
{"1D40E", nil, "𝐎", "4F", nil, "O"}
{"1D40F", nil, "𝐏", "50", nil, "P"}
{"1D410", nil, "𝐐", "51", nil, "Q"}
{"1D411", nil, "𝐑", "52", nil, "R"}
{"1D412", nil, "𝐒", "53", nil, "S"}
{"1D413", nil, "𝐓", "54", nil, "T"}
{"1D414", nil, "𝐔", "55", nil, "U"}
{"1D415", nil, "𝐕", "56", nil, "V"}
{"1D416", nil, "𝐖", "57", nil, "W"}
{"1D417", nil, "𝐗", "58", nil, "X"}
{"1D418", nil, "𝐘", "59", nil, "Y"}
{"1D419", nil, "𝐙", "5A", nil, "Z"}
{"1D41A", nil, "𝐚", "61", nil, "a"}
{"1D41B", nil, "𝐛", "62", nil, "b"}
{"1D41C", nil, "𝐜", "63", nil, "c"}
{"1D41D", nil, "𝐝", "64", nil, "d"}
{"1D41E", nil, "𝐞", "65", nil, "e"}
{"1D41F", nil, "𝐟", "66", nil, "f"}
{"1D420", nil, "𝐠", "67", nil, "g"}
{"1D421", nil, "𝐡", "68", nil, "h"}
{"1D422", nil, "𝐢", "69", nil, "i"}
{"1D423", nil, "𝐣", "6A", nil, "j"}
{"1D424", nil, "𝐤", "6B", nil, "k"}
{"1D425", nil, "𝐥", "6C", nil, "l"}
{"1D426", nil, "𝐦", "6D", nil, "m"}
{"1D427", nil, "𝐧", "6E", nil, "n"}
{"1D428", nil, "𝐨", "6F", nil, "o"}
{"1D429", nil, "𝐩", "70", nil, "p"}
{"1D42A", nil, "𝐪", "71", nil, "q"}
{"1D42B", nil, "𝐫", "72", nil, "r"}
{"1D42C", nil, "𝐬", "73", nil, "s"}
{"1D42D", nil, "𝐭", "74", nil, "t"}
{"1D42E", nil, "𝐮", "75", nil, "u"}
{"1D42F", nil, "𝐯", "76", nil, "v"}
{"1D430", nil, "𝐰", "77", nil, "w"}
{"1D431", nil, "𝐱", "78", nil, "x"}
{"1D432", nil, "𝐲", "79", nil, "y"}
{"1D433", nil, "𝐳", "7A", nil, "z"}
{"1D434", nil, "𝐴", "41", nil, "A"}
{"1D435", nil, "𝐵", "42", nil, "B"}
{"1D436", nil, "𝐶", "43", nil, "C"}
{"1D437", nil, "𝐷", "44", nil, "D"}
{"1D438", nil, "𝐸", "45", nil, "E"}
{"1D439", nil, "𝐹", "46", nil, "F"}
{"1D43A", nil, "𝐺", "47", nil, "G"}
{"1D43B", nil, "𝐻", "48", nil, "H"}
{"1D43C", nil, "𝐼", "49", nil, "I"}
{"1D43D", nil, "𝐽", "4A", nil, "J"}
{"1D43E", nil, "𝐾", "4B", nil, "K"}
{"1D43F", nil, "𝐿", "4C", nil, "L"}
{"1D440", nil, "𝑀", "4D", nil, "M"}
{"1D441", nil, "𝑁", "4E", nil, "N"}
{"1D442", nil, "𝑂", "4F", nil, "O"}
{"1D443", nil, "𝑃", "50", nil, "P"}
{"1D444", nil, "𝑄", "51", nil, "Q"}
{"1D445", nil, "𝑅", "52", nil, "R"}
{"1D446", nil, "𝑆", "53", nil, "S"}
{"1D447", nil, "𝑇", "54", nil, "T"}
{"1D448", nil, "𝑈", "55", nil, "U"}
{"1D449", nil, "𝑉", "56", nil, "V"}
{"1D44A", nil, "𝑊", "57", nil, "W"}
{"1D44B", nil, "𝑋", "58", nil, "X"}
{"1D44C", nil, "𝑌", "59", nil, "Y"}
{"1D44D", nil, "𝑍", "5A", nil, "Z"}
{"1D44E", nil, "𝑎", "61", nil, "a"}
{"1D44F", nil, "𝑏", "62", nil, "b"}
{"1D450", nil, "𝑐", "63", nil, "c"}
{"1D451", nil, "𝑑", "64", nil, "d"}
{"1D452", nil, "𝑒", "65", nil, "e"}
{"1D453", nil, "𝑓", "66", nil, "f"}
{"1D454", nil, "𝑔", "67", nil, "g"}
{"1D456", nil, "𝑖", "69", nil, "i"}
{"1D457", nil, "𝑗", "6A", nil, "j"}
{"1D458", nil, "𝑘", "6B", nil, "k"}
{"1D459", nil, "𝑙", "6C", nil, "l"}
{"1D45A", nil, "𝑚", "6D", nil, "m"}
{"1D45B", nil, "𝑛", "6E", nil, "n"}
{"1D45C", nil, "𝑜", "6F", nil, "o"}
{"1D45D", nil, "𝑝", "70", nil, "p"}
{"1D45E", nil, "𝑞", "71", nil, "q"}
{"1D45F", nil, "𝑟", "72", nil, "r"}
{"1D460", nil, "𝑠", "73", nil, "s"}
{"1D461", nil, "𝑡", "74", nil, "t"}
{"1D462", nil, "𝑢", "75", nil, "u"}
{"1D463", nil, "𝑣", "76", nil, "v"}
{"1D464", nil, "𝑤", "77", nil, "w"}
{"1D465", nil, "𝑥", "78", nil, "x"}
{"1D466", nil, "𝑦", "79", nil, "y"}
{"1D467", nil, "𝑧", "7A", nil, "z"}
{"1D468", nil, "𝑨", "41", nil, "A"}
{"1D469", nil, "𝑩", "42", nil, "B"}
{"1D46A", nil, "𝑪", "43", nil, "C"}
{"1D46B", nil, "𝑫", "44", nil, "D"}
{"1D46C", nil, "𝑬", "45", nil, "E"}
{"1D46D", nil, "𝑭", "46", nil, "F"}
{"1D46E", nil, "𝑮", "47", nil, "G"}
{"1D46F", nil, "𝑯", "48", nil, "H"}
{"1D470", nil, "𝑰", "49", nil, "I"}
{"1D471", nil, "𝑱", "4A", nil, "J"}
{"1D472", nil, "𝑲", "4B", nil, "K"}
{"1D473", nil, "𝑳", "4C", nil, "L"}
{"1D474", nil, "𝑴", "4D", nil, "M"}
{"1D475", nil, "𝑵", "4E", nil, "N"}
{"1D476", nil, "𝑶", "4F", nil, "O"}
{"1D477", nil, "𝑷", "50", nil, "P"}
{"1D478", nil, "𝑸", "51", nil, "Q"}
{"1D479", nil, "𝑹", "52", nil, "R"}
{"1D47A", nil, "𝑺", "53", nil, "S"}
{"1D47B", nil, "𝑻", "54", nil, "T"}
{"1D47C", nil, "𝑼", "55", nil, "U"}
{"1D47D", nil, "𝑽", "56", nil, "V"}
{"1D47E", nil, "𝑾", "57", nil, "W"}
{"1D47F", nil, "𝑿", "58", nil, "X"}
{"1D480", nil, "𝒀", "59", nil, "Y"}
{"1D481", nil, "𝒁", "5A", nil, "Z"}
{"1D482", nil, "𝒂", "61", nil, "a"}
{"1D483", nil, "𝒃", "62", nil, "b"}
{"1D484", nil, "𝒄", "63", nil, "c"}
{"1D485", nil, "𝒅", "64", nil, "d"}
{"1D486", nil, "𝒆", "65", nil, "e"}
{"1D487", nil, "𝒇", "66", nil, "f"}
{"1D488", nil, "𝒈", "67", nil, "g"}
{"1D489", nil, "𝒉", "68", nil, "h"}
{"1D48A", nil, "𝒊", "69", nil, "i"}
{"1D48B", nil, "𝒋", "6A", nil, "j"}
{"1D48C", nil, "𝒌", "6B", nil, "k"}
{"1D48D", nil, "𝒍", "6C", nil, "l"}
{"1D48E", nil, "𝒎", "6D", nil, "m"}
{"1D48F", nil, "𝒏", "6E", nil, "n"}
{"1D490", nil, "𝒐", "6F", nil, "o"}
{"1D491", nil, "𝒑", "70", nil, "p"}
{"1D492", nil, "𝒒", "71", nil, "q"}
{"1D493", nil, "𝒓", "72", nil, "r"}
{"1D494", nil, "𝒔", "73", nil, "s"}
{"1D495", nil, "𝒕", "74", nil, "t"}
{"1D496", nil, "𝒖", "75", nil, "u"}
{"1D497", nil, "𝒗", "76", nil, "v"}
{"1D498", nil, "𝒘", "77", nil, "w"}
{"1D499", nil, "𝒙", "78", nil, "x"}
{"1D49A", nil, "𝒚", "79", nil, "y"}
{"1D49B", nil, "𝒛", "7A", nil, "z"}
{"1D49C", nil, "𝒜", "41", nil, "A"}
{"1D49E", nil, "𝒞", "43", nil, "C"}
{"1D49F", nil, "𝒟", "44", nil, "D"}
{"1D4A2", nil, "𝒢", "47", nil, "G"}
{"1D4A5", nil, "𝒥", "4A", nil, "J"}
{"1D4A6", nil, "𝒦", "4B", nil, "K"}
{"1D4A9", nil, "𝒩", "4E", nil, "N"}
{"1D4AA", nil, "𝒪", "4F", nil, "O"}
{"1D4AB", nil, "𝒫", "50", nil, "P"}
{"1D4AC", nil, "𝒬", "51", nil, "Q"}
{"1D4AE", nil, "𝒮", "53", nil, "S"}
{"1D4AF", nil, "𝒯", "54", nil, "T"}
{"1D4B0", nil, "𝒰", "55", nil, "U"}
{"1D4B1", nil, "𝒱", "56", nil, "V"}
{"1D4B2", nil, "𝒲", "57", nil, "W"}
{"1D4B3", nil, "𝒳", "58", nil, "X"}
{"1D4B4", nil, "𝒴", "59", nil, "Y"}
{"1D4B5", nil, "𝒵", "5A", nil, "Z"}
{"1D4B6", nil, "𝒶", "61", nil, "a"}
{"1D4B7", nil, "𝒷", "62", nil, "b"}
{"1D4B8", nil, "𝒸", "63", nil, "c"}
{"1D4B9", nil, "𝒹", "64", nil, "d"}
{"1D4BB", nil, "𝒻", "66", nil, "f"}
{"1D4BD", nil, "𝒽", "68", nil, "h"}
{"1D4BE", nil, "𝒾", "69", nil, "i"}
{"1D4BF", nil, "𝒿", "6A", nil, "j"}
{"1D4C0", nil, "𝓀", "6B", nil, "k"}
{"1D4C1", nil, "𝓁", "6C", nil, "l"}
{"1D4C2", nil, "𝓂", "6D", nil, "m"}
{"1D4C3", nil, "𝓃", "6E", nil, "n"}
{"1D4C5", nil, "𝓅", "70", nil, "p"}
{"1D4C6", nil, "𝓆", "71", nil, "q"}
{"1D4C7", nil, "𝓇", "72", nil, "r"}
{"1D4C8", nil, "𝓈", "73", nil, "s"}
{"1D4C9", nil, "𝓉", "74", nil, "t"}
{"1D4CA", nil, "𝓊", "75", nil, "u"}
{"1D4CB", nil, "𝓋", "76", nil, "v"}
{"1D4CC", nil, "𝓌", "77", nil, "w"}
{"1D4CD", nil, "𝓍", "78", nil, "x"}
{"1D4CE", nil, "𝓎", "79", nil, "y"}
{"1D4CF", nil, "𝓏", "7A", nil, "z"}
{"1D4D0", nil, "𝓐", "41", nil, "A"}
{"1D4D1", nil, "𝓑", "42", nil, "B"}
{"1D4D2", nil, "𝓒", "43", nil, "C"}
{"1D4D3", nil, "𝓓", "44", nil, "D"}
{"1D4D4", nil, "𝓔", "45", nil, "E"}
{"1D4D5", nil, "𝓕", "46", nil, "F"}
{"1D4D6", nil, "𝓖", "47", nil, "G"}
{"1D4D7", nil, "𝓗", "48", nil, "H"}
{"1D4D8", nil, "𝓘", "49", nil, "I"}
{"1D4D9", nil, "𝓙", "4A", nil, "J"}
{"1D4DA", nil, "𝓚", "4B", nil, "K"}
{"1D4DB", nil, "𝓛", "4C", nil, "L"}
{"1D4DC", nil, "𝓜", "4D", nil, "M"}
{"1D4DD", nil, "𝓝", "4E", nil, "N"}
{"1D4DE", nil, "𝓞", "4F", nil, "O"}
{"1D4DF", nil, "𝓟", "50", nil, "P"}
{"1D4E0", nil, "𝓠", "51", nil, "Q"}
{"1D4E1", nil, "𝓡", "52", nil, "R"}
{"1D4E2", nil, "𝓢", "53", nil, "S"}
{"1D4E3", nil, "𝓣", "54", nil, "T"}
{"1D4E4", nil, "𝓤", "55", nil, "U"}
{"1D4E5", nil, "𝓥", "56", nil, "V"}
{"1D4E6", nil, "𝓦", "57", nil, "W"}
{"1D4E7", nil, "𝓧", "58", nil, "X"}
{"1D4E8", nil, "𝓨", "59", nil, "Y"}
{"1D4E9", nil, "𝓩", "5A", nil, "Z"}
{"1D4EA", nil, "𝓪", "61", nil, "a"}
{"1D4EB", nil, "𝓫", "62", nil, "b"}
{"1D4EC", nil, "𝓬", "63", nil, "c"}
{"1D4ED", nil, "𝓭", "64", nil, "d"}
{"1D4EE", nil, "𝓮", "65", nil, "e"}
{"1D4EF", nil, "𝓯", "66", nil, "f"}
{"1D4F0", nil, "𝓰", "67", nil, "g"}
{"1D4F1", nil, "𝓱", "68", nil, "h"}
{"1D4F2", nil, "𝓲", "69", nil, "i"}
{"1D4F3", nil, "𝓳", "6A", nil, "j"}
{"1D4F4", nil, "𝓴", "6B", nil, "k"}
{"1D4F5", nil, "𝓵", "6C", nil, "l"}
{"1D4F6", nil, "𝓶", "6D", nil, "m"}
{"1D4F7", nil, "𝓷", "6E", nil, "n"}
{"1D4F8", nil, "𝓸", "6F", nil, "o"}
{"1D4F9", nil, "𝓹", "70", nil, "p"}
{"1D4FA", nil, "𝓺", "71", nil, "q"}
{"1D4FB", nil, "𝓻", "72", nil, "r"}
{"1D4FC", nil, "𝓼", "73", nil, "s"}
{"1D4FD", nil, "𝓽", "74", nil, "t"}
{"1D4FE", nil, "𝓾", "75", nil, "u"}
{"1D4FF", nil, "𝓿", "76", nil, "v"}
{"1D500", nil, "𝔀", "77", nil, "w"}
{"1D501", nil, "𝔁", "78", nil, "x"}
{"1D502", nil, "𝔂", "79", nil, "y"}
{"1D503", nil, "𝔃", "7A", nil, "z"}
{"1D504", nil, "𝔄", "41", nil, "A"}
{"1D505", nil, "𝔅", "42", nil, "B"}
{"1D507", nil, "𝔇", "44", nil, "D"}
{"1D508", nil, "𝔈", "45", nil, "E"}
{"1D509", nil, "𝔉", "46", nil, "F"}
{"1D50A", nil, "𝔊", "47", nil, "G"}
{"1D50D", nil, "𝔍", "4A", nil, "J"}
{"1D50E", nil, "𝔎", "4B", nil, "K"}
{"1D50F", nil, "𝔏", "4C", nil, "L"}
{"1D510", nil, "𝔐", "4D", nil, "M"}
{"1D511", nil, "𝔑", "4E", nil, "N"}
{"1D512", nil, "𝔒", "4F", nil, "O"}
{"1D513", nil, "𝔓", "50", nil, "P"}
{"1D514", nil, "𝔔", "51", nil, "Q"}
{"1D516", nil, "𝔖", "53", nil, "S"}
{"1D517", nil, "𝔗", "54", nil, "T"}
{"1D518", nil, "𝔘", "55", nil, "U"}
{"1D519", nil, "𝔙", "56", nil, "V"}
{"1D51A", nil, "𝔚", "57", nil, "W"}
{"1D51B", nil, "𝔛", "58", nil, "X"}
{"1D51C", nil, "𝔜", "59", nil, "Y"}
{"1D51E", nil, "𝔞", "61", nil, "a"}
{"1D51F", nil, "𝔟", "62", nil, "b"}
{"1D520", nil, "𝔠", "63", nil, "c"}
{"1D521", nil, "𝔡", "64", nil, "d"}
{"1D522", nil, "𝔢", "65", nil, "e"}
{"1D523", nil, "𝔣", "66", nil, "f"}
{"1D524", nil, "𝔤", "67", nil, "g"}
{"1D525", nil, "𝔥", "68", nil, "h"}
{"1D526", nil, "𝔦", "69", nil, "i"}
{"1D527", nil, "𝔧", "6A", nil, "j"}
{"1D528", nil, "𝔨", "6B", nil, "k"}
{"1D529", nil, "𝔩", "6C", nil, "l"}
{"1D52A", nil, "𝔪", "6D", nil, "m"}
{"1D52B", nil, "𝔫", "6E", nil, "n"}
{"1D52C", nil, "𝔬", "6F", nil, "o"}
{"1D52D", nil, "𝔭", "70", nil, "p"}
{"1D52E", nil, "𝔮", "71", nil, "q"}
{"1D52F", nil, "𝔯", "72", nil, "r"}
{"1D530", nil, "𝔰", "73", nil, "s"}
{"1D531", nil, "𝔱", "74", nil, "t"}
{"1D532", nil, "𝔲", "75", nil, "u"}
{"1D533", nil, "𝔳", "76", nil, "v"}
{"1D534", nil, "𝔴", "77", nil, "w"}
{"1D535", nil, "𝔵", "78", nil, "x"}
{"1D536", nil, "𝔶", "79", nil, "y"}
{"1D537", nil, "𝔷", "7A", nil, "z"}
{"1D538", nil, "𝔸", "41", nil, "A"}
{"1D539", nil, "𝔹", "42", nil, "B"}
{"1D53B", nil, "𝔻", "44", nil, "D"}
{"1D53C", nil, "𝔼", "45", nil, "E"}
{"1D53D", nil, "𝔽", "46", nil, "F"}
{"1D53E", nil, "𝔾", "47", nil, "G"}
{"1D540", nil, "𝕀", "49", nil, "I"}
{"1D541", nil, "𝕁", "4A", nil, "J"}
{"1D542", nil, "𝕂", "4B", nil, "K"}
{"1D543", nil, "𝕃", "4C", nil, "L"}
{"1D544", nil, "𝕄", "4D", nil, "M"}
{"1D546", nil, "𝕆", "4F", nil, "O"}
{"1D54A", nil, "𝕊", "53", nil, "S"}
{"1D54B", nil, "𝕋", "54", nil, "T"}
{"1D54C", nil, "𝕌", "55", nil, "U"}
{"1D54D", nil, "𝕍", "56", nil, "V"}
{"1D54E", nil, "𝕎", "57", nil, "W"}
{"1D54F", nil, "𝕏", "58", nil, "X"}
{"1D550", nil, "𝕐", "59", nil, "Y"}
{"1D552", nil, "𝕒", "61", nil, "a"}
{"1D553", nil, "𝕓", "62", nil, "b"}
{"1D554", nil, "𝕔", "63", nil, "c"}
{"1D555", nil, "𝕕", "64", nil, "d"}
{"1D556", nil, "𝕖", "65", nil, "e"}
{"1D557", nil, "𝕗", "66", nil, "f"}
{"1D558", nil, "𝕘", "67", nil, "g"}
{"1D559", nil, "𝕙", "68", nil, "h"}
{"1D55A", nil, "𝕚", "69", nil, "i"}
{"1D55B", nil, "𝕛", "6A", nil, "j"}
{"1D55C", nil, "𝕜", "6B", nil, "k"}
{"1D55D", nil, "𝕝", "6C", nil, "l"}
{"1D55E", nil, "𝕞", "6D", nil, "m"}
{"1D55F", nil, "𝕟", "6E", nil, "n"}
{"1D560", nil, "𝕠", "6F", nil, "o"}
{"1D561", nil, "𝕡", "70", nil, "p"}
{"1D562", nil, "𝕢", "71", nil, "q"}
{"1D563", nil, "𝕣", "72", nil, "r"}
{"1D564", nil, "𝕤", "73", nil, "s"}
{"1D565", nil, "𝕥", "74", nil, "t"}
{"1D566", nil, "𝕦", "75", nil, "u"}
{"1D567", nil, "𝕧", "76", nil, "v"}
{"1D568", nil, "𝕨", "77", nil, "w"}
{"1D569", nil, "𝕩", "78", nil, "x"}
{"1D56A", nil, "𝕪", "79", nil, "y"}
{"1D56B", nil, "𝕫", "7A", nil, "z"}
{"1D56C", nil, "𝕬", "41", nil, "A"}
{"1D56D", nil, "𝕭", "42", nil, "B"}
{"1D56E", nil, "𝕮", "43", nil, "C"}
{"1D56F", nil, "𝕯", "44", nil, "D"}
{"1D570", nil, "𝕰", "45", nil, "E"}
{"1D571", nil, "𝕱", "46", nil, "F"}
{"1D572", nil, "𝕲", "47", nil, "G"}
{"1D573", nil, "𝕳", "48", nil, "H"}
{"1D574", nil, "𝕴", "49", nil, "I"}
{"1D575", nil, "𝕵", "4A", nil, "J"}
{"1D576", nil, "𝕶", "4B", nil, "K"}
{"1D577", nil, "𝕷", "4C", nil, "L"}
{"1D578", nil, "𝕸", "4D", nil, "M"}
{"1D579", nil, "𝕹", "4E", nil, "N"}
{"1D57A", nil, "𝕺", "4F", nil, "O"}
{"1D57B", nil, "𝕻", "50", nil, "P"}
{"1D57C", nil, "𝕼", "51", nil, "Q"}
{"1D57D", nil, "𝕽", "52", nil, "R"}
{"1D57E", nil, "𝕾", "53", nil, "S"}
{"1D57F", nil, "𝕿", "54", nil, "T"}
{"1D580", nil, "𝖀", "55", nil, "U"}
{"1D581", nil, "𝖁", "56", nil, "V"}
{"1D582", nil, "𝖂", "57", nil, "W"}
{"1D583", nil, "𝖃", "58", nil, "X"}
{"1D584", nil, "𝖄", "59", nil, "Y"}
{"1D585", nil, "𝖅", "5A", nil, "Z"}
{"1D586", nil, "𝖆", "61", nil, "a"}
{"1D587", nil, "𝖇", "62", nil, "b"}
{"1D588", nil, "𝖈", "63", nil, "c"}
{"1D589", nil, "𝖉", "64", nil, "d"}
{"1D58A", nil, "𝖊", "65", nil, "e"}
{"1D58B", nil, "𝖋", "66", nil, "f"}
{"1D58C", nil, "𝖌", "67", nil, "g"}
{"1D58D", nil, "𝖍", "68", nil, "h"}
{"1D58E", nil, "𝖎", "69", nil, "i"}
{"1D58F", nil, "𝖏", "6A", nil, "j"}
{"1D590", nil, "𝖐", "6B", nil, "k"}
{"1D591", nil, "𝖑", "6C", nil, "l"}
{"1D592", nil, "𝖒", "6D", nil, "m"}
{"1D593", nil, "𝖓", "6E", nil, "n"}
{"1D594", nil, "𝖔", "6F", nil, "o"}
{"1D595", nil, "𝖕", "70", nil, "p"}
{"1D596", nil, "𝖖", "71", nil, "q"}
{"1D597", nil, "𝖗", "72", nil, "r"}
{"1D598", nil, "𝖘", "73", nil, "s"}
{"1D599", nil, "𝖙", "74", nil, "t"}
{"1D59A", nil, "𝖚", "75", nil, "u"}
{"1D59B", nil, "𝖛", "76", nil, "v"}
{"1D59C", nil, "𝖜", "77", nil, "w"}
{"1D59D", nil, "𝖝", "78", nil, "x"}
{"1D59E", nil, "𝖞", "79", nil, "y"}
{"1D59F", nil, "𝖟", "7A", nil, "z"}
{"1D5A0", nil, "𝖠", "41", nil, "A"}
{"1D5A1", nil, "𝖡", "42", nil, "B"}
{"1D5A2", nil, "𝖢", "43", nil, "C"}
{"1D5A3", nil, "𝖣", "44", nil, "D"}
{"1D5A4", nil, "𝖤", "45", nil, "E"}
{"1D5A5", nil, "𝖥", "46", nil, "F"}
{"1D5A6", nil, "𝖦", "47", nil, "G"}
{"1D5A7", nil, "𝖧", "48", nil, "H"}
{"1D5A8", nil, "𝖨", "49", nil, "I"}
{"1D5A9", nil, "𝖩", "4A", nil, "J"}
{"1D5AA", nil, "𝖪", "4B", nil, "K"}
{"1D5AB", nil, "𝖫", "4C", nil, "L"}
{"1D5AC", nil, "𝖬", "4D", nil, "M"}
{"1D5AD", nil, "𝖭", "4E", nil, "N"}
{"1D5AE", nil, "𝖮", "4F", nil, "O"}
{"1D5AF", nil, "𝖯", "50", nil, "P"}
{"1D5B0", nil, "𝖰", "51", nil, "Q"}
{"1D5B1", nil, "𝖱", "52", nil, "R"}
{"1D5B2", nil, "𝖲", "53", nil, "S"}
{"1D5B3", nil, "𝖳", "54", nil, "T"}
{"1D5B4", nil, "𝖴", "55", nil, "U"}
{"1D5B5", nil, "𝖵", "56", nil, "V"}
{"1D5B6", nil, "𝖶", "57", nil, "W"}
{"1D5B7", nil, "𝖷", "58", nil, "X"}
{"1D5B8", nil, "𝖸", "59", nil, "Y"}
{"1D5B9", nil, "𝖹", "5A", nil, "Z"}
{"1D5BA", nil, "𝖺", "61", nil, "a"}
{"1D5BB", nil, "𝖻", "62", nil, "b"}
{"1D5BC", nil, "𝖼", "63", nil, "c"}
{"1D5BD", nil, "𝖽", "64", nil, "d"}
{"1D5BE", nil, "𝖾", "65", nil, "e"}
{"1D5BF", nil, "𝖿", "66", nil, "f"}
{"1D5C0", nil, "𝗀", "67", nil, "g"}
{"1D5C1", nil, "𝗁", "68", nil, "h"}
{"1D5C2", nil, "𝗂", "69", nil, "i"}
{"1D5C3", nil, "𝗃", "6A", nil, "j"}
{"1D5C4", nil, "𝗄", "6B", nil, "k"}
{"1D5C5", nil, "𝗅", "6C", nil, "l"}
{"1D5C6", nil, "𝗆", "6D", nil, "m"}
{"1D5C7", nil, "𝗇", "6E", nil, "n"}
{"1D5C8", nil, "𝗈", "6F", nil, "o"}
{"1D5C9", nil, "𝗉", "70", nil, "p"}
{"1D5CA", nil, "𝗊", "71", nil, "q"}
{"1D5CB", nil, "𝗋", "72", nil, "r"}
{"1D5CC", nil, "𝗌", "73", nil, "s"}
{"1D5CD", nil, "𝗍", "74", nil, "t"}
{"1D5CE", nil, "𝗎", "75", nil, "u"}
{"1D5CF", nil, "𝗏", "76", nil, "v"}
{"1D5D0", nil, "𝗐", "77", nil, "w"}
{"1D5D1", nil, "𝗑", "78", nil, "x"}
{"1D5D2", nil, "𝗒", "79", nil, "y"}
{"1D5D3", nil, "𝗓", "7A", nil, "z"}
{"1D5D4", nil, "𝗔", "41", nil, "A"}
{"1D5D5", nil, "𝗕", "42", nil, "B"}
{"1D5D6", nil, "𝗖", "43", nil, "C"}
{"1D5D7", nil, "𝗗", "44", nil, "D"}
{"1D5D8", nil, "𝗘", "45", nil, "E"}
{"1D5D9", nil, "𝗙", "46", nil, "F"}
{"1D5DA", nil, "𝗚", "47", nil, "G"}
{"1D5DB", nil, "𝗛", "48", nil, "H"}
{"1D5DC", nil, "𝗜", "49", nil, "I"}
{"1D5DD", nil, "𝗝", "4A", nil, "J"}
{"1D5DE", nil, "𝗞", "4B", nil, "K"}
{"1D5DF", nil, "𝗟", "4C", nil, "L"}
{"1D5E0", nil, "𝗠", "4D", nil, "M"}
{"1D5E1", nil, "𝗡", "4E", nil, "N"}
{"1D5E2", nil, "𝗢", "4F", nil, "O"}
{"1D5E3", nil, "𝗣", "50", nil, "P"}
{"1D5E4", nil, "𝗤", "51", nil, "Q"}
{"1D5E5", nil, "𝗥", "52", nil, "R"}
{"1D5E6", nil, "𝗦", "53", nil, "S"}
{"1D5E7", nil, "𝗧", "54", nil, "T"}
{"1D5E8", nil, "𝗨", "55", nil, "U"}
{"1D5E9", nil, "𝗩", "56", nil, "V"}
{"1D5EA", nil, "𝗪", "57", nil, "W"}
{"1D5EB", nil, "𝗫", "58", nil, "X"}
{"1D5EC", nil, "𝗬", "59", nil, "Y"}
{"1D5ED", nil, "𝗭", "5A", nil, "Z"}
{"1D5EE", nil, "𝗮", "61", nil, "a"}
{"1D5EF", nil, "𝗯", "62", nil, "b"}
{"1D5F0", nil, "𝗰", "63", nil, "c"}
{"1D5F1", nil, "𝗱", "64", nil, "d"}
{"1D5F2", nil, "𝗲", "65", nil, "e"}
{"1D5F3", nil, "𝗳", "66", nil, "f"}
{"1D5F4", nil, "𝗴", "67", nil, "g"}
{"1D5F5", nil, "𝗵", "68", nil, "h"}
{"1D5F6", nil, "𝗶", "69", nil, "i"}
{"1D5F7", nil, "𝗷", "6A", nil, "j"}
{"1D5F8", nil, "𝗸", "6B", nil, "k"}
{"1D5F9", nil, "𝗹", "6C", nil, "l"}
{"1D5FA", nil, "𝗺", "6D", nil, "m"}
{"1D5FB", nil, "𝗻", "6E", nil, "n"}
{"1D5FC", nil, "𝗼", "6F", nil, "o"}
{"1D5FD", nil, "𝗽", "70", nil, "p"}
{"1D5FE", nil, "𝗾", "71", nil, "q"}
{"1D5FF", nil, "𝗿", "72", nil, "r"}
{"1D600", nil, "𝘀", "73", nil, "s"}
{"1D601", nil, "𝘁", "74", nil, "t"}
{"1D602", nil, "𝘂", "75", nil, "u"}
{"1D603", nil, "𝘃", "76", nil, "v"}
{"1D604", nil, "𝘄", "77", nil, "w"}
{"1D605", nil, "𝘅", "78", nil, "x"}
{"1D606", nil, "𝘆", "79", nil, "y"}
{"1D607", nil, "𝘇", "7A", nil, "z"}
{"1D608", nil, "𝘈", "41", nil, "A"}
{"1D609", nil, "𝘉", "42", nil, "B"}
{"1D60A", nil, "𝘊", "43", nil, "C"}
{"1D60B", nil, "𝘋", "44", nil, "D"}
{"1D60C", nil, "𝘌", "45", nil, "E"}
{"1D60D", nil, "𝘍", "46", nil, "F"}
{"1D60E", nil, "𝘎", "47", nil, "G"}
{"1D60F", nil, "𝘏", "48", nil, "H"}
{"1D610", nil, "𝘐", "49", nil, "I"}
{"1D611", nil, "𝘑", "4A", nil, "J"}
{"1D612", nil, "𝘒", "4B", nil, "K"}
{"1D613", nil, "𝘓", "4C", nil, "L"}
{"1D614", nil, "𝘔", "4D", nil, "M"}
{"1D615", nil, "𝘕", "4E", nil, "N"}
{"1D616", nil, "𝘖", "4F", nil, "O"}
{"1D617", nil, "𝘗", "50", nil, "P"}
{"1D618", nil, "𝘘", "51", nil, "Q"}
{"1D619", nil, "𝘙", "52", nil, "R"}
{"1D61A", nil, "𝘚", "53", nil, "S"}
{"1D61B", nil, "𝘛", "54", nil, "T"}
{"1D61C", nil, "𝘜", "55", nil, "U"}
{"1D61D", nil, "𝘝", "56", nil, "V"}
{"1D61E", nil, "𝘞", "57", nil, "W"}
{"1D61F", nil, "𝘟", "58", nil, "X"}
{"1D620", nil, "𝘠", "59", nil, "Y"}
{"1D621", nil, "𝘡", "5A", nil, "Z"}
{"1D622", nil, "𝘢", "61", nil, "a"}
{"1D623", nil, "𝘣", "62", nil, "b"}
{"1D624", nil, "𝘤", "63", nil, "c"}
{"1D625", nil, "𝘥", "64", nil, "d"}
{"1D626", nil, "𝘦", "65", nil, "e"}
{"1D627", nil, "𝘧", "66", nil, "f"}
{"1D628", nil, "𝘨", "67", nil, "g"}
{"1D629", nil, "𝘩", "68", nil, "h"}
{"1D62A", nil, "𝘪", "69", nil, "i"}
{"1D62B", nil, "𝘫", "6A", nil, "j"}
{"1D62C", nil, "𝘬", "6B", nil, "k"}
{"1D62D", nil, "𝘭", "6C", nil, "l"}
{"1D62E", nil, "𝘮", "6D", nil, "m"}
{"1D62F", nil, "𝘯", "6E", nil, "n"}
{"1D630", nil, "𝘰", "6F", nil, "o"}
{"1D631", nil, "𝘱", "70", nil, "p"}
{"1D632", nil, "𝘲", "71", nil, "q"}
{"1D633", nil, "𝘳", "72", nil, "r"}
{"1D634", nil, "𝘴", "73", nil, "s"}
{"1D635", nil, "𝘵", "74", nil, "t"}
{"1D636", nil, "𝘶", "75", nil, "u"}
{"1D637", nil, "𝘷", "76", nil, "v"}
{"1D638", nil, "𝘸", "77", nil, "w"}
{"1D639", nil, "𝘹", "78", nil, "x"}
{"1D63A", nil, "𝘺", "79", nil, "y"}
{"1D63B", nil, "𝘻", "7A", nil, "z"}
{"1D63C", nil, "𝘼", "41", nil, "A"}
{"1D63D", nil, "𝘽", "42", nil, "B"}
{"1D63E", nil, "𝘾", "43", nil, "C"}
{"1D63F", nil, "𝘿", "44", nil, "D"}
{"1D640", nil, "𝙀", "45", nil, "E"}
{"1D641", nil, "𝙁", "46", nil, "F"}
{"1D642", nil, "𝙂", "47", nil, "G"}
{"1D643", nil, "𝙃", "48", nil, "H"}
{"1D644", nil, "𝙄", "49", nil, "I"}
{"1D645", nil, "𝙅", "4A", nil, "J"}
{"1D646", nil, "𝙆", "4B", nil, "K"}
{"1D647", nil, "𝙇", "4C", nil, "L"}
{"1D648", nil, "𝙈", "4D", nil, "M"}
{"1D649", nil, "𝙉", "4E", nil, "N"}
{"1D64A", nil, "𝙊", "4F", nil, "O"}
{"1D64B", nil, "𝙋", "50", nil, "P"}
{"1D64C", nil, "𝙌", "51", nil, "Q"}
{"1D64D", nil, "𝙍", "52", nil, "R"}
{"1D64E", nil, "𝙎", "53", nil, "S"}
{"1D64F", nil, "𝙏", "54", nil, "T"}
{"1D650", nil, "𝙐", "55", nil, "U"}
{"1D651", nil, "𝙑", "56", nil, "V"}
{"1D652", nil, "𝙒", "57", nil, "W"}
{"1D653", nil, "𝙓", "58", nil, "X"}
{"1D654", nil, "𝙔", "59", nil, "Y"}
{"1D655", nil, "𝙕", "5A", nil, "Z"}
{"1D656", nil, "𝙖", "61", nil, "a"}
{"1D657", nil, "𝙗", "62", nil, "b"}
{"1D658", nil, "𝙘", "63", nil, "c"}
{"1D659", nil, "𝙙", "64", nil, "d"}
{"1D65A", nil, "𝙚", "65", nil, "e"}
{"1D65B", nil, "𝙛", "66", nil, "f"}
{"1D65C", nil, "𝙜", "67", nil, "g"}
{"1D65D", nil, "𝙝", "68", nil, "h"}
{"1D65E", nil, "𝙞", "69", nil, "i"}
{"1D65F", nil, "𝙟", "6A", nil, "j"}
{"1D660", nil, "𝙠", "6B", nil, "k"}
{"1D661", nil, "𝙡", "6C", nil, "l"}
{"1D662", nil, "𝙢", "6D", nil, "m"}
{"1D663", nil, "𝙣", "6E", nil, "n"}
{"1D664", nil, "𝙤", "6F", nil, "o"}
{"1D665", nil, "𝙥", "70", nil, "p"}
{"1D666", nil, "𝙦", "71", nil, "q"}
{"1D667", nil, "𝙧", "72", nil, "r"}
{"1D668", nil, "𝙨", "73", nil, "s"}
{"1D669", nil, "𝙩", "74", nil, "t"}
{"1D66A", nil, "𝙪", "75", nil, "u"}
{"1D66B", nil, "𝙫", "76", nil, "v"}
{"1D66C", nil, "𝙬", "77", nil, "w"}
{"1D66D", nil, "𝙭", "78", nil, "x"}
{"1D66E", nil, "𝙮", "79", nil, "y"}
{"1D66F", nil, "𝙯", "7A", nil, "z"}
{"1D670", nil, "𝙰", "41", nil, "A"}
{"1D671", nil, "𝙱", "42", nil, "B"}
{"1D672", nil, "𝙲", "43", nil, "C"}
{"1D673", nil, "𝙳", "44", nil, "D"}
{"1D674", nil, "𝙴", "45", nil, "E"}
{"1D675", nil, "𝙵", "46", nil, "F"}
{"1D676", nil, "𝙶", "47", nil, "G"}
{"1D677", nil, "𝙷", "48", nil, "H"}
{"1D678", nil, "𝙸", "49", nil, "I"}
{"1D679", nil, "𝙹", "4A", nil, "J"}
{"1D67A", nil, "𝙺", "4B", nil, "K"}
{"1D67B", nil, "𝙻", "4C", nil, "L"}
{"1D67C", nil, "𝙼", "4D", nil, "M"}
{"1D67D", nil, "𝙽", "4E", nil, "N"}
{"1D67E", nil, "𝙾", "4F", nil, "O"}
{"1D67F", nil, "𝙿", "50", nil, "P"}
{"1D680", nil, "𝚀", "51", nil, "Q"}
{"1D681", nil, "𝚁", "52", nil, "R"}
{"1D682", nil, "𝚂", "53", nil, "S"}
{"1D683", nil, "𝚃", "54", nil, "T"}
{"1D684", nil, "𝚄", "55", nil, "U"}
{"1D685", nil, "𝚅", "56", nil, "V"}
{"1D686", nil, "𝚆", "57", nil, "W"}
{"1D687", nil, "𝚇", "58", nil, "X"}
{"1D688", nil, "𝚈", "59", nil, "Y"}
{"1D689", nil, "𝚉", "5A", nil, "Z"}
{"1D68A", nil, "𝚊", "61", nil, "a"}
{"1D68B", nil, "𝚋", "62", nil, "b"}
{"1D68C", nil, "𝚌", "63", nil, "c"}
{"1D68D", nil, "𝚍", "64", nil, "d"}
{"1D68E", nil, "𝚎", "65", nil, "e"}
{"1D68F", nil, "𝚏", "66", nil, "f"}
{"1D690", nil, "𝚐", "67", nil, "g"}
{"1D691", nil, "𝚑", "68", nil, "h"}
{"1D692", nil, "𝚒", "69", nil, "i"}
{"1D693", nil, "𝚓", "6A", nil, "j"}
{"1D694", nil, "𝚔", "6B", nil, "k"}
{"1D695", nil, "𝚕", "6C", nil, "l"}
{"1D696", nil, "𝚖", "6D", nil, "m"}
{"1D697", nil, "𝚗", "6E", nil, "n"}
{"1D698", nil, "𝚘", "6F", nil, "o"}
{"1D699", nil, "𝚙", "70", nil, "p"}
{"1D69A", nil, "𝚚", "71", nil, "q"}
{"1D69B", nil, "𝚛", "72", nil, "r"}
{"1D69C", nil, "𝚜", "73", nil, "s"}
{"1D69D", nil, "𝚝", "74", nil, "t"}
{"1D69E", nil, "𝚞", "75", nil, "u"}
{"1D69F", nil, "𝚟", "76", nil, "v"}
{"1D6A0", nil, "𝚠", "77", nil, "w"}
{"1D6A1", nil, "𝚡", "78", nil, "x"}
{"1D6A2", nil, "𝚢", "79", nil, "y"}
{"1D6A3", nil, "𝚣", "7A", nil, "z"}
{"1D6A4", nil, "𝚤", "131", 1, "ı"}
{"1D6A5", nil, "𝚥", "237", 1, "ȷ"}
{"1D6A8", nil, "𝚨", "391", 4398046511104, "Α"}
{"1D6A9", nil, "𝚩", "392", 4398046511104, "Β"}
{"1D6AA", nil, "𝚪", "393", 4398046511104, "Γ"}
{"1D6AB", nil, "𝚫", "394", 4398046511104, "Δ"}
{"1D6AC", nil, "𝚬", "395", 4398046511104, "Ε"}
{"1D6AD", nil, "𝚭", "396", 4398046511104, "Ζ"}
{"1D6AE", nil, "𝚮", "397", 4398046511104, "Η"}
{"1D6AF", nil, "𝚯", "398", 4398046511104, "Θ"}
{"1D6B0", nil, "𝚰", "399", 4398046511104, "Ι"}
{"1D6B1", nil, "𝚱", "39A", 4398046511104, "Κ"}
{"1D6B2", nil, "𝚲", "39B", 4398046511104, "Λ"}
{"1D6B3", nil, "𝚳", "39C", 4398046511104, "Μ"}
{"1D6B4", nil, "𝚴", "39D", 4398046511104, "Ν"}
{"1D6B5", nil, "𝚵", "39E", 4398046511104, "Ξ"}
{"1D6B6", nil, "𝚶", "39F", 4398046511104, "Ο"}
{"1D6B7", nil, "𝚷", "3A0", 4398046511104, "Π"}
{"1D6B8", nil, "𝚸", "3A1", 4398046511104, "Ρ"}
{"1D6B9", nil, "𝚹", "398", 4398046511104, "Θ"}
{"1D6BA", nil, "𝚺", "3A3", 4398046511104, "Σ"}
{"1D6BB", nil, "𝚻", "3A4", 4398046511104, "Τ"}
{"1D6BC", nil, "𝚼", "3A5", 4398046511104, "Υ"}
{"1D6BD", nil, "𝚽", "3A6", 4398046511104, "Φ"}
{"1D6BE", nil, "𝚾", "3A7", 4398046511104, "Χ"}
{"1D6BF", nil, "𝚿", "3A8", 4398046511104, "Ψ"}
{"1D6C0", nil, "𝛀", "3A9", 4398046511104, "Ω"}
{"1D6C2", nil, "𝛂", "3B1", 4398046511104, "α"}
{"1D6C3", nil, "𝛃", "3B2", 4398046511104, "β"}
{"1D6C4", nil, "𝛄", "3B3", 4398046511104, "γ"}
{"1D6C5", nil, "𝛅", "3B4", 4398046511104, "δ"}
{"1D6C6", nil, "𝛆", "3B5", 4398046511104, "ε"}
{"1D6C7", nil, "𝛇", "3B6", 4398046511104, "ζ"}
{"1D6C8", nil, "𝛈", "3B7", 4398046511104, "η"}
{"1D6C9", nil, "𝛉", "3B8", 4398046511104, "θ"}
{"1D6CA", nil, "𝛊", "3B9", 4398046511104, "ι"}
{"1D6CB", nil, "𝛋", "3BA", 4398046511104, "κ"}
{"1D6CC", nil, "𝛌", "3BB", 4398046511104, "λ"}
{"1D6CD", nil, "𝛍", "3BC", 4398046511104, "μ"}
{"1D6CE", nil, "𝛎", "3BD", 4398046511104, "ν"}
{"1D6CF", nil, "𝛏", "3BE", 4398046511104, "ξ"}
{"1D6D0", nil, "𝛐", "3BF", 4398046511104, "ο"}
{"1D6D1", nil, "𝛑", "3C0", 4398046511104, "π"}
{"1D6D2", nil, "𝛒", "3C1", 4398046511104, "ρ"}
{"1D6D3", nil, "𝛓", "3C2", 4398046511104, "ς"}
{"1D6D4", nil, "𝛔", "3C3", 4398046511104, "σ"}
{"1D6D5", nil, "𝛕", "3C4", 4398046511104, "τ"}
{"1D6D6", nil, "𝛖", "3C5", 4398046511104, "υ"}
{"1D6D7", nil, "𝛗", "3C6", 4398046511104, "φ"}
{"1D6D8", nil, "𝛘", "3C7", 4398046511104, "χ"}
{"1D6D9", nil, "𝛙", "3C8", 4398046511104, "ψ"}
{"1D6DA", nil, "𝛚", "3C9", 4398046511104, "ω"}
{"1D6DC", nil, "𝛜", "3B5", 4398046511104, "ε"}
{"1D6DD", nil, "𝛝", "3B8", 4398046511104, "θ"}
{"1D6DE", nil, "𝛞", "3BA", 4398046511104, "κ"}
{"1D6DF", nil, "𝛟", "3C6", 4398046511104, "φ"}
{"1D6E0", nil, "𝛠", "3C1", 4398046511104, "ρ"}
{"1D6E1", nil, "𝛡", "3C0", 4398046511104, "π"}
{"1D6E2", nil, "𝛢", "391", 4398046511104, "Α"}
{"1D6E3", nil, "𝛣", "392", 4398046511104, "Β"}
{"1D6E4", nil, "𝛤", "393", 4398046511104, "Γ"}
{"1D6E5", nil, "𝛥", "394", 4398046511104, "Δ"}
{"1D6E6", nil, "𝛦", "395", 4398046511104, "Ε"}
{"1D6E7", nil, "𝛧", "396", 4398046511104, "Ζ"}
{"1D6E8", nil, "𝛨", "397", 4398046511104, "Η"}
{"1D6E9", nil, "𝛩", "398", 4398046511104, "Θ"}
{"1D6EA", nil, "𝛪", "399", 4398046511104, "Ι"}
{"1D6EB", nil, "𝛫", "39A", 4398046511104, "Κ"}
{"1D6EC", nil, "𝛬", "39B", 4398046511104, "Λ"}
{"1D6ED", nil, "𝛭", "39C", 4398046511104, "Μ"}
{"1D6EE", nil, "𝛮", "39D", 4398046511104, "Ν"}
{"1D6EF", nil, "𝛯", "39E", 4398046511104, "Ξ"}
{"1D6F0", nil, "𝛰", "39F", 4398046511104, "Ο"}
{"1D6F1", nil, "𝛱", "3A0", 4398046511104, "Π"}
{"1D6F2", nil, "𝛲", "3A1", 4398046511104, "Ρ"}
{"1D6F3", nil, "𝛳", "398", 4398046511104, "Θ"}
{"1D6F4", nil, "𝛴", "3A3", 4398046511104, "Σ"}
{"1D6F5", nil, "𝛵", "3A4", 4398046511104, "Τ"}
{"1D6F6", nil, "𝛶", "3A5", 4398046511104, "Υ"}
{"1D6F7", nil, "𝛷", "3A6", 4398046511104, "Φ"}
{"1D6F8", nil, "𝛸", "3A7", 4398046511104, "Χ"}
{"1D6F9", nil, "𝛹", "3A8", 4398046511104, "Ψ"}
{"1D6FA", nil, "𝛺", "3A9", 4398046511104, "Ω"}
{"1D6FC", nil, "𝛼", "3B1", 4398046511104, "α"}
{"1D6FD", nil, "𝛽", "3B2", 4398046511104, "β"}
{"1D6FE", nil, "𝛾", "3B3", 4398046511104, "γ"}
{"1D6FF", nil, "𝛿", "3B4", 4398046511104, "δ"}
{"1D700", nil, "𝜀", "3B5", 4398046511104, "ε"}
{"1D701", nil, "𝜁", "3B6", 4398046511104, "ζ"}
{"1D702", nil, "𝜂", "3B7", 4398046511104, "η"}
{"1D703", nil, "𝜃", "3B8", 4398046511104, "θ"}
{"1D704", nil, "𝜄", "3B9", 4398046511104, "ι"}
{"1D705", nil, "𝜅", "3BA", 4398046511104, "κ"}
{"1D706", nil, "𝜆", "3BB", 4398046511104, "λ"}
{"1D707", nil, "𝜇", "3BC", 4398046511104, "μ"}
{"1D708", nil, "𝜈", "3BD", 4398046511104, "ν"}
{"1D709", nil, "𝜉", "3BE", 4398046511104, "ξ"}
{"1D70A", nil, "𝜊", "3BF", 4398046511104, "ο"}
{"1D70B", nil, "𝜋", "3C0", 4398046511104, "π"}
{"1D70C", nil, "𝜌", "3C1", 4398046511104, "ρ"}
{"1D70D", nil, "𝜍", "3C2", 4398046511104, "ς"}
{"1D70E", nil, "𝜎", "3C3", 4398046511104, "σ"}
{"1D70F", nil, "𝜏", "3C4", 4398046511104, "τ"}
{"1D710", nil, "𝜐", "3C5", 4398046511104, "υ"}
{"1D711", nil, "𝜑", "3C6", 4398046511104, "φ"}
{"1D712", nil, "𝜒", "3C7", 4398046511104, "χ"}
{"1D713", nil, "𝜓", "3C8", 4398046511104, "ψ"}
{"1D714", nil, "𝜔", "3C9", 4398046511104, "ω"}
{"1D716", nil, "𝜖", "3B5", 4398046511104, "ε"}
{"1D717", nil, "𝜗", "3B8", 4398046511104, "θ"}
{"1D718", nil, "𝜘", "3BA", 4398046511104, "κ"}
{"1D719", nil, "𝜙", "3C6", 4398046511104, "φ"}
{"1D71A", nil, "𝜚", "3C1", 4398046511104, "ρ"}
{"1D71B", nil, "𝜛", "3C0", 4398046511104, "π"}
{"1D71C", nil, "𝜜", "391", 4398046511104, "Α"}
{"1D71D", nil, "𝜝", "392", 4398046511104, "Β"}
{"1D71E", nil, "𝜞", "393", 4398046511104, "Γ"}
{"1D71F", nil, "𝜟", "394", 4398046511104, "Δ"}
{"1D720", nil, "𝜠", "395", 4398046511104, "Ε"}
{"1D721", nil, "𝜡", "396", 4398046511104, "Ζ"}
{"1D722", nil, "𝜢", "397", 4398046511104, "Η"}
{"1D723", nil, "𝜣", "398", 4398046511104, "Θ"}
{"1D724", nil, "𝜤", "399", 4398046511104, "Ι"}
{"1D725", nil, "𝜥", "39A", 4398046511104, "Κ"}
{"1D726", nil, "𝜦", "39B", 4398046511104, "Λ"}
{"1D727", nil, "𝜧", "39C", 4398046511104, "Μ"}
{"1D728", nil, "𝜨", "39D", 4398046511104, "Ν"}
{"1D729", nil, "𝜩", "39E", 4398046511104, "Ξ"}
{"1D72A", nil, "𝜪", "39F", 4398046511104, "Ο"}
{"1D72B", nil, "𝜫", "3A0", 4398046511104, "Π"}
{"1D72C", nil, "𝜬", "3A1", 4398046511104, "Ρ"}
{"1D72D", nil, "𝜭", "398", 4398046511104, "Θ"}
{"1D72E", nil, "𝜮", "3A3", 4398046511104, "Σ"}
{"1D72F", nil, "𝜯", "3A4", 4398046511104, "Τ"}
{"1D730", nil, "𝜰", "3A5", 4398046511104, "Υ"}
{"1D731", nil, "𝜱", "3A6", 4398046511104, "Φ"}
{"1D732", nil, "𝜲", "3A7", 4398046511104, "Χ"}
{"1D733", nil, "𝜳", "3A8", 4398046511104, "Ψ"}
{"1D734", nil, "𝜴", "3A9", 4398046511104, "Ω"}
{"1D736", nil, "𝜶", "3B1", 4398046511104, "α"}
{"1D737", nil, "𝜷", "3B2", 4398046511104, "β"}
{"1D738", nil, "𝜸", "3B3", 4398046511104, "γ"}
{"1D739", nil, "𝜹", "3B4", 4398046511104, "δ"}
{"1D73A", nil, "𝜺", "3B5", 4398046511104, "ε"}
{"1D73B", nil, "𝜻", "3B6", 4398046511104, "ζ"}
{"1D73C", nil, "𝜼", "3B7", 4398046511104, "η"}
{"1D73D", nil, "𝜽", "3B8", 4398046511104, "θ"}
{"1D73E", nil, "𝜾", "3B9", 4398046511104, "ι"}
{"1D73F", nil, "𝜿", "3BA", 4398046511104, "κ"}
{"1D740", nil, "𝝀", "3BB", 4398046511104, "λ"}
{"1D741", nil, "𝝁", "3BC", 4398046511104, "μ"}
{"1D742", nil, "𝝂", "3BD", 4398046511104, "ν"}
{"1D743", nil, "𝝃", "3BE", 4398046511104, "ξ"}
{"1D744", nil, "𝝄", "3BF", 4398046511104, "ο"}
{"1D745", nil, "𝝅", "3C0", 4398046511104, "π"}
{"1D746", nil, "𝝆", "3C1", 4398046511104, "ρ"}
{"1D747", nil, "𝝇", "3C2", 4398046511104, "ς"}
{"1D748", nil, "𝝈", "3C3", 4398046511104, "σ"}
{"1D749", nil, "𝝉", "3C4", 4398046511104, "τ"}
{"1D74A", nil, "𝝊", "3C5", 4398046511104, "υ"}
{"1D74B", nil, "𝝋", "3C6", 4398046511104, "φ"}
{"1D74C", nil, "𝝌", "3C7", 4398046511104, "χ"}
{"1D74D", nil, "𝝍", "3C8", 4398046511104, "ψ"}
{"1D74E", nil, "𝝎", "3C9", 4398046511104, "ω"}
{"1D750", nil, "𝝐", "3B5", 4398046511104, "ε"}
{"1D751", nil, "𝝑", "3B8", 4398046511104, "θ"}
{"1D752", nil, "𝝒", "3BA", 4398046511104, "κ"}
{"1D753", nil, "𝝓", "3C6", 4398046511104, "φ"}
{"1D754", nil, "𝝔", "3C1", 4398046511104, "ρ"}
{"1D755", nil, "𝝕", "3C0", 4398046511104, "π"}
{"1D756", nil, "𝝖", "391", 4398046511104, "Α"}
{"1D757", nil, "𝝗", "392", 4398046511104, "Β"}
{"1D758", nil, "𝝘", "393", 4398046511104, "Γ"}
{"1D759", nil, "𝝙", "394", 4398046511104, "Δ"}
{"1D75A", nil, "𝝚", "395", 4398046511104, "Ε"}
{"1D75B", nil, "𝝛", "396", 4398046511104, "Ζ"}
{"1D75C", nil, "𝝜", "397", 4398046511104, "Η"}
{"1D75D", nil, "𝝝", "398", 4398046511104, "Θ"}
{"1D75E", nil, "𝝞", "399", 4398046511104, "Ι"}
{"1D75F", nil, "𝝟", "39A", 4398046511104, "Κ"}
{"1D760", nil, "𝝠", "39B", 4398046511104, "Λ"}
{"1D761", nil, "𝝡", "39C", 4398046511104, "Μ"}
{"1D762", nil, "𝝢", "39D", 4398046511104, "Ν"}
{"1D763", nil, "𝝣", "39E", 4398046511104, "Ξ"}
{"1D764", nil, "𝝤", "39F", 4398046511104, "Ο"}
{"1D765", nil, "𝝥", "3A0", 4398046511104, "Π"}
{"1D766", nil, "𝝦", "3A1", 4398046511104, "Ρ"}
{"1D767", nil, "𝝧", "398", 4398046511104, "Θ"}
{"1D768", nil, "𝝨", "3A3", 4398046511104, "Σ"}
{"1D769", nil, "𝝩", "3A4", 4398046511104, "Τ"}
{"1D76A", nil, "𝝪", "3A5", 4398046511104, "Υ"}
{"1D76B", nil, "𝝫", "3A6", 4398046511104, "Φ"}
{"1D76C", nil, "𝝬", "3A7", 4398046511104, "Χ"}
{"1D76D", nil, "𝝭", "3A8", 4398046511104, "Ψ"}
{"1D76E", nil, "𝝮", "3A9", 4398046511104, "Ω"}
{"1D770", nil, "𝝰", "3B1", 4398046511104, "α"}
{"1D771", nil, "𝝱", "3B2", 4398046511104, "β"}
{"1D772", nil, "𝝲", "3B3", 4398046511104, "γ"}
{"1D773", nil, "𝝳", "3B4", 4398046511104, "δ"}
{"1D774", nil, "𝝴", "3B5", 4398046511104, "ε"}
{"1D775", nil, "𝝵", "3B6", 4398046511104, "ζ"}
{"1D776", nil, "𝝶", "3B7", 4398046511104, "η"}
{"1D777", nil, "𝝷", "3B8", 4398046511104, "θ"}
{"1D778", nil, "𝝸", "3B9", 4398046511104, "ι"}
{"1D779", nil, "𝝹", "3BA", 4398046511104, "κ"}
{"1D77A", nil, "𝝺", "3BB", 4398046511104, "λ"}
{"1D77B", nil, "𝝻", "3BC", 4398046511104, "μ"}
{"1D77C", nil, "𝝼", "3BD", 4398046511104, "ν"}
{"1D77D", nil, "𝝽", "3BE", 4398046511104, "ξ"}
{"1D77E", nil, "𝝾", "3BF", 4398046511104, "ο"}
{"1D77F", nil, "𝝿", "3C0", 4398046511104, "π"}
{"1D780", nil, "𝞀", "3C1", 4398046511104, "ρ"}
{"1D781", nil, "𝞁", "3C2", 4398046511104, "ς"}
{"1D782", nil, "𝞂", "3C3", 4398046511104, "σ"}
{"1D783", nil, "𝞃", "3C4", 4398046511104, "τ"}
{"1D784", nil, "𝞄", "3C5", 4398046511104, "υ"}
{"1D785", nil, "𝞅", "3C6", 4398046511104, "φ"}
{"1D786", nil, "𝞆", "3C7", 4398046511104, "χ"}
{"1D787", nil, "𝞇", "3C8", 4398046511104, "ψ"}
{"1D788", nil, "𝞈", "3C9", 4398046511104, "ω"}
{"1D78A", nil, "𝞊", "3B5", 4398046511104, "ε"}
{"1D78B", nil, "𝞋", "3B8", 4398046511104, "θ"}
{"1D78C", nil, "𝞌", "3BA", 4398046511104, "κ"}
{"1D78D", nil, "𝞍", "3C6", 4398046511104, "φ"}
{"1D78E", nil, "𝞎", "3C1", 4398046511104, "ρ"}
{"1D78F", nil, "𝞏", "3C0", 4398046511104, "π"}
{"1D790", nil, "𝞐", "391", 4398046511104, "Α"}
{"1D791", nil, "𝞑", "392", 4398046511104, "Β"}
{"1D792", nil, "𝞒", "393", 4398046511104, "Γ"}
{"1D793", nil, "𝞓", "394", 4398046511104, "Δ"}
{"1D794", nil, "𝞔", "395", 4398046511104, "Ε"}
{"1D795", nil, "𝞕", "396", 4398046511104, "Ζ"}
{"1D796", nil, "𝞖", "397", 4398046511104, "Η"}
{"1D797", nil, "𝞗", "398", 4398046511104, "Θ"}
{"1D798", nil, "𝞘", "399", 4398046511104, "Ι"}
{"1D799", nil, "𝞙", "39A", 4398046511104, "Κ"}
{"1D79A", nil, "𝞚", "39B", 4398046511104, "Λ"}
{"1D79B", nil, "𝞛", "39C", 4398046511104, "Μ"}
{"1D79C", nil, "𝞜", "39D", 4398046511104, "Ν"}
{"1D79D", nil, "𝞝", "39E", 4398046511104, "Ξ"}
{"1D79E", nil, "𝞞", "39F", 4398046511104, "Ο"}
{"1D79F", nil, "𝞟", "3A0", 4398046511104, "Π"}
{"1D7A0", nil, "𝞠", "3A1", 4398046511104, "Ρ"}
{"1D7A1", nil, "𝞡", "398", 4398046511104, "Θ"}
{"1D7A2", nil, "𝞢", "3A3", 4398046511104, "Σ"}
{"1D7A3", nil, "𝞣", "3A4", 4398046511104, "Τ"}
{"1D7A4", nil, "𝞤", "3A5", 4398046511104, "Υ"}
{"1D7A5", nil, "𝞥", "3A6", 4398046511104, "Φ"}
{"1D7A6", nil, "𝞦", "3A7", 4398046511104, "Χ"}
{"1D7A7", nil, "𝞧", "3A8", 4398046511104, "Ψ"}
{"1D7A8", nil, "𝞨", "3A9", 4398046511104, "Ω"}
{"1D7AA", nil, "𝞪", "3B1", 4398046511104, "α"}
{"1D7AB", nil, "𝞫", "3B2", 4398046511104, "β"}
{"1D7AC", nil, "𝞬", "3B3", 4398046511104, "γ"}
{"1D7AD", nil, "𝞭", "3B4", 4398046511104, "δ"}
{"1D7AE", nil, "𝞮", "3B5", 4398046511104, "ε"}
{"1D7AF", nil, "𝞯", "3B6", 4398046511104, "ζ"}
{"1D7B0", nil, "𝞰", "3B7", 4398046511104, "η"}
{"1D7B1", nil, "𝞱", "3B8", 4398046511104, "θ"}
{"1D7B2", nil, "𝞲", "3B9", 4398046511104, "ι"}
{"1D7B3", nil, "𝞳", "3BA", 4398046511104, "κ"}
{"1D7B4", nil, "𝞴", "3BB", 4398046511104, "λ"}
{"1D7B5", nil, "𝞵", "3BC", 4398046511104, "μ"}
{"1D7B6", nil, "𝞶", "3BD", 4398046511104, "ν"}
{"1D7B7", nil, "𝞷", "3BE", 4398046511104, "ξ"}
{"1D7B8", nil, "𝞸", "3BF", 4398046511104, "ο"}
{"1D7B9", nil, "𝞹", "3C0", 4398046511104, "π"}
{"1D7BA", nil, "𝞺", "3C1", 4398046511104, "ρ"}
{"1D7BB", nil, "𝞻", "3C2", 4398046511104, "ς"}
{"1D7BC", nil, "𝞼", "3C3", 4398046511104, "σ"}
{"1D7BD", nil, "𝞽", "3C4", 4398046511104, "τ"}
{"1D7BE", nil, "𝞾", "3C5", 4398046511104, "υ"}
{"1D7BF", nil, "𝞿", "3C6", 4398046511104, "φ"}
{"1D7C0", nil, "𝟀", "3C7", 4398046511104, "χ"}
{"1D7C1", nil, "𝟁", "3C8", 4398046511104, "ψ"}
{"1D7C2", nil, "𝟂", "3C9", 4398046511104, "ω"}
{"1D7C4", nil, "𝟄", "3B5", 4398046511104, "ε"}
{"1D7C5", nil, "𝟅", "3B8", 4398046511104, "θ"}
{"1D7C6", nil, "𝟆", "3BA", 4398046511104, "κ"}
{"1D7C7", nil, "𝟇", "3C6", 4398046511104, "φ"}
{"1D7C8", nil, "𝟈", "3C1", 4398046511104, "ρ"}
{"1D7C9", nil, "𝟉", "3C0", 4398046511104, "π"}
{"1D7CA", nil, "𝟊", "3DC", 4398046511104, "Ϝ"}
{"1D7CB", nil, "𝟋", "3DD", 4398046511104, "ϝ"}
{"1D7CE", nil, "𝟎", "30", nil, "0"}
{"1D7CF", nil, "𝟏", "31", nil, "1"}
{"1D7D0", nil, "𝟐", "32", nil, "2"}
{"1D7D1", nil, "𝟑", "33", nil, "3"}
{"1D7D2", nil, "𝟒", "34", nil, "4"}
{"1D7D3", nil, "𝟓", "35", nil, "5"}
{"1D7D4", nil, "𝟔", "36", nil, "6"}
{"1D7D5", nil, "𝟕", "37", nil, "7"}
{"1D7D6", nil, "𝟖", "38", nil, "8"}
{"1D7D7", nil, "𝟗", "39", nil, "9"}
{"1D7D8", nil, "𝟘", "30", nil, "0"}
{"1D7D9", nil, "𝟙", "31", nil, "1"}
{"1D7DA", nil, "𝟚", "32", nil, "2"}
{"1D7DB", nil, "𝟛", "33", nil, "3"}
{"1D7DC", nil, "𝟜", "34", nil, "4"}
{"1D7DD", nil, "𝟝", "35", nil, "5"}
{"1D7DE", nil, "𝟞", "36", nil, "6"}
{"1D7DF", nil, "𝟟", "37", nil, "7"}
{"1D7E0", nil, "𝟠", "38", nil, "8"}
{"1D7E1", nil, "𝟡", "39", nil, "9"}
{"1D7E2", nil, "𝟢", "30", nil, "0"}
{"1D7E3", nil, "𝟣", "31", nil, "1"}
{"1D7E4", nil, "𝟤", "32", nil, "2"}
{"1D7E5", nil, "𝟥", "33", nil, "3"}
{"1D7E6", nil, "𝟦", "34", nil, "4"}
{"1D7E7", nil, "𝟧", "35", nil, "5"}
{"1D7E8", nil, "𝟨", "36", nil, "6"}
{"1D7E9", nil, "𝟩", "37", nil, "7"}
{"1D7EA", nil, "𝟪", "38", nil, "8"}
{"1D7EB", nil, "𝟫", "39", nil, "9"}
{"1D7EC", nil, "𝟬", "30", nil, "0"}
{"1D7ED", nil, "𝟭", "31", nil, "1"}
{"1D7EE", nil, "𝟮", "32", nil, "2"}
{"1D7EF", nil, "𝟯", "33", nil, "3"}
{"1D7F0", nil, "𝟰", "34", nil, "4"}
{"1D7F1", nil, "𝟱", "35", nil, "5"}
{"1D7F2", nil, "𝟲", "36", nil, "6"}
{"1D7F3", nil, "𝟳", "37", nil, "7"}
{"1D7F4", nil, "𝟴", "38", nil, "8"}
{"1D7F5", nil, "𝟵", "39", nil, "9"}
{"1D7F6", nil, "𝟶", "30", nil, "0"}
{"1D7F7", nil, "𝟷", "31", nil, "1"}
{"1D7F8", nil, "𝟸", "32", nil, "2"}
{"1D7F9", nil, "𝟹", "33", nil, "3"}
{"1D7FA", nil, "𝟺", "34", nil, "4"}
{"1D7FB", nil, "𝟻", "35", nil, "5"}
{"1D7FC", nil, "𝟼", "36", nil, "6"}
{"1D7FD", nil, "𝟽", "37", nil, "7"}
{"1D7FE", nil, "𝟾", "38", nil, "8"}
{"1D7FF", nil, "𝟿", "39", nil, "9"}

My suggestion would be for us to allow all characters that are normalized to greek with a scriptset of "ALL - GREEK". This means 1D6A8..1D7CB plus the micro sign.

WDYT?

@josevalim
Copy link
Member

Actually, ignore me. This is not a good proposal because we could have confusion between the mathematical symbol A and the ascii letter A. So we would need to judge characters based on their similarity, which is a job done by the Unicode standard. Therefore I can see two options:

  1. We permanently support only micro as everything but greek

  2. Or we allow all Not_NFKC characters to be parsed but later warn (keeping backwards compability, albeit with a warning)

I am really inclined to go with 1 though, because micro is a recognized SI unit and being unable to use it is very limiting IMO.

@mrluc
Copy link
Contributor

mrluc commented Apr 10, 2022

@josevalim that makes sense, and as highlighted in PR, any mechanism for documenting exceptions is in the spirit of the UTF standard.

(Edit: The following relates to the 'Greek micro, Common micro distinction is annoying' question only, not to the actual error)

Data points on how it's handled 2 other places that I saw, one of which we may want to imitate:

  • Rust doesn't allow that Common micro -- Restricted! -- only the Greek one. As commented in the other PR, this is a bummer because eg. OSX lets us type 'Opt + m' and type the mathematical, non-Greek one.

  • Julia does custom normalization -- they translate the two micro characters to the same character when parsing! Which makes sense, though it blew some people's minds that bytes/different codepoints are treated as the same, and of course it had to be restricted to a few extremely similar characters, not all confusables. Here's how they do it: they extended, and eventually took over maintenance of, a C utf library that's vendored/included in the Julia implementation. Their extensions were around configurable normalization, by providing a codepoint mapping function, like a pre-processing step, which they used in this PR initially just to 'canonicalize' micro and epsilon! My recollection was that they do maintain those mappings basically 'by hand', that may or may not be how it's done now.

So, while we may feel a bit cheesy doing it 'by hand', we'd be in good company! 😄

We could normalize like that too -- maintain a map of characters that should just always be translated to a specific 'canonical' character as needed.

How to get around 'Restricted'? Subtract the by-hand list of mapped codepoints from the list of Restricted codepoints.

  • For comparison, Julia doesn't implement UTS39 protections; a certain class of confusable issues are prevented because of their normalization, but only the ones involving confusable eg. micro or mathematical characters, which are limited to curated symbols, for reasons they discuss -- eg. 'why not canonicalize all confusable things?' Which does seem practical at first, until you realize some people will actually want to write mathy code where the italic/bold/fullwidth variants of a letter should be considered different -- and also, plenty of other cases where confusability would NOT be a good fit in practice, like lower-case Greek omega being confusable with 'w'.

How to prevent mixed-script issues with eg. a Greek char in a Latin string? Canonicalize to the Common codepoint, not the Greek codepoint.

Edit: ie, the mapping would be from greek_micro => common_micro, and that replacement would happen before all other processing; the number of exceptions being low could make it decently fast. 'Common' codepoints (like underscore, or latin number one) are ignored for script-mixing -- so while it would be funny that, internally, a greek variable name would be turned from being all-greek (like {Greek}, {Greek}, {Greek} resolved script sets) into potentially {Greek, Common, Greek}, from the standpoint of script mixing it'd be exempt, and from the standpoint of confusability there'd only ever be one of the 2 codepoints.

@mrluc
Copy link
Contributor

mrluc commented Apr 10, 2022

@josevalim sorry for conversation in 2 places -- I posted with a potential root cause in the PR.

@josevalim
Copy link
Member

Closing in favor of #11753.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

3 participants