Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Cyrillic and Greek support #100

Merged
merged 2 commits into from
Jul 24, 2024
Merged

Conversation

lsimeonov
Copy link

Added support for Greek and Cyrillic unicodes

This is done by tracking the byte count and correctly handling the multibyte strings

Added simple test for this

parse.go Outdated
(unicode.Is(first, r) || unicode.Is(second, r) || string(r) == "*")
(unicode.Is(first, r) ||
unicode.Is(second, r) ||
unicode.Is(cyrillic, r) ||
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should we need add this new condition for cyrillic and greek. We already knows Non-English letters yet.

@zhengchun
Copy link
Contributor

Thanks for your work. I have some question about this code:

unicode.Is(cyrillic, r) || unicode.Is(greek, r)

Why should we need this condition. Removing this code still can works, I test now supports Non-English like Chinese or another language now.

Thanks again.

@lsimeonov
Copy link
Author

Good catch. That was a leftover test from removed code. The idea was to allow certain charsets, but it’s better left to the user to decide post-parsing.

I removed the change

@zhengchun zhengchun merged commit 4286dab into antchfx:master Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants