-
-
Notifications
You must be signed in to change notification settings - Fork 91
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add semantic tokenization tests #5381
Conversation
3b943ea
to
5a742d1
Compare
Todos and things to consider for a follow-up PR:
Notes in case we decide to capture all symbols some day:currently, we are not tokenizing the following:
|
Those are great comments - I added a link in #5259 to that comment so we don't lose track.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work here - shame the auto-formatter does at it does, but those tests will look tidier soon. We'll be thankful for these tests.
This PR fixes semantic tokenization, and adds tests for it