Plugin features

[lppedd]

Hey @Aloso, need help to come up with a regex that supports all the symbols Pomsky accepts as valid for group names and variable names. Basically I need to replace a-zA-Z with the actual range.

idea-pomsky/src/main/java/com/github/lppedd/idea/pomsky/lang/lexer/pomsky.flex

Line 39 in 430e722

Identifier = [a-zA-Z_][a-zA-Z_0-9]*

See https://pomsky-lang.org/docs/reference/grammar/#name

[Aloso]

Pomsky's lexer uses the is_alphabetic and is_alphanumeric functions to determine whether a code point is a letter or number.

Concretely, this means that an identifier must consist of

• code points with the Alphabetic property, which you can match in Java with \p{Alpha}
• code points from the Number general categories, which you can match in Java with \pN
• _

Note that an identifier must not start with a number, so the first code point is either a letter or _. So you can match an identifier with

(?U)[\p{Alpha}_][\p{Alpha}\pN_]*

The (?U) is needed to make Java's regex engine Unicode-aware.

Also note that capturing group names must be ASCII-only (so they could be matched with [a-zA-Z_][a-zA-Z_0-9]*). However, I currently don't distinguish between different kinds of identifiers in the lexer and only check if a capturing group name is valid in the parser, which simplifies things a bit.

[lppedd]

@Aloso thanks for the previous answer!

Now, I just want to check if the direction I'm taking with the plugin is ok.
See the following GIF.

This is currently invoking the CLI executable, which has been set in the Pomsky settings.

[lppedd]

@Aloso btw

so I'll work on a flag to enable JSON output next

In what format would you like to do that?
Are you going to parse the entire script too and report all the errors/warning, or only the first one?

Just consider if JSON is worth the time, when you can just ship a native library.

Another flag worth support is --flavors. This will let the IDE know what the CLI supports, in case you plan on adding other stuff in the future.

[Aloso]

The JSON output will report the same diagnostics as the "normal" CLI, so it may report multiple errors/warnings at once. Note that this is a work in progress, because Pomsky currently can't recover from parser errors, which I'd like to change at some point so Pomsky can report all parser errors at once.

Just consider if JSON is worth the time, when you can just ship a native library.

My concern is that, given the requirement to use the installed version of Pomsky, using a native library adds a lot of complexity that's not worth the rather small performance gain. Furthermore, pomsky's library API is still very much in flux, but if I publish a dynamic library, I have to commit to an API that I can't easily change later. With JSON, I can add fields freely.

Another flag worth support is --flavors. This will let the IDE know what the CLI supports, in case you plan on adding other stuff in the future.

This is a good idea!

[lppedd]

@Aloso I've added the compile time at the bottom of the editor.
Btw, the compile process is not modal/blocking, and it can be canceled anytime by the user.

[lppedd]

@Aloso are you up to test it out on the Rust IDE? Requires 2022.3.
I'm interested in the initial workflow. If it feels natural or not.

idea-pomsky-0.0.1.zip

[Aloso]

The plugin doesn't have an image yet. Here's an SVG you can use:

Here are some things I noticed that could be improved. Sorry for the long list, this isn't meant as criticism 😄

UI

• The toolbar with the buttons to show/hide the preview is bugging me. It seems wasteful to add an entire toolbar just to show two buttons. Could you maybe make them more like the buttons in the markdown editor?
• Since IntelliJ IDEA is first and foremost a Java/Kotlin IDE, it would make sense to make Java the default Regex flavor in the dropdown (or make this configurable in the settings).

UX

• Strings aren't closed automatically
• Pressing enter in an indented line adds indentation, but pressing backspace on an indented line doesn't remove it in one go
• Pressing enter between parentheses should increase the indentation
• Extending the selection with Ctrl+W works well, but there should be more intermediate steps: In the expression 'a' | 'b' 'c'* extending the selection from the c should first select 'c', then 'c'*, then 'b' 'c'*, then everything

Parser

• base isn't considered to be a keyword
• if and else are reserved words; they should be treated as keywords since it's forbidden to use them
• Start and End are considered keywords, and that was correct before version 0.6, but now they're just built-in variables, like Codepoint, Grapheme, C and G
• Backreferences without a number or group name (::) produce no error
• 'this'+? produces no error. Repetitions can't be chained, so that +? can't be confused with a lazy repetition.
• The red squiggly underline of a parser error is sometimes at the token after the error, e.g. when entering @test
• Error messages always announces what was expected, but these things are sometimes wrong, e.g. "range or regex expected" after a !

Other

• Scoping doesn't work when renaming variables or going to the definition. Test case:

  let x = 'test';
  let y = ( let x = 'test'; x );
  let z = ( let x = 'test'; x );
  x y z

[lppedd]

@Aloso

Thanks for your report. No problem, I accept criticisms 😄
Let me just say one thing: as the first release will be 0.1.0, I don't expect to support everything, or that everything works as expected.
Especially on the language side (lexing/parsing/formatting) there is a lot of work to do that is going to take a lot of time, not only two weeks.

My target for the first release is to allow people to edit Pomsky files and compile them without switching to the Playground.
And obviously to gather feedbacks, if it's somewhat useful or not.

UI

• Could you maybe make them more like the buttons in the markdown editor

  Not really. This is what I did in the beginning.
  The location of the invisibile toolbar overlaps with the preview editor header, making it really hard to click on "Compile".

• it would make sense to make Java the default Regex flavor

  That's a valid suggestion. I'll set the default depending on the product (IDEA: Java, PyCharm: Python, Rider: .NET, WebStorm: JavaScript).
  Fixed with 5fc550d

UX

• Strings aren't closed automatically

  This should be handled. Could you post a reproducer GIF or text example I can use?

Parser

• base isn't considered to be a keyword

  Fixed. Thanks! Along with if and else.

Other

• Scoping doesn't work

  An issue introduced with recent grammar changes. Fixed with 6d2f55b

[lppedd]

@Aloso I'll add code completion for variables and named groups and then I'll consider version 0.1.0 complete.
Ready by the end of next week.

[Aloso]

Thanks for letting me know! I was planning to release Pomsky 0.9 (with JSON support) tomorrow, so you can work on showing diagnostics. But if that functionality doesn't make it into the first release, don't worry! I'm very pleased by the rapid progress this plugin makes, and I don't want to pressure you into doing more work than you want to 🙂

[lppedd]

@Aloso the only request I'd like to make for diagnostics using JSON, is that I'd like to also show the standard beautified error report on the right side. That means the JSON output should also include that. Is this an unreasonable request?

I'm pretty happy with the current result too. Usually I do this stuff for small internal languages or tools, but not in this short time and not as refined (although lacking functionalities, this has already a great foundation).

[Aloso]

Diagnostics in the JSON output now have a visual field with the formatted error/warning.

Pomsky 0.9 is released now here's the announcement.

[lppedd]

@Aloso thank you! I'm left with handling built-in variables/properties, and then I'll give you a second build to test.

[lppedd]

@Aloso I've updated the plugin to cover the latest changes in 0.10.0.
Almost ready to publish to marketplace. Maybe tomorrow.

[lppedd]

@Aloso thanks! Done.
Is the --json output documented somewhere? I'm adding the CLI error reporting capability and would like to know what each field represents.

[lppedd]

Ah nevermind for now. I think it's best to wait until Pomsky report all errors, not only the first one.

[lppedd]

@Aloso for completeness, what do you want the name to be on the JetBrains Marketplace? IDEA Pomsky, or just Pomsky?

[Aloso]

It should be named "Pomsky".

[lppedd]

@Aloso perfect, thanks