IRB support

[st0012]

Following @jemmaissroff's suggestion, here's a list of things IRB'd need in order to use YARP instead of Ripper:

• A way to simulate the output of Ripper.sexp. Maybe something like YARP.lex(code).value.to_array?
  • https://github.com/ruby/irb/blob/master/lib/irb.rb#L889
  • https://github.com/ruby/irb/blob/df6907aca9c618561537e37da0ef0626f675c42c/lib/irb/cmd/nop.rb#L28
• A way to do simulate Ripper::Lexer#scan. I believe YARP.lex_compat is created for this.
  • However, the result needs to have :on_sp token as noted in Investigate emitting :on_sp event for ripper lex #722
  • https://github.com/ruby/irb/blob/df6907aca9c618561537e37da0ef0626f675c42c/lib/irb/color.rb#L200
  • https://github.com/ruby/irb/blob/df6907aca9c618561537e37da0ef0626f675c42c/lib/irb/ruby-lex.rb#L164
• IRB also needs a way to map Ripper's state bits, like Ripper::EXPR_CMDARG with YARP's, and access them through similar constants, like YARP::EXPR_CMDARG?
  • Perhaps IRB could first simplify/change its state checks to make this easier? (example). @tompng is it possible we reduce RubyLex 's coupling on Ripper state bits?
• Finally, IRB will need a way to know if the current parser is YARP so it can switch to use different APIs/constants.

Let me know if I misunderstood anything or what I can improve from the IRB part 🙏

[eregon]

Regarding :on_sp is it really necessary? (not for/against it, I just want to understand the need for it)
From what I understand of #722 (calculating leading whitespace), maybe it could be done with just line[/^\s*/]?

[tompng]

is it possible we reduce RubyLex's coupling on Ripper state bits?

Yes, it's possible. For example in ruby/irb#500, use of token.state.allbits?(Ripper::EXPR_SOMETHING) is reduced. (17 deletion, 4 addition)

[kddnewton]

For the state bits, you can just use Ripper::EXPR_CMDARG, they map 1:1. We could copy over the constants, but in the interest of getting it working, I would say just use the existing constants.

Something I want to know is if you are trying to just use the Ripper compatibility layer or if you want to migrate fully over to using YARP.lex. Personally, I would much prefer YARP.lex. There are a couple of benefits:

• It's going to be faster. Instead of going through a translation layer that goes YARP -> Ripper -> IRB you would cut out the middle step.
• It's going to be more precise. YARP keeps all of its tokens distinct, as opposed to Ripper which groups tokens. (Tokens like on_kw and on_op are grouped in Ripper, whereas in YARP they are things like YP_TOKEN_KEYWORD_BEGIN or YP_TOKEN_PLUS.) That would mean you could refactor things like the following places to be much more readable:
  • https://github.com/ruby/irb/blob/master/lib/irb/ruby-lex.rb#L269
  • https://github.com/ruby/irb/blob/master/lib/irb/ruby-lex.rb#L405-L418
  • https://github.com/ruby/irb/blob/master/lib/irb/ruby-lex.rb#L459
  • https://github.com/ruby/irb/blob/master/lib/irb/ruby-lex.rb#L545-L568
  • https://github.com/ruby/irb/blob/master/lib/irb/ruby-lex.rb#L640-L675
  • https://github.com/ruby/irb/blob/master/lib/irb/ruby-lex.rb#L814
• The other reason is that if you have any issues with the lexer we could find them sooner rather than later. It would be nice if before we shipped 3.3 we could have IRB running in YARP.lex so that if there is anything you would want we can have it in the first version released.

[kddnewton]

Hey @st0012 - I know it's been half a year, but I just wanted to check in on where we are with this. I know a lot of changes have gone into IRB since you opened this issue. Is everything here still relevant/necessary?

[st0012]

When I opened the issue, I assumed IRB could adopt Prism when it's ready. But because IRB still needs to support Ruby 2.7 for a while, we won't be able to do that for at least another half year.
So I'm closing this issue now and will revisit it later when we're about to adopt it.