Replacing Ripper by Prism

[eregon]

Description

It would be great to replace the usages of Ripper in IRB by Prism.
There are many advantages:

• Prism is faster than Ripper, and the difference is even more visible on TruffleRuby, where irb is visibly less reactive because of Ripper.
• Prism provides a much better API
• Prism works well on all Ruby implementations, Ripper is a big hack which is very hard to support on all Ruby implementations (at least it has proven difficult to maintain on CRuby, JRuby and TruffleRuby, right now TruffleRuby is updating to Ruby 3.3 and Ripper is proving to be again a pain to get working, it even has a copy of st.c now to give an idea, on top of depending on many internal CRuby files).
• Prism is actually used in CRuby now as the default parser since Ruby 3.4 preview 2 and is the only parser in TruffleRuby, so it is the source of truth when it comes to parsing Ruby code.

Could this be considered? What are the challenges?

[st0012]

I do consider this but not prioritizing it before Ruby 3.4. Its priority will depend on when IRB becomes a bundled gem too, which will make having prism gem as its dependency easier.

@tompng what do you think?

[eregon]

Its priority will depend on when IRB becomes a bundled gem too

According to https://github.com/ruby/ruby/blob/90ef28f59b4a2214d64f21d7e6d033273a0cce89/lib/bundled_gems.rb#L33 in 3.5.0.

[hsbt]

The boilerplate of Rails 8 contained irb. And the minimum support version of Rails 8 is Ruby 3.2.

If we replace ripper to prism, we need to add prism to dependency of irb for Ruby 3.2 and Rails 8. It means everyone compile prism gem for their Rails application.

I wonder if this is really beneficial to irb and Rails users.

[eregon]

With the transition to more bundled gems it seems Rails already has many native dependencies.
For instance:

$ ruby -v
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
$ gem i rails
...
Building native extensions. This could take a while...
Successfully installed racc-1.8.1
...
Successfully installed websocket-extensions-0.1.5
Building native extensions. This could take a while...
Successfully installed websocket-driver-0.7.6
Building native extensions. This could take a while...
...
Successfully installed rails-7.2.2
46 gems installed

Also:

$ rails new foo
$ cd foo
$ rm -rf vendor/bundle
$ bundle install | grep 'with native'
Installing stringio 3.1.1 with native extensions
Installing io-console 0.7.2 with native extensions
Installing bindex 0.8.1 with native extensions
Installing racc 1.8.1 with native extensions
Installing msgpack 1.7.3 with native extensions
Installing bigdecimal 3.1.8 with native extensions
Installing date 3.3.4 with native extensions
Installing nio4r 2.7.4 with native extensions
Installing json 2.7.5 with native extensions
Installing websocket-driver 0.7.6 with native extensions
Installing psych 5.1.2 with native extensions
Installing bootsnap 1.18.4 with native extensions
Installing debug 1.9.2 with native extensions
Installing puma 6.4.3 with native extensions

So there are already plenty of native gem dependencies for rails, and I think there is no choice with the transition to more default & bundled gems.

[eregon]

Ah that was rails 7.2.2, but anyway the situation is similar on rails-8.0.0.rc2:

$ gem i rails --pre
$ rails new --skip-bundle rails8
$ cd rails8
$ bundle install | grep 'with native'
Installing stringio 3.1.1 with native extensions
Installing io-console 0.7.2 with native extensions
Installing bcrypt_pbkdf 1.1.1 with native extensions
Installing bindex 0.8.1 with native extensions
Installing racc 1.8.1 with native extensions
Installing msgpack 1.7.3 with native extensions
Installing date 3.3.4 with native extensions
Installing nio4r 2.7.4 with native extensions
Installing bigdecimal 3.1.8 with native extensions
Installing ed25519 1.3.0 with native extensions
Installing json 2.7.5 with native extensions
Installing websocket-driver 0.7.6 with native extensions
Installing psych 5.1.2 with native extensions
Installing bootsnap 1.18.4 with native extensions
Installing puma 6.4.3 with native extensions
Installing debug 1.9.2 with native extensions
$ grep ' rails ' Gemfile.lock
    rails (8.0.0.rc2)
  rails (~> 8.0.0.rc2)

[eregon]

It means everyone compile prism gem for their Rails application.

Is it any different than other extensions like bigdecimal (which is really unavoidable for any web app realistically)?
I checked if prism took longer to install but prism and bigdecimal both take around 10 seconds to build for me locally:

gem i prism  16.41s user 1.04s system 161% cpu 10.794 total
gem i bigdecimal  8.95s user 1.31s system 103% cpu 9.946 total

[tompng]

I think it is a good idea to start implementing with Prism. But currently I'm not prioritizing it high. It already work in CRuby with ripper. Needs some effort to completely reimplement colorizing and nesting level calculating, and it will take time.

irb is visibly less reactive because of Ripper.

Maybe this is because IRB is calling Ripper::Lexer#scan three times per each key stroke.

[headius]

it has proven difficult to maintain on CRuby, JRuby and TruffleRuby

Ripper took some work to implement initially but has been no more difficult than the parser to maintain going forward. JRuby has fully-functional Ripper support without any native dependency currently (though this may change as we move toward Prism).

With the transition to more bundled gems it seems Rails already has many native dependencies

Only on implementations that require native libraries for their extensions. On JRuby, none of the standard libraries require building at install time and they can all be installed without a C compiler present on the host system (though some, like fiddle, still need make).

[eregon]

no more difficult than the parser to maintain going forward.

Which is a huge effort on every Ruby update (I would say by far the biggest task for any Ruby version update in alternative Ruby implementations, obviously depends on how many lexer/parser changes there were, thankfully Prism solves this now).
Also this means JRuby would have to keep maintaining that Java lexer & parser just for Ripper, that's a big overhead.
BTW there seems to have been plenty of inconsistencies.

On JRuby, none of the standard libraries require building at install time and they can all be installed without a C compiler present on the host system (though some, like fiddle, still need make).

Prism is a standard library since 3.3 (default gem to be exact) so I guess that's something to figure out for installing Prism on JRuby (e.g. via a -java gem using WASM/Chicory or by publishing precompiled binary gems for prism like some other gems using FFI).

[headius]

JRuby would have to keep maintaining that Java lexer & parser just for Ripper, that's a big overhead.

As long as CRuby ships with Ripper, we'll have to do this regardless. So it isn't going to matter much to us if Ripper or Prism are used by IRB, except that Prism will require an external native library (WASM could indeed help here).

Prism is a standard library since 3.3 (default gem to be exact) so I guess that's something to figure out

Once we start shipping with Prism (JRuby 10) we will include in the distribution a build of Prism for as many supported platforms as possible. Outside those platforms, an additional install will be required. That will either build a binary at install time, fetch a pre-built binary, or use WASM.

We will not ship without a parse.y-based parser until well after JRuby 10, and perhaps longer if we have to continue supporting Ripper.

To be fair, if everything moved to Prism and Ripper was no longer needed tomorrow, we'd be willing to drop it. But it has become quite established and it will take years for third-party libraryes to replace it. Chicken/egg... CRuby will keep shipping it because libraries use it, and libraries will keep using it because CRuby ships it.

Moving IRB to Prism from Ripper would be a positive move, but perhaps premature; Ruby 3.3 has been out for less than a year and all previous releases do not ship with Prism.

(I hit update too soon...)

Once we've had a few releases of CRuby that ship with Prism standard, I'd be more comfortable having key standard libraries like IRB depend on it.

[rubyFeedback]

I also concur with headius, coming from another point of view (indirectly that is). 3.3.6 was recently released (which also has issues; ruby-gtk3 does not work right now, and I use the latter as my primary test-GUI, for other GUIs e. g. ruby-libui, jruby-swing - I use the same underlying code base for all GUI-work by the way - but also for the same variant on the web, where I also use ruby consistentyl), and I had a few minor issues with ruby-dev before 3.3.6, so I think taking this slowly is better right now - let stability kick in first. Naturally I'd like to see for things becoming a bit more stable overall (I also had the rust jit fail to compile ... too many things aren't working that well right now from my point of view), so perhaps in 2025 this here could be reviewed. eregon is naturally preferring a switch, but any more balanced review should, in my opinion, also take in trade-offs users and developers may face.

[tompng]

Currently, these default/bundled gems uses Ripper.

irb power_assert rbs rdoc syntax_suggest

Maybe... removing ripper dependency from all of these gems can be discussed at bugs.ruby-lang.org?

In RDoc, I think there is a strong motivation to switch to Prism because of maintainability reason. RDoc's traditional parser using ripper still does not support endless method yet. RDoc already have experimental implementation using Prism.
In other gems, motivation to switch might not be high.

I'm currently trying to make an experimental implementation using Prism.

• Colorization: Working on this branch. Test all passes with tweaking some tests. Symbol colorization improves.
• Indent and nesting calculation: Trying and struggling now. I think it will take time to implement and to verify the quality.

[headius]

In defense of the idea in general... with prism becoming the default parser in 3.4, ripper is bound to start drifting away as prism enables syntax changes that are difficult to represent in parse.y. Eventually all libraries will need to move off of ripper, but it may be years before that's feasible.