Possibility to add a custom lexer

[areiterer]

Hello,

I'm using a software called SoftCover that calls pygments.rb.
Unfortunately, pygments does not support JSX (ReactJS) highlighting, but there is a custom lexer called "pygments-lexer-babylon" that can be installed via pip.

I'm not a Ruby developer, but I could find out that pygments.rb comes with it's own version of pygments and therefore it can't find the custom lexer.

Also I could see that there is a custom_lexers directory, but I don't know yet how to get pygments.rb to use the custom lexer.

Is there a way to add the package to pygments.rb?
I could try to do it on my own with a little help if there is a possibility for that.

Thanks very much in advance,
Andreas

[maestromac]

Yeah, this would be great.

[grafikrobot]

I just had to figure this out.. It's actually already possible with some additional work. In my case it was a bit more complicated as I have a language that doesn't have a convenient existing lexer. So here's what I did:

1. Created a Pygments lexer locally for my language.
2. Created a PIP package from it.. jam_pygments
3. Installed with pip by pointing to my local git clone of that.
4. Created/registered the plugin lexer in ruby code as Pygments::Lexer.create name: 'Jam', aliases: ['jam','bjam','b2'], filenames: ['*.jam','Jamfile','Jamroot'], mimetypes: ['text/x-jam'] so that the find function will return it as needed.
5. Now follow regular use.

For you use case you only need to do steps 3-5 obviously.

HTH

[slonopotamus]

Okay, #46 is about custom styles while this issue is about custom lexers.

[slonopotamus]

What we can do here is to teach pygments.rb to use preinstalled Pygments (if it is present) instead of the bundled one. This way if user installed Pygments plugins, they will work.

[rbrich]

Easier workaround:

--- popen.rb.orig	2021-02-17 16:13:18.000000000 +0100
+++ popen.rb	2021-02-17 16:14:02.000000000 +0100
@@ -141,11 +141,7 @@
     #
     # Returns an array of lexers.
     def lexers
-      lexer_file = File.expand_path('../../lexers', __dir__)
-      raw = File.open(lexer_file, 'rb').read
-      Marshal.load(raw)
-    rescue Errno::ENOENT
-      raise MentosError, 'Error loading lexer file. Was it created and vendored?'
+      lexers!
     end

     # Public: Get back all available lexers from mentos itself

This makes pygments.rb find lexer plugin installed in this way: https://stackoverflow.com/a/39540683/6013024

The way I found this:

1. I've checked which python interpreter is called -> it's normal python3
2. I've checked the lexer plugin is installed for this interpreter -> yes, from pygments import lexers; lexers.get_lexer_by_name('my') works
3. I've added hook to get_lexer_by_name() in gems/pygments.rb-2.0.0/vendor/ -> it's not called for "my" language
4. I've investigated how pygments.rb gets the list of lexers -> it reads files in the vendor version of pygments, ignoring plugins!

Disabling this optimization (listing files instead of asking pygments API) makes normal plugins work.

[slonopotamus]

@rbrich do I understand correctly that you install plugins into Pygments that is bundled inside pygments.rb?

[rbrich]

Sort of, but not really. The plugins are standalone Python packages, they don't need to depend on Pygments.

You can have this combination and it works fine:

• python3, without pygments
• a pygments plugin installed in python3's site-packages
• pygments.rb using python3 and its own installation of pygments

The only condition is that python3 is the same interpreter, in all cases. I'm using both Python and Ruby from Homebrew. Pygments.rb was installed with gem, the custom lexer was installed with python3 setup.py install. The lexer is using plugin structure as described in the SO answer.

[slonopotamus]

Oh, I see. So, bundled Pygments picks up system-wide installed plugins.