Using with_prototype in a syntax definition can cause ST to use infinite memory

[keith-hall]

Summary

Using with_prototype in a syntax definition can (100% reproduceable) cause ST to immediately start to constantly consume more and more memory, making the OS become unstable a minute afterwards.

Expected behavior

ST wouldn't consume infinite memory, but show a recursion error.

Actual behavior

ST doesn't complain, just locks up while consuming more and more memory.

Steps to reproduce

Create a file called infinite_memory.sublime-syntax in the Packages/User folder with the following contents:

%YAML 1.2

---
# See http://www.sublimetext.com/docs/3/syntax.html
scope: source.example-recursion-out-of-memory-soon
contexts:
  main:
    - match: 'test'
      set: after_test
      with_prototype:
       - match: 'example'
  after_test:
    - match: 'foobar'
      set: main

Then check how much RAM ST is using, and try to interact with ST's UI. Everything is fine until you create a new tab and set the syntax to infinite_memory - check how much RAM ST is using now, and how it is rapidly increasing, and try to interact with ST's UI. (i.e. memory consumption only occurs when a view is open with the dodgy syntax assigned to it.)

Commenting out the set: main line (or with_prototype: line and the - match immediately below it) will enable ST to operate normally. (But you'll have to do it outside ST or destroy your sublime-session or this syntax file first, to get ST to launch again.)

Environment

• Operating system and version:
  • Windows 7 x64
  • Mac OS (untested)
  • Linux - Ubuntu 16.04 x64
• Sublime Text:
  • Build 3114, 3119 (but has been there for ages)

[FichteFoll]

There actually is a recursion limit for this in place (or so I've been told), but ST requires so much resources to hit it that the OS becomes unstable and everything starts to slow down before you ever reach that point.

[wbond]

I believe this is happening because new anonymous contexts are being created via with_prototype (it duplicates a context and adds the rules from with_prototype), and you have an include loop.

There is a sanity limit of 25,000 contexts (I've hit it plenty of times), but I haven't looked into why the two of you have hit such memory issues. Most likely we'll need to add a second heuristic to detect a with_prototype combined with recursion.

[viluon]

This is still a thing on 3176. If what @wbond is suggesting is true, then a simple fix would be to deduplicate rules when switching contexts with with_prototype.

[wbond]

then a simple fix would be to deduplicate rules when switching contexts with with_prototype

Unfortunately these issue tend to happen through include loops, and thus aren't simple to detect.

Just an FYI, stable build 3200 is out now.

[viluon]

Just upgraded, this severe memory leak is still present in 3200.

@wbond I don't think these issues happen through include loops at all, include loops are detected and warned about. Worse, I don't think this happens while ST is parsing a file to highlight it. I just tried it with an empty file (nothing but three newlines). The syntax definition couldn't even enter the context with with_prototype, which would require matching against an identifier first. Nevertheless, the editor started leaking the moment I saved the .sublime-syntax with the problematic directive uncommented.

I can share the syntax definition I'm writing if doing so would help fix the issue.

[wbond]

We compile a syntax definition, which involved inlining all contexts from all included syntaxes into one structure. Once that structure includes 25,000 contexts, we bail, since it indicates that most likely the author has included a syntax that recursively includes the original syntax, or one of its contexts. If we can't compile a syntax to have less than 25,000 contexts, we don't compile it at all.

This bug is about creating a syntax that doesn't hit the 25,000 context limit, but uses up all of the user's memory. Since it was reported we've made a number of enhancements to the syntax engine such that regex strings are de-duped. I'm not sure if this exact issue would still happen. It sound like your issue is different. I would recommend creating a new issue with as many details as possible, and a link to the syntax will almost certainly be necessary to reproduce.

[viluon]

So the memory leak happens during the compilation process?

[wbond]

Build 4075 includes checks to prevent this from happening.