Recover from files with initial syntax errors

[vinistock]

Motivation

We are getting some errors on user's machines of the document instance being nil. The only possibility for this, is if we weren't able to parse the first version of the file due to syntax errors and never recovered.

Implementation

This PR attempts to make this process more robust by always inserting the document into the store regardless of syntax errors. If a syntax error occurs, the tree will be nil, which we have to handle in requests.

When edits are made to the file with syntax errors, push_edits will take care of re-parsing.

Automated Tests

Added a couple of examples and changed some existing ones.

Manual Tests

If you wanna try this manually

1. Start the LSP on this branch
2. Created a new file and save it with a syntax error (e.g.: def Foo)
3. Close the file to clear its state from the store
4. Re-open the file (this re-open will fail to parse)
5. Fix the syntax error by adding end
6. Verify that the LSP recovered and features are now available

[Morriar]

I wonder if we should let the exception bubble up to to handle and catch it there rather than making all those types nilable?

[Morriar]

Should we log something in the console in this case?

[vinistock]

I think it's valuable, but I'll suggest doing this in a follow up PR. If we're going to log something, we should include the URI, which is currently not passed to Document. I can create a follow up as soon as this one goes in.

[vinistock]

Would we be able to recover from the errors if we let the handler take care of it? The complication is that we only receive the full document source once, when opening the file and then only receive incremental text changes to it.

So we need to save the initial document, regardless of whether there are errors or not.

[Morriar]

So we need to save the initial document, regardless of whether there are errors or not.

Ah I see. Ok then 👍

[st0012]

I think this may not be the best solution for the problem. Currently, successful document parsing and document processing logic (including caching) go hand-in-hand. And that's why it's safe for the latter because it saves us the need to handle nil in a lot of places.

If we want to make parsing part failable, we shouldn't compromise the safety we have on document processing. Instead, we should probably consider handling requests in stages, like:

1. Document parsing + storing
2. Parsing result check + early return
3. Document processing

It'll likely require structural change on the Handler class but I think it'd be better.

Another concern is that VSCode will hang if it doesn't receive a request's response (see #149). So we may need to pay more attention on the early return's implementation and make sure it still returns something?

Update: I gave the crossed part a try and it doesn't work because requests like didOpen do need to handle the text -> document conversion. So if we separate a request into stages, it'll just make those requests hard to understand. Also, a lot of nilable change will disappear because #221 has been merged so the code will look less confusing.

[vinistock]

@st0012 the void vs nil returns has been addressed in #100. Returning nil for all of our current requests are valid responses and VS Code handles them properly.

The reason it was hanging in #149 is because RuboCop would print argument errors to STDOUT. Printing anything to STDOUT from inside the LSP will always put it in an endless loop.