Optimize indexing comment collection

[vinistock]

Motivation

Closes #2340

This PR makes the index fetch entry documentation only when requested, rather than eagerly indexing all comments. The idea is that not storing all documentation for all entries eagerly will reduce memory usage and speed up initial indexing.

In benchmarks, the impact was not as high as I had hoped. About 7.5% memory reduction and 5% indexing time reduction.

Implementation

The idea is that we add a flag to ignore comments on the initial indexing. When a file is modified, we turn on the flag so that we can capture the comments on a file currently opened in the UI.

When anything tries to read comments, the index fetches them lazily using Prism.parse_file_comments.

Automated Tests

Existing tests cover it.

[andyw8]

Overall this seems failry unobtrusive for a small but decent reduction in memory use.

[st0012]

I think it should be considered a breaking change when Entry#comments is changed to return a string instead of an array of strings. Is this change necessary for the performance improvement?

[vinistock]

I think it should be considered a breaking change when Entry#comments is changed to return a string instead of an array of strings. Is this change necessary for the performance improvement?

Yes, it reduces the number of objects allocated since we're not longer maintaining and array of multiple strings, but instead a single string.

Conceptually, I agree, it's a breaking change. But do we know if any addon is actually invoking comments directly on entries? The Rails addon doesn't do that.

[st0012]

Conceptually, I agree, it's a breaking change. But do we know if any addon is actually invoking comments directly on entries? The Rails addon doesn't do that.

I don't know any, and I don't plan to block the PR for this. But IMO it's worth adding the breaking change label here just in case.

Do we know how much of the memory reduction is contributed by lazy indexing, and how much is contributed by the type change?
If, for example, the lazy comment indexing just contributed to 1~2% of memory reduction by itself, then I'd prefer not maintaining 2 different comment collection logic.

[vinistock]

Do we know how much of the memory reduction is contributed by lazy indexing, and how much is contributed by the type change?
If, for example, the lazy comment indexing just contributed to 1~2% of memory reduction by itself, then I'd prefer not maintaining 2 different comment collection logic.

I just benchmarked this to compare. The lazy logic is responsible for 6.8% out of the 7.5% reduction. The reduction related to turning the comments from an array into strings is the smaller part.

[st0012]

@vinistock Thanks for benchmarking the difference. IMO we can update this PR's title to be more generic, like "Optimize comment indexing", as the comment type change is also not trivial.