Slow to edit large markdown file

[kenng]

What is the problem?

Slow to edit large markdown file when the plugin is active. If optimisation is difficult, probably offer an option to toggle on/off it for large file. As the plugin is really useful to generate the table of content.

How can I reproduce it?

You can use this file to test
https://gist.github.com/kenng/b04e246ee269ebd2e38b102047d1b1ec

Is there any error message in the console?

No error.

[yzhang-gh]

I can confirm

[muuvmuuv]

Something like that would help to just render the viewed content: https://twitter.com/ChromiumDev/status/1062433100804702208

[josephzjw]

me too, after disable this extension it is faster...

[michaelbarkdoll]

I'm also experiencing this issue.

[yzhang-gh]

Will investigate it when I come back from the holiday

[upupming]

I think there are mainly two problems:

1. Cannot hit Enter, Backspace, Tab, etc. when opening a markdown file for the first time. This is because of vscode's lazy load mechanism and the extension blocks our keyboard input when loading. Such as Cannot use backspace or enter after first opening markdown file #335, Unable to Backspace, Enter, Tab #354.
2. Slow to edit large markdown files, Such as this issue.

@neilsustc If I'm not mistaken, I will do some research about these two problems and try to fix them~

[yzhang-gh]

That will be great!

And I would suggest uninstalling this extension before debugging this issue (the problem 2 you mentioned). The extension overwriting might cause some confusion. (see here)

Another related issue #181.

[upupming]

For problem 2, I have tested many times, here is the conclusion:

1. The code you wrote is fast, just takes at most 30ms to finish.
2. When our callback function such as onEnterKey has an edit on the markdown file using commands.executeCommand('type'... etc, the code runs fast too.
3. However, when we call the handler onEnterKey twice continuously, the second call has to wait for about 5~6s before the first call finishes, don't know why. When the second call gets stuck, I can see VS Code is using almost 1.5GB memory.
4. If I don't edit the file, i.e. comment all code like commands.executeCommand('type'..., two continuous calls of the handler are running fast again.

So there must be something wrong on commands.executeCommand('type'(by the way, I cannot see find any documentation about this type command).

I have also tried using TextEditorEdit.insert to insert contents, but continuous calls of onEnterKey will get the following warning:

Edits from command markdown.extension.onEnterKey were not applied.
extensionHostProcess.js:715
Edits from command markdown.extension.onEnterKey were not applied.
extensionHostProcess.js:715
Edits from command markdown.extension.onEnterKey were not applied.
extensionHostProcess.js:715
Edits from command markdown.extension.onEnterKey were not applied.
extensionHostProcess.js:715
Edits from command markdown.extension.onEnterKey were not applied.

I'm trying to find another vs code extension which handles this problem gracefully.

This issue is just what we are facing: microsoft/vscode#65876

[upupming]

Probably this is a problem with vs code's extension API, I'm not quite sure now.

Update: It's an issue of our self, see comment below.

[upupming]

Okay, I found the problem is not on the side of vs code itself.

After some profile following the performance guide, I find the most cost function is editDistance, which takes 84.34% of the total time.

If I comment this line: https://github.com/neilsustc/vscode-markdown/blob/08da4d53aac689531f3fc3dc3224fb3f5aeb2e2d/src/extension.ts#L34
the editing of large markdown files will be very fast!

Some analysis of the editDistance: https://github.com/neilsustc/vscode-markdown/blob/08da4d53aac689531f3fc3dc3224fb3f5aeb2e2d/src/toc.ts#L156-L182

It takes O(n^2) time, which is too slow, so we should use dynamic programming to make it faster. Need further investigating!

[yzhang-gh]

Edit: This comment is outdated.

(You found it!)

---

😅 Thanks anyway.

I tried to disable some of the features (e.g. TOC) to find out a smallest extension that still has this issue. That was the microsoft/vscode#50254.

Now I am curious about whether this issue is still there if we comment out all other features except for
listEditing.activate(context); here

https://github.com/neilsustc/vscode-markdown/blob/08da4d53aac689531f3fc3dc3224fb3f5aeb2e2d/src/extension.ts#L29-L51

If so, I think we can continue on this train of thought to find the smallest extension sample. (The one in microsoft/vscode#50254 is small but is not issue-reproducible...)

[yzhang-gh]

Some analysis of the editDistance:
It takes O(n^2) time, which is too slow, so we should use dynamic programming to make it faster. Need further investigating!

Or a better heuristics for TOC detection.

[upupming]

Another question, you generated a new TOC in detectTocRange to compare with all lists, but don't update it, it may be time-consuming. Although we should not update when there is no TOC found, we can update the existed TOC on the fly.

---

Can we just simplify the TOC detection by adding something like this:

<!-- begin table of content(Markdown All in One) -->
...
<!-- end table of content(Markdown All in One) -->

This is exactly what vscode-markdown-toc does.

However, this may not work on the old files without these comments. We should tell our user to delete the old TOC and regenerate it. What do you think?

[upupming]

The calculation of Levenshtein_distance cannot be faster than O(n^2 - ε) for any ε greater than zero. We must use another approach for TOC detection.

[upupming]

Currently, one can generate many TOCs, however, only the first TOC canbe detected:

Need a way to handle this, too.

[quicktrick]

I've encountered this issue too. Too slow work with large markdown files (5000 lines and more). I have to disable this extension to work normally. It's very inconvenient, because I use this extension intensively. Is it possible to disable some features in settings that slow down editing? I don't use toc, I've disabled toc.updateOnSave, but it does not help.

[yzhang-gh]

Could you profile this extension as described here? Then we will know the reason.

[quicktrick]

I don't know if I made everything correctly, but here it is.

CPU-20190628T051951.575Z.cpuprofile.txt

[yzhang-gh]

Tracked in a separate issue.

---

For the people who have also experienced the performance issue, feel free to open a new one (with the CPU profile).