vim very slow with semantic tokens enabled (with a possible fix)

[cridemichel]

Result from CocInfo

## versions

vim version: VIM - Vi IMproved 9.1 9010050
node version: v21.6.1
coc.nvim version: 0.0.82-d1568d56 2023-09-29 19:43:34 +0800
coc.nvim directory: /Users/demichel/.vim/plugged/coc.nvim
term: iTerm.app
platform: darwin

## Log of coc.nvim

2024-02-02T16:25:43.166 INFO (pid:63048) [plugin] - coc.nvim initialized with node: v21.6.1 after 245
2024-02-02T16:25:43.868 INFO (pid:63048) [attach] - receive notification: showInfo []

VIM VERSION

:version
:version
VIM - Vi IMproved 9.1 (2024 Jan 02, compiled Jan 23 2024 22:19:02)
macOS version - x86_64
Included patches: 1-50
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl               +comments          +find_in_path      +lispindent        +multi_byte        +rightleft         +terminal          +wildignore
+arabic            +conceal           +float             +listcmds          +multi_lang        +ruby              +terminfo          +wildmenu
+autocmd           +cryptv            +folding           +localmap          -mzscheme          +scrollbind        +termresponse      +windows
+autochdir         +cscope            -footer            +lua               +netbeans_intg     +signs             +textobjects       +writebackup
-autoservername    +cursorbind        +fork()            +menu              +num64             +smartindent       +textprop          -X11
-balloon_eval      +cursorshape       +gettext           +mksession         +packages          +sodium            +timers            -xattr
+balloon_eval_term +dialog_con        -hangul_input      +modify_fname      +path_extra        +sound             +title             -xfontset
-browse            +diff              +iconv             +mouse             +perl              +spell             -toolbar           -xim
++builtin_terms    +digraphs          +insert_expand     -mouseshape        +persistent_undo   +startuptime       +user_commands     -xpm
+byte_offset       -dnd               +ipv6              +mouse_dec         +popupwin          +statusline        +vartabs           -xsmp
+channel           -ebcdic            +job               -mouse_gpm         +postscript        -sun_workshop      +vertsplit         -xterm_clipboard
+cindent           +emacs_tags        +jumplist          -mouse_jsbterm     +printer           +syntax            +vim9script        -xterm_save
-clientserver      +eval              +keymap            +mouse_netterm     +profile           +tag_binary        +viminfo
+clipboard         +ex_extra          +lambda            +mouse_sgr         -python            -tag_old_static    +virtualedit
+cmdline_compl     +extra_search      +langmap           -mouse_sysmouse    +python3           -tag_any_white     +visual
+cmdline_hist      -farsi             +libcall           +mouse_urxvt       +quickfix          -tcl               +visualextra
+cmdline_info      +file_in_path      +linebreak         +mouse_xterm       +reltime           +termguicolors     +vreplace
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X -DMACOS_X_DARWIN -g -O2 -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang -L/usr/local/lib -o vim -lm -lncurses -lsodium -liconv -lintl -framework AppKit -L/usr/local/opt/lua/lib -llua5.4 -mmacosx-version-min=14.2
 -fstack-protector-strong -L/usr/local/lib -L/usr/local/opt/perl/lib/perl5/5.38/darwin-thread-multi-2level/CORE -lperl -L/usr/local/opt/[email protected]/Framew
orks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin -lpython3.12 -framework CoreFoundation -lruby.3.3 -L/usr/local/Cellar/ruby/3.3.0/lib

coc json file

{
  "diagnostic.refreshOnInsertMode": true,
  "diagnostic.checkCurrentLine": true,
  "suggest.enablePreview": true,
  "suggest.triggerAfterInsertEnter": true,
  "diagnostic.messageTarget": "float",
  "diagnostic.enable": false,
  "inlayHint.enable" : true,
  "suggest.noselect": true,
  "suggest.minTriggerInputLength": 2,
  "inlayHint.enableParameter": false,       
  "coc.source.vimtex.priority": 99,
  "python.pythonPath": "/usr/local/bin/python3",
  "markdown-preview-enhanced.enableScriptExecution": true,
  "coc.preferences.diagnostic.displayByAle": true,
  "list.source.files.args": ["--hidden", "--files"],
  "yank.highlight.enable": false, 
  "pyright.inlayHints.parameterTypes": false,
  "pyright.inlayHints.variableTypes": false,
  "pyright.disableDocumentation": false,
  "pyright.enable": true,
  "semanticTokens.enable": true,
  "colors.enable": false,
  "links.enable": false,
  "links.highlight": false,
  "links.tooltip": false, 
  "suggest.floatConfig": {"border": false, "rounded": false}   
}

Describe the bug

If semantic tokens are enabled and a large file is opened,
vim is very slow, not to say unusuable, until adding highlights is over.
For example just after having opened a large file vim "stutters" a lot on scrolling
by pressing <PgUp> or <PgDown>.
In the following I propose also a possible way to fix this issue. If you find it more convenient,
I could create a pull request.

Reproduce the bug

1. download the the zip file "files.zip", which you can find below
2. unzip it with

> unzip files.zip

3. open the file "montecarlo.c" which is in the folder "ellipsoid" by using
   the command:

  > vim -u minimal_vimrc ellipsoid/montecarlo.c

4. just after the file is opened start scrolling by pressing <PgDown>

5. vim "stutters" and it is generally slow (even if you try to move around with left/right arrows)
   until adding highlights is over.

POSSIBLE FIX

After some debugging I found that the cause of this slowdown is due to
the for loop in function "s:add_highlights_timer", i.e

  for i in range(0, len(a:highlights) - 1)
      if i < g:coc_highlight_maximum_count
        call add(hls, a:highlights[i])
      else
        call add(next, a:highlights[i])
      endif
    endfor

hence I decided to resort to vim9 scripting, which is supposed to be much faster.
What I suggest is to replace the function "s:add_highlights_timer", i.e.

function! s:add_highlights_timer(bufnr, ns, highlights, priority) abort
  let hls = []
  let next = []
  if has('vim9script')
     call CreateHlLists(hls, next, a:highlights, g:coc_highlight_maximum_count)
  else
    for i in range(0, len(a:highlights) - 1)
      if i < g:coc_highlight_maximum_count
        call add(hls, a:highlights[i])
      else
        call add(next, a:highlights[i])
      endif
    endfor
  endif
  if bufwinnr(a:bufnr)!=-1 " check if buffer exists 
     call s:add_highlights(a:bufnr, a:ns, hls, a:priority)
  endif
  if len(next) && bufwinnr(a:bufnr)!=-1
    call timer_start(g:coc_highlight_timer, {->s:add_highlights_timer(a:bufnr, a:ns, next, a:priority)})
  endif
endfunction

with

if has('vim9script')
def CreateHlLists(hls: list<any>, next: list<any>, highlights: list<any>, maxc: number)
   for i in range(0, len(highlights) - 1)
      if i < maxc
        add(hls, highlights[i])
      else
        add(next, highlights[i])
      endif
   endfor
   return
enddef
endif

function! s:add_highlights_timer(bufnr, ns, highlights, priority) abort
  let hls = []
  let next = []
  if has('vim9script')
     call CreateHlLists(hls, next, a:highlights, g:coc_highlight_maximum_count)
  else
    for i in range(0, len(a:highlights) - 1)
      if i < g:coc_highlight_maximum_count
        call add(hls, a:highlights[i])
      else
        call add(next, a:highlights[i])
      endif
    endfor
  endif
  if bufwinnr(a:bufnr)!=-1 " check if buffer exists 
     call s:add_highlights(a:bufnr, a:ns, hls, a:priority)
  endif
  if len(next) && bufwinnr(a:bufnr)!=-1
    call timer_start(g:coc_highlight_timer, {->s:add_highlights_timer(a:bufnr, a:ns, next, a:priority)})
  endif
endfunction

Note that the patch suggested in #4872 has been already applied and a new function, called CreateHlLists(), has been added. This function is written in vim9 scripting language and it turned out to be insanely faster than the legacy solution.

file used to reproduce the bug

files.zip

Minimal vimrc file

set nocompatible
set runtimepath^=/Users/demichel/.vim/plugged/coc.nvim/
" Use <c-space> to trigger completion
filetype plugin indent on
let mapleader = ' '
let maplocalleader=' '
syntax on
set nohidden

Screenshots (optional)

before the fix:

stuttering.mp4

after the fix:

nostuttering.mp4

[cridemichel]

ALTERNATIVE FIX

if one does not want to resort to vim9 scripting, list slicing can used and the function "s:add_highlights_timer"
can be modified as follows:

function! s:add_highlights_timer(bufnr, ns, highlights, priority) abort
  let lhl = len(a:highlights)
  let maxc = g:coc_highlight_maximum_count
  if maxc < lhl
    let hls = a:highlights[:maxc-1]
    let next = a:highlights[maxc:]
  else
    let hls = a:highlights[:]
    let next = []
  endif
  if bufwinnr(a:bufnr)!=-1 " check if buffer exists 
    call s:add_highlights(a:bufnr, a:ns, hls, a:priority)
  endif
  if len(next) && bufwinnr(a:bufnr)!=-1
    call timer_start(g:coc_highlight_timer, {->s:add_highlights_timer(a:bufnr, a:ns, next, a:priority)})
  endif
endfunction

this solution is as efficient as the previous one and does not require vim9 scripting

[fannheyward]

PR is welcome!

[cridemichel]

thanks and please give a look at second fix in #4872 related to error E964, are you able to reproduce it?

[cridemichel]

Hi, I have just created a pull request to fix #4879 and #4872 (error E964)