docs: recommended way to test a node plugin on GitHub CI?

[saidelike]

TLDR

There seems to be lots of testing frameworks for vim scripts:

• https://github.com/cdelledonne/vim-utest
• https://github.com/dhruvasagar/vim-testify
• https://github.com/LucHermitte/vim-UT
• https://github.com/junegunn/vader.vim
• https://github.com/google/vroom
• https://github.com/kana/vim-vspec
• https://github.com/thinca/vim-themis
• ...

Should I use one of them and just call my node exported test function from vim script? If so, how do I get the test results output?

Should I use a technique similar to what node-client is doing instead, see https://github.com/neovim/node-client/tree/master/packages/integration-tests ? If so, how do I get the test results output?

Any pointers or help is appreciated.

This is how I currently test my neovim node plugin locally

I've got a neovim node plugin that exposes a test framework entry point. This function ends up calling into mocha in order to run all the tests.

I am able to run the tests nicely from a local environment after neovim starts by calling the test framework entry point from the neovim config. And I can read the output from the log file that is written thanks to the NVIM_NODE_LOG_FILE environment variable.

2024-05-14 21:16:47 INF     ✔ [[1,3],4,2]
2024-05-14 21:16:47 INF     ✔ [[0,2],1]
2024-05-14 21:16:47 INF     ✔ [[0,2],1,0]
2024-05-14 21:16:47 INF
2024-05-14 21:16:47 INF
2024-05-14 21:16:47 INF   4130 passing (19s)
2024-05-14 21:16:47 INF   2881 pending
2024-05-14 21:16:47 INF   1 failing
2024-05-14 21:16:47 INF
2024-05-14 21:16:47 INF   1) recorded test cases
       recorded/actions/copySecondToken:

      Unexpected final state
      + expected - actual

       {
      -  "clipboard": ""
      +  "clipboard": "value"
         "documentContents": "\nconst value = \"Hello world\";\n"
         "selections": [
           {
             "active": {

      at runTest (packages\test-harness\dist\cursorless-neovim-e2e\src\suite\recorded.neovim.test.cjs:28661:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[justinmk]

We should document this (probably in :help remote-plugin too), I'll keep this open to track that.

Should I use one of them and just call my node exported test function from vim script? If so, how do I get the test results output?

If getting test results in a structured format like TAP or XML is important, I would use busted.

Otherwise I wouldn't use a test framework. I would just write tests in a Lua script and use assert().

• The script can use os.exit() or :cquit to exit with an error code.
• If you run the script with nvim -l it can accept CLI arguments, see :help lua-args.
• You will need to pass the location of your node-client module so that your Lua test script can start it (jobstart()).

That's assuming you are testing a plugin (Lua code that invokes RPC methods on your node-client module), so testing end-to-end is a good idea. But f your Lua plugin code is minimal, you could skip that part and write tests directly against your node module like any other node project.

Should I use a technique similar to what node-client is doing instead, see https://github.com/neovim/node-client/tree/master/packages/integration-tests ? If so, how do I get the test results output?

The only purpose of that sub-package is to run a separate process to fully isolate the test runner from the node-client module being tested. But if you use the end-to-end approach described above, driven by Lua, then you don't need the packages/integration-tests structure (which is for a node-based test runner).

I've got a neovim node plugin that exposes a test framework entry point. This function ends up calling into mocha in order to run all the tests.

LGTM. Though NvimPlugin will be deprecated pretty soon. Instead you will just define handlers that are called by your Lua (or Vimscript) plugin.

[saidelike]

@justinmk Thanks for all your insights, definitely helpful.

I have a single exported node function TestHarnessRun() being called from lua to do all the tests (since mocha is invoked by node to run all the tests directly from node). I tried to make this node function TestHarnessRun() synchronous but it results in a dead lock where each test can't run properly. I think it is due to lua being single threaded and so an async node function can't be executed if executed from a sync lua function. So I am back to using TestHarnessRun() async and I can't really use a lua/vim test suite if I can't retrieve its return value. (Maybe another issue should be created to discuss this behaviour but we basically can't call an "async node function" from a "sync lua function" without a deadlock)

Am I understanding it correctly that writing tests in lua with assert() makes sense if most of the plugin code is in lua and it only calls into the node code for certain features? In my case, most of the plugin code is in node, so using assert() in lua means I would basically only have ONE assert(). And as discussed above, another problem is I don't see how this is doable with node calling into async functions.

[pokey]

I think the issue with running the tests synchronously is that during the execution of the node test, we exercise code that calls from node back to lua, eg to modify editors / run neovim commands. I'm guessing that the lua execution context is single-threaded, so if it is hanging waiting for the tests to finish, it's not able to respond to the requests from the node process, so node just hangs, and we effectively have a deadlock. Does that make sense?

[justinmk]

I'm guessing that the lua execution context is single-threaded, so if it is hanging waiting for the tests to finish, it's not able to respond to the requests from the node process, so node just hangs, and we effectively have a deadlock. Does that make sense?

Yes. If if it wasn't "single threaded", what would you expect the result to be? Doing roundtrips in a single request, between processes that are inspecting state of each process, can't have any reasonable behavior, in mutable systems.