Test Output in JSON, making it difficult to read

[Vjchi]

Good afternoon all,

I’m currently transitioning from VS Code to NeoVim, and decided to get started with LazyVim (a pre-configuration using the package manager Lazy.nvim. Apologies in advance if I have missed something obvious;

Neotest seems to complete the testing as expected (closest test, test file, test all files, etc…), I am experiencing two issues:

• the output (both in the floating window and the output panel) is formatted in JSON,
• althought the test passes, the test is shown as a fail (red cross)

See example output below:

For the setup, I followed the recommendations from the lazyVim website, and simply added the plugins in the relevant folder:

neotest.lua

[the file preconfigured by LazyVim]

return {
  {
    "folke/which-key.nvim",
    optional = true,
    opts = {
      defaults = {
        ["<leader>t"] = { name = "+test" },
      },
    },
  },
  {
    "nvim-neotest/neotest",
    opts = {
      -- Can be a list of adapters like what neotest expects,
      -- or a list of adapter names,
      -- or a table of adapter names, mapped to adapter configs.
      -- The adapter will then be automatically loaded with the config.
      adapters = {},
      -- Example for loading neotest-go with a custom config
      -- adapters = {
      --   ["neotest-go"] = {
      --     args = { "-tags=integration" },
      --   },
      -- },
      status = { virtual_text = true },
      output = { open_on_run = true },
      quickfix = {
        open = function()
          if require("lazyvim.util").has("trouble.nvim") then
            vim.cmd("Trouble quickfix")
          else
            vim.cmd("copen")
          end
        end,
      },
    },
    config = function(_, opts)
      local neotest_ns = vim.api.nvim_create_namespace("neotest")
      vim.diagnostic.config({
        virtual_text = {
          format = function(diagnostic)
            -- Replace newline and tab characters with space for more compact diagnostics
            local message = diagnostic.message:gsub("\n", " "):gsub("\t", " "):gsub("%s+", " "):gsub("^%s+", "")
            return message
          end,
        },
      }, neotest_ns)
      if opts.adapters then
        local adapters = {}
        for name, config in pairs(opts.adapters or {}) do
          if type(name) == "number" then
            if type(config) == "string" then
              config = require(config)
            end
            adapters[#adapters + 1] = config
          elseif config ~= false then
            local adapter = require(name)
            if type(config) == "table" and not vim.tbl_isempty(config) then
              local meta = getmetatable(adapter)
              if adapter.setup then
                adapter.setup(config)
              elseif meta and meta.__call then
                adapter(config)
              else
                error("Adapter " .. name .. " does not support setup")
              end
            end
            adapters[#adapters + 1] = adapter
          end
        end
        opts.adapters = adapters
      end
      require("neotest").setup(opts)
    end,
    -- stylua: ignore
    keys = {
      { "<leader>tt", function() require("neotest").run.run(vim.fn.expand("%")) end, desc = "Run File" },
      { "<leader>tT", function() require("neotest").run.run(vim.loop.cwd()) end, desc = "Run All Test Files" },
      { "<leader>tr", function() require("neotest").run.run() end, desc = "Run Nearest" },
      { "<leader>ts", function() require("neotest").summary.toggle() end, desc = "Toggle Summary" },
      { "<leader>to", function() require("neotest").output.open({ enter = true, auto_close = true }) end, desc = "Show Output" },
      { "<leader>tO", function() require("neotest").output_panel.toggle() end, desc = "Toggle Output Panel" },
      { "<leader>tS", function() require("neotest").run.stop() end, desc = "Stop" },
    },
  },
  {
    "mfussenegger/nvim-dap",
    optional = true,
    -- stylua: ignore
    keys = {
      { "<leader>td", function() require("neotest").run.run({strategy = "dap"}) end, desc = "Debug Nearest" },
    },
  },
}

neotest.lua

• Local file that I use to declare the neotest-go adapter:

return {
  "nvim-neotest/neotest",
  optional = true,
  dependencies = {
    "nvim-lua/plenary.nvim",
    "nvim-treesitter/nvim-treesitter",
    "nvim-neotest/neotest-go",
  },
  opts = {
    adapters = {
      ["neotest-go"] = {
        -- Here we can set options for neotest-go, e.g.
        -- args = { "-tags=integration" }
      },
    },
  },
}

neotest-go.lua

[the file I use to declare and load neotest-go]

return {
  "nvim-neotest/neotest-go",
}

I don’t know what I’m missing at this point, I’ve restarted the installation of LazyVim from scratch twice, and the only thing I can think of is the adapter not interpreting or parsing the output of the test, for whatever reason,

Any help would be greatly appreciated :)

[pehowell]

the only thing I can think of is the adapter not interpreting or parsing the output of the test, for whatever reason

I had a similar issue, and it was due to clang compiler warnings being printed due to cgo. Once I disabled the warnings, the test output was correctly parsed.

[fredrikaverpil]

Neotest-go does not properly parse the test output when using Go provided via pkgx:

If e.g. using Go provided through Homebrew, the test output works fine and this specific test in the screenshot passes.

[Syakhisk]

I had a similar issue, and it was due to clang compiler warnings being printed due to cgo. Once I disabled the warnings, the test output was correctly parsed.

@pehowell Can you help to explain how to do this?

[fredrikaverpil]

EDIT: hm, strange ... I no longer see these JSON lines mixed in with the non-JSON lines in the test output window.

Original post

I am working in a project right now where I get both nicely printed output along with a huge amount of JSON printed in the test output panel.

When I run "nearest test" and debug print some values (see git diff at the bottom of post), I can see that neotest-go writes 4 files to tmp:

❯ ll /var/folders/85/kqpgt4g50kg91tm0j34bmtc80000gp/T/LazyVim.fredrik/kBSdIB/ 
Permissions Size User    Group Date Modified Name
.rw-r--r--  282k fredrik staff  6 Feb 08:18  1
.rw-r--r--   41k fredrik staff  6 Feb 08:18  2
.rw-r--r--   833 fredrik staff  6 Feb 08:18  3
.rw-r--r--   233 fredrik staff  6 Feb 08:18  4

• File 1 contains all the newline-delimited JSON. This is what is undesirable to have in the test output as it makes the test output super difficult to read. This file contains thousands of lines and shows how all tests in my test suite ran.
• File 2 contains all parsed JSON output, which is likely what you would've wanted to actually see in the test output, as it is a lot more readable. This contains logs from e.g. testcontainers about spinning up and killing containers. This file contains around 600 lines and shows how all tests in my test suite ran.
• File 3 contains very limited parsed JSON output, but slightly modified output, prepending lines with === RUN, === PAUSE, === CONT. Maybe some internal format used by neotest or neotest-go. Maybe this is meant to be shown in a small text box with limited space. This file contains 13 lines of lines of text.
• File 4 contains a subset of file 3. Likely also some internal format used by neotest or neotest-go. This file contains 4 lines of text, somewhat similar to what shows at the very end of file 3 but prepending the lines with the RUN/PAUSE/CONT...

When I compare the output I see in Neovim with the contents of these files, the Test Output panel prints file 1, 3 and 4. Sometimes they get garbled together but most of the time they print one at a time, but not always in the same order.

I wouldn't want file 1 printed in the test output panel, as this is what makes the test output so difficult to read.

It's also possible I get these four files because of how my test is structured:

func Test_Something(t *testing.T) {
	t.Parallel()
	t.Run("Create something", func(t *testing.T) {
		t.Parallel()
		t.Run("Create something with minimal data", func(t *testing.T) {
        ...

Here's a diff showing how I obtained these filepaths and was able to inspect their contents:

diff --git a/lua/neotest-go/init.lua b/lua/neotest-go/init.lua
index 8c97dc4..048fdc1 100644
--- a/lua/neotest-go/init.lua
+++ b/lua/neotest-go/init.lua
@@ -198,11 +198,16 @@ function adapter.results(spec, result, tree)
   end
 
   local success, lines = pcall(lib.files.read_lines, result.output)
+  print("file containing all output", vim.inspect(result.output))
   if not success then
     logger.error("neotest-go: could not read output: " .. lines)
     return {}
   end
-  return adapter.prepare_results(tree, lines, go_root, go_module)
+  local r = adapter.prepare_results(tree, lines, go_root, go_module)
+
+  print("FINAL")
+  print(vim.inspect(r))
+  return r
 end
 
 ---@param tree neotest.Tree
@@ -239,6 +244,7 @@ function adapter.prepare_results(tree, lines, go_root, go_module)
       -- file level node
       if test_result then
         local fname = async.fn.tempname()
+        print("nicely parsed results", fname, vim.inspect(test_result.output))
         fn.writefile(test_result.output, fname)
         results[value.id] = {
           status = test_result.status,
@@ -247,14 +253,15 @@ function adapter.prepare_results(tree, lines, go_root, go_module)

Here's another diff which removes the 1 file from disk before returning to neotest

diff --git a/lua/neotest-go/init.lua b/lua/neotest-go/init.lua
index 8c97dc4..98117d7 100644
--- a/lua/neotest-go/init.lua
+++ b/lua/neotest-go/init.lua
@@ -202,7 +202,10 @@ function adapter.results(spec, result, tree)
     logger.error("neotest-go: could not read output: " .. lines)
     return {}
   end
-  return adapter.prepare_results(tree, lines, go_root, go_module)
+
+  local r = adapter.prepare_results(tree, lines, go_root, go_module)
+  vim.fn.delete(result.output)
+  return r
 end

 ---@param tree neotest.Tree

This gives me a readable result in the test output, but yields an error, since neotest is attempting to read this (now non-existing) file:

   Error  09:10:33 msg_show.lua_error Error executing luv callback:
...drik/.local/share/LazyVim/lazy/neotest/lua/nio/tasks.lua:95: Async task failed without callback: The coroutine failed with this message: 
...share/LazyVim/lazy/neotest/lua/neotest/lib/file/init.lua:24: ENOENT: no such file or directory: /var/folders/85/kqpgt4g50kg91tm0j34bmtc80000gp/T/LazyVim.fredrik/ZHwAW6/1
stack traceback:
	[C]: in function 'assert'
	...share/LazyVim/lazy/neotest/lua/neotest/lib/file/init.lua:24: in function 'read'
	...lazy/neotest/lua/neotest/consumers/output_panel/init.lua:51: in function 'listener'
	.../LazyVim/lazy/neotest/lua/neotest/client/events/init.lua:51: in function <.../LazyVim/lazy/neotest/lua/neotest/client/events/init.lua:47>
stack traceback:
	[C]: in function 'error'
	...drik/.local/share/LazyVim/lazy/neotest/lua/nio/tasks.lua:95: in function 'close_task'
	...drik/.local/share/LazyVim/lazy/neotest/lua/nio/tasks.lua:117: in function 'cb'
	...drik/.local/share/LazyVim/lazy/neotest/lua/nio/tasks.lua:181: in function <...drik/.local/share/LazyVim/lazy/neotest/lua/nio/tasks.lua:180>

This means the 1 file is being read (from disk) by neotest even if you are not returning it from the adapter.results function.

@sergii4 is it possible that the contents of file 1 (containing all the JSON) is written to disk in error and shouldn't be written to disk?

[sergii4]

Hi @fredrikaverpil, wasn't it fixed by #54?

[fredrikaverpil]

Hi @fredrikaverpil, wasn't it fixed by #54?

@sergii4 yes, I must have been on some weird, locally modified, version of neotest-go... Sorry for the noise.

[fredrikaverpil]

@sergii4 I once again got json output in the "Neotest Output" pane. I don't quite understand what's triggering this behavior.

Sometimes I only see JSON output, sometimes I see a mix of nicely formatted non-JSON output. It's when I have to manually read through all the JSON output that this becomes really tedious. Weird... I wonder if this might have anything to do with testcontainers, which is being used in this project...

[Syakhisk]

@sergii4 can confirm that this is still happening as I don't locally modify any neotest-go files.

However, the output is only showing raw JSON when I open the individual test and the test output panel.

1. Output of the test function, 1st level(Correct)

2. Output of the nested test, 2nd level (Correct)

3. Output of individual test, 3rd level (Incorrect)

4. Output of test output panel (Incorrect)

This behavior seems to differ from files to files. Sometime it shows the correct output, sometimes not, depending on the file. (I was getting all raw JSON output on file using testify suite)

[fredrikaverpil]

I'm also working on and off with nested tests which could explain why I see this irregularly.

There's a showstopper here, unfortunately, which makes certain failing tests so hard to manually read (the JSON output) that I have to keep vscode open on the side and re-run the test there, only so I can see why the test was failing.

[sergii4]

Thank @Vjchi, @fredrikaverpil for brining this up and @Syakhisk for an excellent illustration. I have reproduced the bug. Taking a look 👁️

[Syakhisk]

Hey @sergii4, thanks a lot for taking the time to fix this.
But I'm still seeing the same result. Checked it on the new test file you added in the PR:

I'm using Go 1.21.6 if that helps

[sergii4]

Hi @Syakhisk, sorry for false hope. Could you please pull and try again?

[Syakhisk]

Yupp, it works on the tests in your PR. But I've found yet another bug.

if the 2nd level test has space in it, the output still messed up

If i remove the space, it works.

neotest-go/lua/neotest-go/init.lua

Lines 237 to 239 in e12fe00

	-- to remove quotes, for example three_level_nested_test.go::TestOdd::"odd"::5_is_odd
	local value_id = value.id:gsub('%"', "")
	local normalized_id = utils.normalize_id(value_id, go_root, go_module)

      -- to remove quotes, for example three_level_nested_test.go::TestOdd::"odd"::5_is_odd
      local value_id = value.id:gsub('%"', "")
      local normalized_id = utils.normalize_id(value_id, go_root, go_module)
      local test_result = tests[normalized_id]

e.g.

// expected
three_level_nested_test.go::TestOdd::odd_with_space::5_is_odd

// existing
three_level_nested_test.go::TestOdd::odd with space::5_is_odd

I think we should also substitute spaces with underscore beside removing the quotes.

[Syakhisk]

But upon checking on my real projects, it still shows the JSON.
I will come back to make a reproducible repo and possibly help debug (no promises :D) later in the week.

I think for now let's keep the issue open, wdyt? @sergii4

[sergii4]

@Syakhisk of course

[fredrikaverpil]

Thanks for your efforts on looking into this. I've added a minimal example here:
https://github.com/fredrikaverpil/go-playground/tree/main/bugs/neotest-go

[sergii4]

Hey @fredrikaverpil, could you please try #82 on your playground? It seems working to me

[fredrikaverpil]

@sergii4 this one actually still causes JSON output for me 😢
fredrikaverpil/go-playground@e5f851b

[sergii4]

@fredrikaverpil what I can say 🤷‍♂️ it's pity. But to be honest I really doubt somebody will write such non-liner test

[sergii4]

@fredrikaverpil seems that we get malformed data from neotest/tree-sitter. Fix requires extensive debugging.

[fredrikaverpil]

But to be honest I really doubt somebody will write such non-liner test

We actually have this kind of test at work. In our case we are doing a big refactoring effort where we port integration tests over onto a new gRPC service. The skipping is because we are porting code on a per-test basis.

So even if it looks exotic, it's actually in use where I work.

[sergii4]

Oh, really, sorry didn't expect that 😄

[rnesytov]

Hi
The output for testify suites is still broken. You can check on suite_test.go example.

Looks like the cause in this example is invalid test name matching by position_id and test_name from output generator.
With a little debug print here I got this message:

 No test result for normalized_id: tmp/main::TestExampleFailure , tests keys:  { "tmp/main::TestExampleTestSuite", "tmp/main::TestExampleTestSuite::TestExampleFailure", "tmp/main::TestExampleTestSuite::TestExample" }

I think it's possible to add suite name to neotest tree node and then use it to build position_id (and also to run single test suite method) with little treesitter query modification like this:

    (method_declaration
      receiver: (parameter_list
        (parameter_declaration 
          type: (pointer_type
            (type_identifier) @test.suite.name
          ) 
        )
      )
      name: (field_identifier) @test.name
      (#match? @test.name "^(Test|Example)")
    ) @test.definition

I tried to implement it, but got stuck on implementation and running tests.

[theoribeiro]

Yep, having the same issue with running individual tests of a Test Suite. Funny is that it still recognizes the test tables I have. It just apparently has issues parsing the results and showing as a successful run.

[theoribeiro]

I did some investigating since I didn't know how neotest adapters worked. The modified treesitter query wouldn't work unless there was an upstream change since Neotest only passes on test.name and test.definition. I don't know enough about Treesitter to know if we can append a captured variable with another (appending the receiver type) but I don't think you can.

I can see we might have three options:

1. Patch Neotest to pass over more capture groups in the tree node values
2. Find a way to get the full path including receiver name as the node id
3. Do a separate query on the document without using the Neotest treesitter query function

@sergii4 Do you see any other paths into fixing this? I'm happy to contribute with a PR if you point me the best way forward.

[sergii4]

Hi @rnesytov, @theoribeiro I need to take a grip on it. I have it on my todo list for the weekend. My last understanding after debugging was that problem comes from neotest/tree-sitter.

I am not sure it's connected to the suite test. I assume it's a separate problem

[fredrikaverpil]

While developing my own Neotest adapter for Go (fredrikaverpil/neotest-golang), I believe I found the root cause of all this (upstream issue): nvim-neotest/neotest#391

[sergii4]

Folks, I am not sure I have enough time to constantly contribute and keep up with upcoming issues since it requires long hours of laser-focused debugging. I will continue to support project but we definitely need more capacity

I assume we need more contributors. @fredrikaverpil is it something you are interested in?
cc: @akinsho

[fredrikaverpil]

It's completely understandable that priorities and availability change with time. Thanks for all the fine work you've done so far @sergii4 ❤️

I'm not interested in taking over this project as I developed my own Neotest adapter for Go (here) some time back and I rather focus my efforts there and on my specific needs.

[sergii4]

@fredrikaverpil thanks you! That's cool! Wish all the best with your shot!