[CHECKPOINT]

compiler/sem/semcall.nim

@@ -113,7 +113,7 @@ proc pickBestCandidate(c: PContext, headSymbol: PNode,
           firstMismatch: z.firstMismatch,
           diagnostics: z.diagnostics,
           isDiagnostic: z.diagnosticsEnabled or efExplain in flags
-          ))
+        ))
 
     else:
       # Symbol table has been modified. Restart and pre-calculate all syms

doc/testament.rst

@@ -91,98 +91,103 @@ you have to run at least 1 test *before* generating a report:
 Writing Tests
 =============
 
-Example "template" **to edit** and write a Testament unit:
 
-.. code-block:: nim
+Test execution options
+----------------------
 
-  discard """
 
-    # What actions to expect completion on.
-    # Options:
-    #   "compile": expect successful compilation
-    #   "run": expect successful compilation and execution
-    #   "reject": expect failed compilation. The "reject" action can catch
-    #             {.error.} pragmas but not {.fatal.} pragmas because
-    #             {.fatal.} pragmas guarantee that compilation will be aborted.
-    action: "run"
-
-    # The exit code that the test is expected to return. Typically, the default
-    # value of 0 is fine. Note that if the test will be run by valgrind, then
-    # the test will exit with either a code of 0 on success or 1 on failure.
-    exitcode: 0
-
-    # Provide an `output` string to assert that the test prints to standard out
-    # exactly the expected string. Provide an `outputsub` string to assert that
-    # the string given here is a substring of the standard out output of the
-    # test.
-    output: ""
-    outputsub: ""
-
-    # Whether to sort the output lines before comparing them to the desired
-    # output.
-    sortoutput: true
-
-    # Each line in the string given here appears in the same order in the
-    # compiler output, but there may be more lines that appear before, after, or
-    # in between them.
-    nimout: '''
-  a very long,
-  multi-line
-  string'''
+- ``action`` - What actions to expect completion on.
+  - ``"compile"``: expect successful compilation
+  - ``"run"``: expect successful compilation and execution
+  - ``"reject"``: expect failed compilation. The "reject" action can catch
+    `{.error.}` pragmas but not `{.fatal.}` pragmas because `{.fatal.}` pragmas
+    guarantee that compilation will be aborted.
 
-    # This is the Standard Input the test should take, if any.
-    input: ""
+- ``batchable``: Can be run in batch mode, or not.
 
-    # Error message the test should print, if any.
-    errormsg: ""
+- ``joinable``: Can be run Joined with other tests to run all togheter, or
+  not. Defaults to `true`
 
-    # Can be run in batch mode, or not.
-    batchable: true
+- ``timeout`` Timeout seconds to run the test. Fractional values are supported.
 
-    # Can be run Joined with other tests to run all togheter, or not.
-    joinable: true
+- ``cmd``: Command the test should use to run. If left out or an empty
+  string is provided, the command is taken to be: ``"nim $target --hints:on
+  -d:testing --nimblePath:build/deps/pkgs $options $file"`` You can use the
+  ``$target``, ``$options``, and ``$file`` placeholders in your own
+  command, too.
 
-    # On Linux 64-bit machines, whether to use Valgrind to check for bad memory
-    # accesses or memory leaks. On other architectures, the test will be run
-    # as-is, without Valgrind.
-    # Options:
-    #   true: run the test with Valgrind
-    #   false: run the without Valgrind
-    #   "leaks": run the test with Valgrind, but do not check for memory leaks
-    valgrind: false   # Can use Valgrind to check for memory leaks, or not (Linux 64Bit only).
+  example: ``"nim c -r $file"``
 
-    # Command the test should use to run. If left out or an empty string is
-    # provided, the command is taken to be:
-    # "nim $target --hints:on -d:testing --nimblePath:build/deps/pkgs $options $file"
-    # You can use the $target, $options, and $file placeholders in your own
-    # command, too.
-    cmd: "nim c -r $file"
+- ``targets`` Targets to run the test into (c, cpp, objc, js).
 
-    # Maximum generated temporary intermediate code file size for the test.
-    maxcodesize: 666
+- ``matrix`` flags with which to run the test, delimited by `;`
 
-    # Timeout seconds to run the test. Fractional values are supported.
-    timeout: 1.5
+- ``disabled`` Conditions that will skip this test. Use of multiple
+  "disabled" clauses is permitted.
 
-    # Targets to run the test into (c, cpp, objc, js).
-    targets: "c js"
+  .. code-block:: nim
 
-    # flags with which to run the test, delimited by `;`
-    matrix: "; -d:release; -d:caseFoo -d:release"
-
-    # Conditions that will skip this test. Use of multiple "disabled" clauses
-    # is permitted.
     disabled: "bsd"   # Can disable OSes...
     disabled: "win"
     disabled: "32bit" # ...or architectures
     disabled: "i386"
     disabled: "azure" # ...or pipeline runners
     disabled: true    # ...or can disable the test entirely
 
-  """
-  assert true
-  assert 42 == 42, "Assert error message"
 
+Compiler output assertions
+--------------------------
+
+- ``errormsg``: Error message the test should print, if any.
+
+
+- ``nimout`` Each line in the string given here appears in the same order
+    in the compiler output, but there may be more lines that appear before,
+    after, or in between them. Note that specifying multiline strings for
+    testament spec inside of the `discard """` section requires using
+    triple single quotes `'`
+
+    .. code-block:: nim
+
+        nimout: '''
+        a very long,
+        multi-line
+        string'''
+
+- ``nimoutFull``: true/false, controls whether full compiler output must be
+  asserted, or only presence of error messages
+
+- ``maxcodesize``: Max side of the resulting codegen file for a test
+
+In addition to ``nimout`` message annotations testament also allows to
+supply error message
+
+Binary output assertions
+------------------------
+
+- ``exitcode``: The exit code that the test is expected to return.
+  Typically, the default value of 0 is fine. Note that if the test will be
+  run by valgrind, then the test will exit with either a code of 0 on
+  success or 1 on failure.
+
+- ``output``, ``outsub``: Provide an `output` string to assert that the
+  test prints to standard out exactly the expected string. Provide an
+  `outputsub` string to assert that the string given here is a substring of
+  the standard out output of the test.
+
+- ``sortoutput`` Whether to sort the output lines before comparing them to
+  the desired output.
+
+- ``input``: this is the Standard Input the test should take, if any.
+
+
+- ``valgrind`` On Linux 64-bit machines, whether to use Valgrind to check
+    for bad memory accesses or memory leaks. On other architectures, the
+    test will be run as-is, without Valgrind.
+
+    - ``true``: run the test with Valgrind
+    - ``false``: run the without Valgrind
+    - ``"leaks"``: run the test with Valgrind, but do not check for memory leaks
 
 * As you can see the "Spec" is just a `discard """ """`.
 * Spec has sane defaults, so you don't need to provide them all, any simple assert will work just fine.
@@ -191,10 +196,129 @@ Example "template" **to edit** and write a Testament unit:
 * Has some built-in CI compatibility, like Azure Pipelines, etc.
 * `Testament supports inlined error messages on tests, basically comments with the expected error directly on the code. <https://github.com/nim-works/nimskull/blob/9a110047cbe2826b1d4afe63e3a1f5a08422b73f/tests/effects/teffects1.nim>`_
 
+Reading test outputs
+====================
+
+Testament supports two different modes of interaction with the compiler -
+structured and unstructured. Unstructured interaction mode (currently
+default) allows user to specify exact compiler output that should be
+produced by input file and then compares it based on the ``nimoutFull``
+configuration options.
+
+If tests are mismatched, failure message is generated, showing failed
+diffs.
+
+.. code-block::
+
+    discard """
+    nimout: '''
+    Expected unstructured compiler output
+    '''
+    """
+
+    static:
+      echo "Expected unstructured output"
+
+In that case comparison is performed between two regular string blocks.
+Since each entry is not wide enough (not wider than current terminal) they
+are printed side-by side to make it easier to spot the difference.
+Mismatches are also highlighted in the terminal.
+
+.. code-block:: diff
+
+    - Expected unstructured compiler output   + Expected unstructured output
+    -                                         ?
+
+.. note:: expected (on the left) outout has two lines deleted - trailing
+          ``'''`` in the testament spec is placed on the next line, so it
+          is considered to be a string literal of ``"Expected unstructured
+          compiler output\n"``
+
+
+Structured mismatches
+---------------------
+
+In structured output mode, compiler writes out S-expressions for each
+output diagnostic entry, one per line.
+
+If testament is used in structured mode, all expected compiler reports -
+both inline and written in ``nimout`` are collected in a single list that
+is matched against produced output directly. Failure message shows best
+possible mismatch annotations for the given output. For example, given test
+like this, testament output will contain two mismatches for both failures.
+
+.. code-block:: nim
+    :linenos:
+
+    discard """
+    nimoutFormat: sexp
+    cmd: "nim c --msgFormat=sexp --skipUserCfg --hints=on --hint=all:off --hint=User:on --filenames:canonical $file"
+    nimout: '''
+    (User :str "User Hint" :location ("tfile.nim" 8 _))
+    '''
+    """
+
+    {.hint: "User hint".}
+
+    {.hint: "Another hint".} #[tt.Hint
+          ^ (User :str "Another hint") ]#
+
+
+Both inline and ``nimout`` annotations are compared. Both have errors, so
+the best possible mapping is presented as an error ('best' because it is
+generally impossible to find correct place to insert inline annotation
+somewhere in ``nimout``, without potentially messing up ordering.
+Unstructured output simply sets inline annotations to a higher priority and
+searches for them first)
+
+.. code-block:: nim
+
+    Expected inline Hint annotation at tfile.nim(11, 7):
+
+    - (User :location ("tfile.nim" 11 7) :severity Hint :str "Another hint")
+
+    Given:
+
+    + (User :location ("tfile.nim" 11 6) :severity Hint :str "Another hint")
+
+
+      :location[2] expected 7, but got 6 ([7->6])
+
+    Expected:
+
+    - (User :location ("tfile.nim" 8 _) :str "User Hint")
+
+    Given:
+
+    + (User :location ("tfile.nim" 9 6) :severity Hint :str "User hint")
+
+
+      :str expected "User Hint", but got "User hint" ("User [Hint"->hint"])
+      :location[1] expected 8, but got 9 ([8->9])
+
+Compiler printed reports
+
+.. code-block:: nim
+
+    (User :severity Hint :str "User hint" :location ("tfile.nim" 8 6))
+    (User :severity Hint :str "Another hint" :location ("tfile.nim" 10 6))
+
+And they were matched against full list of expected entries. For the first
+entry there is a mismatch in ``:location[2]``, and for secdon one there is
+a string value error (in ``:str``) and another mismatch in location data.
+
+To make it easier to spot differences between string values the inline diff
+is added for the message.
 
 Test Examples
 ==============
 
+Structured test examples
+------------------------
+
+Unstructured old, style test examples
+--------------------------------------
+
 Expected to fail:
 
 .. code-block:: nim