Testing Natspec JSON via isoltest

[cameel]

Currently Natspec is tested by SolidityNatspecJSON.cpp, which contains Boost tests. Such tests are harder to maintain than our isoltest test cases. This makes us reluctant to add more of them and leads to insufficient coverage and bugs like #14430.

The task here is to create a new isoltest test case for Natspec and covert existing test cases to it.

[cameel]

As a first step, we need to extract source code and expectations from SolidityNatspecJSON.cpp. That's almost 100 cases and doing that by hand would be very tedious. Instead I wrote a one-off parser using Python's pyparsing library to deal with the problem.

The script itself will not be useful after we run it once so I will not be submitting it as a part of my upcoming PR. It might be still serve as a template for writing similar one-off parsers though so I'm posting it here for future reference.

Parser script

Script expects the content of SolidityNatspecJSON.cpp on standard input. The file needs to be prepared for parsing by:

• Removing everything before the first BOOST_AUTO_TEST_CASE
• Removing everything after the final } of the last BOOST_AUTO_TEST_CASE
• Removing the comment before emit_event_from_foreign_contract_no_inheritance.
• Removing some test cases - these have to be coverted manually:
  • dev_explicit_inehrit_complex case - too complex for the script
  • dev_multiple_params_mixed_whitespace - non-standard whitespace would be mangled by the script

The script will create an input .sol file for every test case and put devdoc and userdoc expectations in expectation comments.

Error expectations are ignored - the current tests do not specify error messages and I will make isoltest just generate them instead.

JSON files are passed through Python's JSON parser to convert them to a uniform style. For Solidity files indentation is converted to our usual style.

#!/usr/bin/env python

import json
import re
from textwrap import dedent
from textwrap import indent
import sys

import pyparsing as pp

pp.ParserElement.inlineLiteralsUsing(pp.Suppress)

CPP_STRING = (
    pp.QuotedString(quote_char='"', esc_char='\\', multiline=True)
    | pp.QuotedString(quote_char='R"(', end_quote_char=')"', multiline=True)
    | pp.QuotedString(quote_char='R"X(', end_quote_char=')X"', multiline=True)
    | pp.QuotedString(quote_char='R"R(', end_quote_char=')R"', multiline=True)
    | pp.QuotedString(quote_char='R"ABCDEF(', end_quote_char=')ABCDEF"', multiline=True)
);
IDENTIFIER = pp.Word(pp.identchars + pp.nums)
BOOL_LITERAL = pp.Literal('true') | pp.Literal('false')

STRING_VARIABLE = pp.Group(
    pp.Suppress('char') - 'const' - '*'
    - IDENTIFIER('variable_name') - '='
    - pp.Group(CPP_STRING[...])('content') - ';'
)
CHECK_NATSPEC_CALL = pp.Group(
    pp.Suppress('checkNatspec') - '('
    - IDENTIFIER('source_variable_name') - ','
    - CPP_STRING('contract_name') - ','
    - IDENTIFIER('natspec_variable_name') - ','
    - BOOL_LITERAL('userdoc_flag')
    - ')' - ';'
)
EXPECT_NATSPEC_ERROR_CALL = pp.Group(
    pp.Suppress('expectNatspecError') - '('
    - IDENTIFIER('source_variable_name')
    - ')' - ';'
)

TEST_STEP = (
    STRING_VARIABLE('string_variable')
    | CHECK_NATSPEC_CALL('positive_expectation')
    | EXPECT_NATSPEC_ERROR_CALL('negative_expectation')
)
TEST_BODY = '{' - TEST_STEP[...]('steps') + '}'
TEST_HEADER = pp.Suppress('BOOST_AUTO_TEST_CASE') - '(' - IDENTIFIER('name') - ')'
NATSPEC_TEST = pp.Group(TEST_HEADER - TEST_BODY)
TEST_FILE = NATSPEC_TEST[...]

def format_error(exception):
    line_number_prefix = f"{exception.lineno} | "
    return dedent(f"""\
        {exception}
        {line_number_prefix}{exception.line}
        {' ':{len(line_number_prefix) + exception.col - 1}}^
    """)

try:
    ast = TEST_FILE.parse_string(sys.stdin.read(), parse_all=True)
except pp.ParseException as exception:
    sys.exit(format_error(exception))
except pp.ParseSyntaxException as exception:
    sys.exit(format_error(exception))

for test_case in ast:
    variables = {
        step.variable_name: dedent(''.join(step.content))
        for step in test_case.steps
        if step.get_name() == 'string_variable'
    }

    expectations = {'userdoc': {}, 'devdoc': {}}
    for step in test_case.steps:
        if step.get_name() == 'positive_expectation':
            assert step.source_variable_name == 'sourceCode'
            expectations[step.contract_name] = expectations.get(step.contract_name, {})

            expectation_kind = 'userdoc' if step.userdoc_flag == 'true' else 'devdoc'
            assert expectation_kind not in expectations[step.contract_name]
            expectations[step.contract_name][expectation_kind] = json.loads(variables[step.natspec_variable_name])

    input_file_name = test_case.name + '.sol'
    print(f'{input_file_name}')
    with open(input_file_name, 'w') as input_file:
        input_file.write(
            variables['sourceCode']
            .strip()
            .replace('\t', '    ')
            .replace('        ', '    ')
            + '\n\n'
        )
        input_file.write('// ----\n')

        first = True
        for contract, expectation_kinds in expectations.items():
            for expectation_kind, expectation_json in sorted(expectation_kinds.items()):
                if not first:
                    input_file.write(f"//\n")
                first = False

                formatted_json = re.sub(
                    r'^((\s*)".*?":)\s*([\[{])($|[^\]}])', r'\1\n\2\3\4',
                    json.dumps(expectation_json, indent=4, sort_keys=True),
                    flags=re.MULTILINE
                )

                input_file.write(f"// :{contract} {expectation_kind}\n")
                input_file.write(indent(formatted_json + '\n', prefix='// '))