Skip to content

Commit

Permalink
Improve test runner a bit
Browse files Browse the repository at this point in the history
Explicitly list the toml-test files we know to fail, so it's easier to
test for regressions and the like on changes. Can also add GitHub
actions if you want.

The stdex/float6 test was already failing on master.

"make check" will run all the tests (closes #83).

Also remove the https://github.com/cktan/toml-spec-tests, based on
@iarna/toml. These are no longer maintained, and toml-lang/toml-test
contains everything that's in there.
  • Loading branch information
arp242 committed Oct 8, 2023
1 parent 88b6b18 commit b669dea
Show file tree
Hide file tree
Showing 17 changed files with 80 additions and 93 deletions.
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@
toml_cat
toml_json
toml_sample
/unittest/t1
/toml-test/toml-test

# Debug files
*.dSYM/
*.su
*.su
11 changes: 10 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,13 @@ clean:
format:
clang-format -i $(shell find . -name '*.[ch]')

.PHONY: all clean install format
toml-test: all
@./toml-test/build.sh
@./toml-test/run.sh

check: toml-test
@make -C unittest
@./unittest/t1
./stdex/RUN.sh

.PHONY: all clean install format toml-test check
24 changes: 5 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,7 @@ TOML in c99; v1.0 compliant.
If you are looking for a C++ library, you might try this wrapper: [https://github.com/cktan/tomlcpp](https://github.com/cktan/tomlcpp).

* Compatible with [TOML v1.0.0](https://toml.io/en/v1.0.0).
* Tested with multiple test suites, including
[toml-lang/toml-test](https://github.com/toml-lang/toml-test) and
[iarna/toml-spec-tests](https://github.com/iarna/toml-spec-tests).
* Tested with [toml-lang/toml-test](https://github.com/toml-lang/toml-test).
* Provides very simple and intuitive interface.


Expand Down Expand Up @@ -174,21 +172,9 @@ Alternatively, specify `make install prefix=/a/file/path` to install into
## Testing
To test against the standard test set provided by toml-lang/toml-test:
Run `make toml-test` to test against the standard test set provided by
toml-lang/toml-test.
```sh
% make
% cd test1
% bash build.sh # do this once
% bash run.sh # this will run the test suite
```


To test against the standard test set provided by iarna/toml:
You will need to have Go installed for this to work.
```sh
% make
% cd test2
% bash build.sh # do this once
% bash run.sh # this will run the test suite
```
Run `make check` to run the toml-test tests and some additional unit tests.
22 changes: 17 additions & 5 deletions stdex/RUN.sh
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,15 +1,27 @@
#!/usr/bin/env bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

cd "$DIR"
rm -f *.out

ok=0
fail=0
for i in *.toml; do
echo -n $i
../toml_cat $i >& $i.out
if [ -f $i.res ]; then
if $(diff $i.out $i.res >& /dev/null); then
echo " [OK]"
ok=$(( ok + 1 ))
else
echo " [FAILED]"
fail=$(( fail + 1 ))
echo >&2 "$0: $i [FAILED]"
diff -u $i.out $i.res >&2
echo >&2
fi
else
echo " [?????]"
echo "$i [?????]"
fail=$(( fail + 1 ))
fi

done

echo "$0: ok: $ok fail: $fail"
2 changes: 1 addition & 1 deletion stdex/float6.toml.res
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@
sf3 = -inf,
sf4 = nan,
sf5 = nan,
sf6 = nan,
sf6 = -nan,
}
3 changes: 0 additions & 3 deletions test1/.gitignore

This file was deleted.

5 changes: 0 additions & 5 deletions test1/run.sh

This file was deleted.

1 change: 0 additions & 1 deletion test2/.gitignore

This file was deleted.

6 changes: 0 additions & 6 deletions test2/Note.txt

This file was deleted.

7 changes: 0 additions & 7 deletions test2/build.sh

This file was deleted.

44 changes: 0 additions & 44 deletions test2/run.sh

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
44 changes: 44 additions & 0 deletions toml-test/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#!/usr/bin/env bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

failing=(
-skip valid/key/escapes # Treats "\n" as invalid because it first replaces escapes and then checks.
-skip valid/key/quoted-unicode # Doesn't print null byte correctly.
-skip valid/string/quoted-unicode
-skip valid/spec/string-7 # Lots of ''''''' ... that somehow goes wrong.
-skip invalid/string/literal-multiline-quotes-1
-skip invalid/string/literal-multiline-quotes-2
-skip invalid/string/multiline-quotes-1
-skip invalid/inline-table/trailing-comma # Trailing comma should be error; not worth fixing as it'll be allowed in 1.1
-skip invalid/inline-table/add # Appending existing tables
-skip invalid/array/extending-table
-skip invalid/table/append-with-dotted-keys-1
-skip invalid/table/append-with-dotted-keys-2
-skip invalid/control/bare-cr # Doesn't reject some forbidden control characters.
-skip invalid/control/bare-null
-skip invalid/control/comment-cr
-skip invalid/control/comment-del
-skip invalid/control/comment-lf
-skip invalid/control/comment-null
-skip invalid/control/comment-us
-skip invalid/encoding/bad-codepoint # Doesn't reject invalid UTF-8; nothing is multi-byte aware, so...
-skip invalid/encoding/bad-utf8-in-comment
-skip invalid/encoding/bad-utf8-in-multiline-literal
-skip invalid/encoding/bad-utf8-in-multiline
-skip invalid/encoding/bad-utf8-in-string-literal
-skip invalid/encoding/bad-utf8-in-string
-skip invalid/encoding/utf16

# These seem broken in toml-test...
-skip valid/spec/local-date-time-0
-skip valid/spec/local-time-0
-skip valid/spec/offset-date-time-0

# Fixed in master:
# https://github.com/toml-lang/toml-test/commit/fe8e1e2fb8b0309d54d6ad677aecb86bcb0ff138
-skip valid/float/inf-and-nan
-skip valid/spec/float-2
)

$DIR/toml-test $DIR/../toml_json ${failing[@]} "$@"

0 comments on commit b669dea

Please sign in to comment.