Common test suite

[tstack]

Are there plans to create a common suite of test configuration files? It looks like the implementations have similar sets of files copied between them at the moment.

[Genarito]

Hi @tstack!
First of all, thanks for your time opening this discussion.
Indeed, the tests were copied since the official implementations are a port of a parser that was made in Python in the first instance. How would you standardize a test suite for the different implementations?

[tstack]

How would you standardize a test suite for the different implementations?

The TOML/compliance sounds like the right approach: have the implementations provide a driver program that takes an input file and then spits out the result; the test harness then takes care of running the driver with the input files and comparing the actual results to the expected. You could even be prescriptive with error messages as well, so they were consistent across impls. But, that might be pushing it.

[Genarito]

Great! I've just submitted a message to discuss this with the community. When I finish some priority things I'll get to work on this

[Genarito]

@tstack We have just published a new repo with all the available test cases for the current implementations. This repo will be introduced in the new version of Gura specs (2.0.0).

Link to the Testing repo

[facundoq]

I think Tim refers to having a separate repository of gura (.gura) test files that all implementations must be able to parse correctly. Therefore if Gura changes (for example adding the empty keyword), and the test files are changed accordingly, the tests for each implementation will also start to fail. This could be implemented as a git dependency. El vie., 30 de jul. de 2021 18:21, Genaro Camele ***@***.***> escribió:

…

Hi @tstack <https://github.com/tstack>! First of all, thanks for your time making this issue. Indeed, the tests were copied since the official implementations are a port of a parser that was made in Python in the first instance. How would you standardize a test suite for the different implementations? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMA6B5NUWMC2H3ADX3CY3DT2MJW3ANCNFSM5BJG5NTQ> .

[Genarito]

I see.. If I didn't misunderstand, It'd be something similar to this TOML repo, wouldn't it? We could make something similar

[Genarito]

Hi @tstack,

I have been thinking about this problem. The TOML proposal is good when the language specifications define only what is valid and what is not. In the case of Gura, there are standardized errors that are exceptions of the programming language in which the encoding/decoding is being implemented, so a language-agnostic solution is not possible. Also, TOML runs the encoder/decoder from the command line, so it is not useful for those implementations that are just libraries but do not have CLI functionality.

What I came up with, to save work to whoever is developing a Gura implementation, is to simply provide several .ura files separated in valid folders and the others in different folders representing the error they should throw. For example ParseErrror, InvalidIndentationError, etc.

This provides, optionally, several use cases that the developer can incorporate into the solution he/she is developing most conveniently.

What do you think about this?