Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'type' field to test case format #18

Open
mkalinin opened this issue Jan 30, 2019 · 6 comments
Open

Add 'type' field to test case format #18

mkalinin opened this issue Jan 30, 2019 · 6 comments

Comments

@mkalinin
Copy link

Proposal

Add an optional type field to test_case section of test format. It might, also, make sense to add a types field to test_suite header with a list of all underlying test case types.

Rationale

It's a necessary thing for building heterogeneous test suites like SSZ, tree hashing , BLS and a lot more. If all test cases in the suite are of the same kind, e.g. state transition or shuffling tests, then type field could be omitted.

Requested by https://github.com/ethereum/eth2.0-tests/issues/8, https://github.com/ethereum/eth2.0-tests/issues/11

@jannikluhn
Copy link
Collaborator

I don't think this needs to be part of the general test case format as a test suite can already define whatever fields they like for a test suite.

@mkalinin
Copy link
Author

mkalinin commented Feb 6, 2019

Adding a type field for all test suites would be convenient in terms of implementation. Otherwise, you will have to add custom fields to each suite which achieves more or less same goal.
@jannikluhn do you see any obstacles to add a general type field to the format?

@djrtwo
Copy link
Contributor

djrtwo commented Feb 6, 2019

I think optional type in base format fits our needs. Some tests might just be generic (potentially fork choice and state tests) and not require a type but standardizing how to distinguish among test types is a win imo

@jannikluhn
Copy link
Collaborator

👍 Adding it as an optional but standardized way of distinguishing different test types seems reasonable.

Not sure about types in the test suite header though, this seems redundant (if you know the test suite, you know the valid types (and if you don't, you haven't implemented the test suite properly, or an outdated version). But I don't feel too strongly about this.

@djrtwo
Copy link
Contributor

djrtwo commented Feb 6, 2019

yeah, don't think we need types in top header

@hwwhww hwwhww transferred this issue from ethereum/eth2.0-tests Feb 14, 2019
@protolambda
Copy link

Please check #29 , it introduces a structured and efficient approach to test types and suites :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants