Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an option to distinguish between runtime and schema-checkable failures #253

Closed
fmigneault opened this issue Jun 5, 2023 · 3 comments · Fixed by #256
Closed

Provide an option to distinguish between runtime and schema-checkable failures #253

fmigneault opened this issue Jun 5, 2023 · 3 comments · Fixed by #256

Comments

@fmigneault
Copy link
Contributor

As mentioned in the conformance tests, we can use should_fail to know the expected outcome:

A single conformance test consist of the path to an input CWL document plus an input CWL object
and the expected outputs (or `should_fail: true` if the test is deliberately broken)

However, some tests are more of "runtime" nature:

- job: input.yml
tool: loadContents-limit.cwl
id: loadcontents_limit
output: {}
doc: "Test that loadContents on a file larger than 64k fails"
should_fail: true
tags: [ command_line_tool, required ]

While others can be validated beforehand, without even runing the tool, only using the document and schema validation:

- job: tests/empty.json
should_fail: true
tool: tests/timelimit2.cwl
id: timelimit_invalid
doc: Test invalid time limit value
tags: [ command_line_tool, timelimit ]

- job: null
tool: invalid-tool-v10.cwl
id: invalid_syntax_v10_uses_v12_tool
doc: test tool with v1.2 syntax marked as v1.0 (should fail)
should_fail: true
tags: [ command_line_tool ]

- job: null
tool: invalid-tool-v11.cwl
doc: test tool with v1.2 syntax marked as v1.1 (should fail)
id: invalid_syntax_v11_uses_v12_tool
should_fail: true
tags: [ command_line_tool ]

It would be useful to provide a flag that allows this distinction.
Using the JSON schema validation I'm working on (opengeospatial/ogcapi-processes#329), there is no way to distinguish between the two.
I could always filter tests based on a known set of cases, but then, I goes against the use of the configurable test conformance.

@mr-c
Copy link
Member

mr-c commented Jun 6, 2023

We can add another tag for the relevant should_fail: true tests, like tags: […, schema_valid]; would that be good enough?

@fmigneault
Copy link
Contributor Author

Yes. That would be sufficient. Thank you.

@mr-c
Copy link
Member

mr-c commented Jun 7, 2023

@fmigneault Feel free to send a PR to add the schema_valid tag to the should_fail: true tests that you think need it; I'll review it and merge it

fmigneault added a commit to fmigneault/cwl-v1.2 that referenced this issue Jun 13, 2023
mr-c pushed a commit to fmigneault/cwl-v1.2 that referenced this issue Jun 19, 2023
mr-c pushed a commit to fmigneault/cwl-v1.2 that referenced this issue Jul 19, 2023
@mr-c mr-c closed this as completed in #256 Jul 19, 2023
mr-c pushed a commit that referenced this issue Jul 19, 2023
GlassOfWhiskey pushed a commit that referenced this issue Nov 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants