-
-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide an option to distinguish between runtime and schema-checkable failures #253
Comments
We can add another tag for the relevant |
Yes. That would be sufficient. Thank you. |
@fmigneault Feel free to send a PR to add the |
fmigneault
added a commit
to fmigneault/cwl-v1.2
that referenced
this issue
Jun 13, 2023
…ow-language#253) + validation tests (2 failed related to common-workflow-language#251)
mr-c
pushed a commit
to fmigneault/cwl-v1.2
that referenced
this issue
Jun 19, 2023
…ow-language#253) + validation tests (2 failed related to common-workflow-language#251)
mr-c
pushed a commit
to fmigneault/cwl-v1.2
that referenced
this issue
Jul 19, 2023
…ow-language#253) + validation tests (2 failed related to common-workflow-language#251)
mr-c
pushed a commit
that referenced
this issue
Jul 19, 2023
…ation tests (2 failed related to #251)
GlassOfWhiskey
pushed a commit
that referenced
this issue
Nov 15, 2023
…ation tests (2 failed related to #251)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As mentioned in the conformance tests, we can use
should_fail
to know the expected outcome:cwl-v1.2/CONFORMANCE_TESTS.md
Lines 115 to 116 in d50fad0
However, some tests are more of "runtime" nature:
cwl-v1.2/tests/loadContents/test-index.yaml
Lines 8 to 14 in d50fad0
While others can be validated beforehand, without even runing the tool, only using the document and schema validation:
cwl-v1.2/conformance_tests.yaml
Lines 2631 to 2636 in d50fad0
cwl-v1.2/tests/mixed-versions/test-index.yaml
Lines 28 to 33 in d50fad0
cwl-v1.2/tests/mixed-versions/test-index.yaml
Lines 35 to 40 in d50fad0
It would be useful to provide a flag that allows this distinction.
Using the JSON schema validation I'm working on (opengeospatial/ogcapi-processes#329), there is no way to distinguish between the two.
I could always filter tests based on a known set of cases, but then, I goes against the use of the configurable test conformance.
The text was updated successfully, but these errors were encountered: