Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add workflow test schema #17128

Closed
wants to merge 14 commits into from
Closed

Conversation

mvdbeek
Copy link
Member

@mvdbeek mvdbeek commented Dec 4, 2023

To generate dataclasses in generated/galaxy from xsd:

pip install 'xsdata[cli]'
cd lib/galaxy/tool_util/schemas
xsdata ../xsd/galaxy.xsd --unnest-classes --kw-only --docstring-style Accessible --ignore-patterns --max-line-length 10000

Fix up tool test schema for workflow particularities (and xsdata
odditites)

python postprocess_schema.py

To generate json schema (requires pydantic 2.0 to deal with recursive
data classes):

python test_file_from_xsd.py > test_file_schema.json

To validate a workflow test file with check-jsonschema

check-jsonschema --traceback-mode full --schemafile test_file_schema.json sra-manifest-to-concatenated-fastqs-tests.yml

TODO:

  • jobs is not modeled
  • element / element_tests needs work
  • special handling for primitive workflow parameter output values
  • de-RST-ify description fields
  • test against known workflow test files

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@mvdbeek mvdbeek changed the base branch from dev to release_23.2 December 5, 2023 10:05
@mvdbeek mvdbeek changed the base branch from release_23.2 to dev December 5, 2023 10:06
@mvdbeek mvdbeek changed the title [23.2] Add workflow test schema Add workflow test schema Dec 5, 2023
@mvdbeek mvdbeek force-pushed the add_test_schema branch 2 times, most recently from 1e6e734 to 5d1a63d Compare December 5, 2023 19:06
@mvdbeek mvdbeek force-pushed the add_test_schema branch 2 times, most recently from f7e758a to 37f908f Compare December 7, 2023 12:05
<xs:enumeration value="yes"/>
<xs:enumeration value="no"/>
</xs:restriction>
<xs:union>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nifty!

To generate dataclasses in generated/galaxy from xsd:

```
pip install 'xsdata[cli]'
cd lib/galaxy/tool_util/schemas
xsdata ../xsd/galaxy.xsd --unnest-classes --kw-only --docstring-style Accessible --ignore-patterns --max-line-length 10000
```

Fix up tool test schema for workflow particularities (and xsdata
odditites)
```
python postprocess_schema.py
```

To generate json schema (requires pydantic 2.0 to deal with recursive
data classes):
```
python test_file_from_xsd.py > test_file_schema.json
```

To validate a workflow test file with check-jsonschema
```
check-jsonschema --traceback-mode full --schemafile test_file_schema.json sra-manifest-to-concatenated-fastqs-tests.yml
```

TODO:
  - jobs is not modeled
  - element / element_tests needs work
  - special handling for primitive workflow parameter output values
  - de-RST-ify description fields
Pydantic's dataclass does not complain about optional attributes before
required attributes, so we can drop `--kw-only` from the `xsdata` call
(which is now `xsdata ../xsd/galaxy.xsd --unnest-classes --output
pydantic --docstring-style Accessible --ignore-patterns
--max-line-length 10000 && python postprocess_schema.py && python
test_file_from_xsd.py > test_file_schema.json && cd ~/src/galaxy && make
format && cd -`)
@mvdbeek
Copy link
Member Author

mvdbeek commented Nov 28, 2024

Much improved in #18884

@mvdbeek mvdbeek closed this Nov 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants