Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save previous expected files for testing #93

Closed
forsyth2 opened this issue Jul 1, 2021 · 1 comment
Closed

Save previous expected files for testing #93

forsyth2 opened this issue Jul 1, 2021 · 1 comment
Assignees
Labels
Documentation Files in `docs` modified priority: low Low priority task Testing Files in `tests` modified

Comments

@forsyth2
Copy link
Collaborator

forsyth2 commented Jul 1, 2021

It may be beneficial to save previous expected images/files, as is done in E3SM Diags: https://e3sm-project.github.io/e3sm_diags/_build/html/master/dev_guide/testing.html

Currently, /lcrc/group/e3sm/public_html/zppy_test_resources contains the following:

  • expected_bash_files
  • expected_complete_run
  • expected_environment_commands
  • image_list_expected_complete_run.txt
  • image_list_expected_environment_commands.txt

The E3SM Diags integration test directory /lcrc/group/e3sm/public_html/e3sm_diags_test_data/integration/expected contains the following:

  • integration_test_images, the current expected images
  • previous_output
    • integration_test_images_<version>_<date>_<hash>
  • README.md containing the version, date, hash for the current expected images.

(/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/expected has a similar setup).

Options:

  • Move the current contents of /lcrc/group/e3sm/public_html/zppy_test_resources into /lcrc/group/e3sm/public_html/zppy_test_resources/expected and put all previous output in /lcrc/group/e3sm/public_html/zppy_test_resources/previous_expected. The README listing the version/date/hash could be placed in /lcrc/group/e3sm/public_html/zppy_test_resources/.
  • Have subdirectories for each of the three integration tests in /lcrc/group/e3sm/public_html/zppy_test_resources, each with their own READMEs and expected/previous_expected directories.

Note that any changes to the directory structure will require updating the paths in the tests. Run git grep "/lcrc/group/e3sm/public_html/zppy_test_resources" in zppy/tests/integration to find these references.

Documentation on testing would also have to be updated to reflect the process for updating expected files.

@forsyth2 forsyth2 added priority: low Low priority task Testing Files in `tests` modified labels Jul 1, 2021
@forsyth2 forsyth2 self-assigned this Jul 1, 2021
@forsyth2 forsyth2 added the Documentation Files in `docs` modified label Jul 1, 2021
@forsyth2 forsyth2 mentioned this issue Jul 1, 2021
@forsyth2
Copy link
Collaborator Author

As of #604, we have scripts for generating archives of expected results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation Files in `docs` modified priority: low Low priority task Testing Files in `tests` modified
Projects
None yet
Development

No branches or pull requests

1 participant