-
-
Notifications
You must be signed in to change notification settings - Fork 404
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
update refdata procedure docs (#1609)
* initial commit * more changes to update refdata page
- Loading branch information
Showing
1 changed file
with
40 additions
and
68 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,84 +1,56 @@ | ||
| ************************************** | ||
| Procedure to update the reference data | ||
| ************************************** | ||
| .. _update refdata: | ||
|
|
||
| We assume that you have added the necessary changes to TARDIS and have an open pull request. | ||
| ************************* | ||
| Update the Reference Data | ||
| ************************* | ||
|
|
||
| #. Fork the ``tardis-refdata`` repository as described in :ref:`development-workflow`. | ||
| A special kind of tests are executed only when ``pytest`` is called alongside the ``--refdata`` flag. | ||
| These tests compares the output of the TARDIS code (mostly arrays) against the information stored | ||
| in the reference data files. | ||
|
|
||
| #. Clone ``tardis-refdata`` to your computer. Then make a new branch named the same as your new TARDIS feature branch. Make sure to have correctly set up the ``upstream`` remote as explained in :ref:`reviewing-and-helping-with-pr`. | ||
| TARDIS stores reference data in the `tardis-refdata <https://github.com/tardis-sn/tardis-refdata>`_ | ||
| repository. This repository also has a mirror hosted in Azure Pipelines (synchronized automatically by a | ||
| GitHub workflow) since this Microsoft service does not have limitations in bandwith nor storage. | ||
|
|
||
| #. Activate your ``tardis`` conda environment. | ||
| Sometimes, this data needs to be updated. The procedure to update these files manually is not trivial | ||
| and has been automatized recently thanks to the `NumFOCUS <https://numfocus.org/>`_ support. | ||
|
|
||
| .. code-block:: none | ||
| conda activate tardis | ||
|
|
||
| #. Navigate to your ``tardis-refdata`` directory and type: | ||
| ================= | ||
| Default Procedure | ||
| ================= | ||
|
|
||
| .. code-block:: none | ||
| git lfs install | ||
| git lfs fetch upstream | ||
| git lfs pull | ||
| Imagine you are working on a new feature (or fix) for TARDIS, you have opened a pull request and the | ||
| reference data tests are failing in the testing pipeline. This could happen for many reasons: | ||
|
|
||
| #. Go to your ``tardis`` directory. Make sure you are working on the correct branch and generate new reference data using: | ||
| A. There's a problem in your code. | ||
| B. Your code is OK, but the reference data is outdated. | ||
| C. The pipeline is broken. | ||
|
|
||
| .. code-block:: none | ||
| If you think your could be dealing with scenario B, then: | ||
|
|
||
| python setup.py test --args="--tardis-refdata=<path to refdata repo on the right branch> --generate-reference" | ||
| #. Write ``/azp run compare-refdata`` in a comment on your PR. | ||
| #. Analyze the results and discuss if the reference data effectively requires an update. | ||
| #. Update the reference data by writing ``/azp run update-refdata`` on a new comment. | ||
|
|
||
| #. Re-run the tests to make sure it does not fail using: | ||
| .. note:: | ||
|
|
||
| .. code-block:: none | ||
| - If you don't have enough privileges to run the pipelines, tag a TARDIS developer capable of doing so. | ||
| - If any of these two pipelines fail, please tag the :ref:`CI/CD responsible <team>`. | ||
|
|
||
| python setup.py test --args="--tardis-refdata=<path to refdata repo on the right branch>" | ||
| If everything went well, the reference data will have been updated by the TARDIS bot and the commit | ||
| message should include the pull request number that triggered the update. | ||
|
|
||
| #. Go back to your ``tardis-refdata`` folder, and run a Jupyter Notebook session inside the ``notebook`` folder. | ||
| ================ | ||
| Manual Procedure | ||
| ================ | ||
|
|
||
| #. Open the ``ref_data_compare.ipynb`` file notebook and look for the cell with the following code: | ||
| The manual procedure is documented for debugging purposes and should not be used in general. | ||
|
|
||
| .. code-block:: none | ||
| comparer = ReferenceComparer(ref2_hash='upstream/pr/24') | ||
| Replace '24' for the number of the last merged pull request showed `here <https://github.com/tardis-sn/tardis-refdata/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed>`_. | ||
|
|
||
| #. Run all cells and check if there are any unexpected changes in the updated reference data respect the previous one. | ||
|
|
||
| #. Commit the changes made to the reference data and open a PR on ``tardis-refdata``. | ||
|
|
||
| #. Switch back to the ``tardis`` directory. Open the ``.travis.yml`` file and change the following lines: | ||
|
|
||
| .. code-block:: none | ||
| - if [[ $TEST_MODE == 'spectrum' ]]; then git fetch origin pull/<your tardis-refdata PR number; not the TARDIS PR number>/head:<some descriptive name>; fi | ||
| - if [[ $TEST_MODE == 'spectrum' ]]; then git checkout <some descriptive name>; fi | ||
| #. Commit the ``.travis.yml`` to your pull request. | ||
|
|
||
| #. Make sure that your TARDIS pull request now passes all tests on Travis-CI. | ||
|
|
||
| #. Ask for review for your PR on ``tardis-refdata`` and wait for merge. | ||
|
|
||
| #. Then change the ``.travis.yml`` in ``tardis`` directory to: | ||
|
|
||
| .. code-block:: none | ||
| - if [[ $TEST_MODE == 'spectrum' ]]; then git fetch origin; fi | ||
| - if [[ $TEST_MODE == 'spectrum' ]]; then git checkout origin/master; fi | ||
| #. Ensure TARDIS pull request passes Travis-CI again and ping someone to merge your PR to the TARDIS master branch. | ||
|
|
||
|
|
||
| Congratulations! You have updated TARDIS to be better. Have a beer and steak (or Tofu if you are vegetarian/vegan). | ||
|
|
||
|
|
||
| Troubleshooting | ||
| ############### | ||
|
|
||
| * Unable to generate reference data | ||
| * If generating fails due to an inability to open ``chianti_He.h5``, make sure that you have activated your `tardis` conda environment and that ``git-lfs`` is installed. Fetch and pull the files from GitHub as explained in step 4. | ||
|
|
||
| * Error when running ``comparer = ReferenceComparer(ref2_hash='upstream/pr/XX')`` on the comparer notebook: ``No such file or directory: '.../unit_test_data.h5'`` | ||
| * If notebook file is unable to find the file ``unit_test_data.h5``, make sure you have correctly set your upstream as explained in :ref:`reviewing-and-helping-with-pr`. | ||
| #. Activate the ``tardis`` environment. | ||
| #. Fork and clone the ``tardis-refdata`` repository. | ||
| #. Follow the instructions at the top of the notebook ``tardis-refdata/notebooks/ref_data_compare.ipynb``. | ||
| #. Go to your local ``tardis`` repository and make sure you are working on the branch you want to generate new reference data from. | ||
| #. Generate new reference data with ``pytest tardis --refdata=/path/to/tardis-refdata --generate-reference``. | ||
| #. Run the ``ref_data_compare.ipynb`` notebook and check the results. | ||
| #. Make a new branch in ``tardis-refdata``, push your new reference data and open a pull request. |