Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use tardis-refdata from Azure mirror (CI/CD) #1304

Merged
merged 54 commits into from
Sep 25, 2020

Conversation

epassaro
Copy link
Member

@epassaro epassaro commented Sep 16, 2020

Description

  • Make a mirror of tardis-sn/tardis-refdata repository in Azure Repos.
  • Use this mirror in simple_test_framework.yml pipeline
  • Use GitHub LFS again to download atomic data in tardis/data/atomic_data_repo.yml
  • Create a default template for TARDIS pipelines with common steps, then move most of the simple_test_framework.ym; pipeline steps to this template.
  • Update TARDIS documentation

Next thing to do is take care of NumPy/macOS issue.

Motivation and Context

Recently we ran out of GitHub LFS bandwidth. To avoid this problem in the future we mirrored the reference data repository to Azure Repos (no LFS limits).

Also, all the TARDIS pipeline share common steps, such as configuring conda and installing the TARDIS environment. The adoption of a template allows us to make pipelines easier to create and maintain.

How Has This Been Tested?

Tests on Azure.

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have assigned/requested two reviewers for this pull request.

@@ -27,43 +24,28 @@ jobs:
maxParallel: 2

steps:
- checkout: self

- checkout: git://TARDIS/tardis-refdata
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we specify branches here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. According to the documentation:

- checkout: git://MyProject/MyRepo@features/tools # checks out the features/tools branch
- checkout: git://MyProject/MyRepo@refs/heads/features/tools # also checks out the features/tools branch
- checkout: git://MyProject/MyRepo@refs/tags/MyTag # checks out the commit referenced by MyTag.

Link: https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#checking-out-a-specific-ref

@codecov
Copy link

codecov bot commented Sep 16, 2020

Codecov Report

Merging #1304 into master will increase coverage by 0.95%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1304      +/-   ##
==========================================
+ Coverage   80.54%   81.50%   +0.95%     
==========================================
  Files          41       46       +5     
  Lines        3423     3784     +361     
==========================================
+ Hits         2757     3084     +327     
- Misses        666      700      +34     
Impacted Files Coverage Δ
tardis/scripts/cmfgen2tardis.py
tardis/io/atom_data/util.py
tardis/plasma/properties/general.py
tardis/montecarlo/base.py
tardis/io/decay.py
tardis/plasma/properties/property_collections.py
tardis/plasma/properties/partition_function.py
tardis/plasma/base.py
tardis/plasma/properties/ion_population.py
tardis/io/util.py
... and 77 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 862025e...fcf67e4. Read the comment docs.

@epassaro epassaro marked this pull request as draft September 16, 2020 23:15
@epassaro epassaro changed the title Use tardis-refdata from Azure mirror Use tardis-refdata from Azure mirror (CI/CD) Sep 17, 2020
Copy link
Member

@wkerzendorf wkerzendorf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks very good.

@wkerzendorf wkerzendorf merged commit c716da6 into tardis-sn:master Sep 25, 2020
atharva-2001 pushed a commit to atharva-2001/tardis that referenced this pull request Oct 1, 2021
* Use tardis-refdata from Azure mirror

* Checkout `self` repo

* Debugging pipeline

* Debugging pipeline

* Debugging pipeline

* Fetch just necessary refdata

* (revert) Fetch just necessary refdata

* Download atom data from Azure

* Download atom data from GitHub LFS

* Debugging pipeline

* Debugging pipeline

* Debugging pipeline

* Debugging pipeline

* Debugging pipeline

* Re-writing CI doc section (1)

* Moving to a template schema

* Moving to a template schema (2)

* Moving to a template schema (3)

* Moving to a template schema (4)

* Moving to a template schema (5)

* Moving to a template schema (6)

* Moving to a template schema (7)

* Moving to a template schema (8)

* Moving to a template schema (9)

* Moving to a template schema (10)

* Moving to a template schema (11)

* Moving to a template schema (12)

* Moving to a template schema (13)

* Moving to a template schema (15)

* Moving to a template schema (16)

* Moving to a template schema (16)

* Moving to a template schema (17)

* Moving to a template schema (18)

* Moving to a template schema (19)

* Moving to a template schema (20)

* Moving to a template schema (21)

* Moving to a template schema (21)

* Moving to a template schema (22)

* Moving to a template schema (23)

* Moving to a template schema (24)

* Moving to a template schema (25)

* Moving to a template schema (26)

* Moving to a template schema (27)

* Moving to a template schema (28)

* Finished template [skip-ci]

* Minor changes to step names

* Minor changes to step names

* Complete documentation

* Use custom variables for repo paths

* Minor changes to documentation

* Minor changes

* More documentation on pipelines

* Minor changes

* Minor changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants