Meeting with Ouranos and CRIM: Dec 2020

[agstephens]

Proposed agenda for meeting - 03/12/2020

1. Minutes and actions from previous meeting

Previous minutes: #71

Previous actions:

• AS: set up a monthly meeting in this Thursday slot.

• AS: to discuss code management procedures with the 34e team

• TJS: Add a Pull Request template for CLISOPS: Add a Pull Request template for CLISOPS clisops#70

• AS: liaise with team to get a new release published this month - in progress

• AS: put a "what we have learnt about dask" item on the next meeting agenda

• AS: discuss with 34e team how we add support for static typing: Add some static type support clisops#67 - in progress

2. Project update: c3s_34e [Ag]

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

4. c3s_34e experience with Dask [Ag/Elle]

• Update on our dask/xarray tests:
  https://docs.google.com/presentation/d/1jMOhTNh96RmPhVFbR0F-yK9g2ScfBXyavK0sAcv_HTQ/edit?usp=sharing

5. Proposed adaptive masking approach for regridding [Martin]

• Proposed approach handle masks in regridding (in 34e and xESMF)

6. Discussion:

• Subsetting issue (Elle)
• release procedure
• managing test datasets
• other activities

7. Agree next steps and next meeting

[huard]

3.1 xESMF 0.5.0 release and integration timeline into clisops

[agstephens]

C3S_34e meeting with Canadian Partners (Ouranos, CRIM) - 03/12/2020

#74

Attendees

Ag Stephens, CEDA
Elle Smith, CEDA
Martin Schupfner, DKRZ
Carsten Ehbrecht, DKRZ
David Huard, Ouranos
Trevor James Smith, Ouranos
Tom Landry, CRIM

Actions

• AS: discuss with 34e team how we add support for static typing: Add some static type support clisops#67 - in progress
• AS: respond to Tom's message about a formal collaboration - in progress

Minutes

1. Review of minutes and actions

2. Project update: c3s_34e [Ag]

• attempt to initiate/set up a test data archive based on an idea from Pascal (xclim, adopted from xarray and seaborn)
• cf-python
  • Regridding: http://www.met.reading.ac.uk/~swsheaps/ncas_data_tools/pdf/cf-python_regridding.pdf
  • cf strictness is mostly not the reality, xarray deals better with "real world" data
• CMIP6 data should be provided through the data set by December, realistically 1-2 months retardation
  • not enough tests are done yet due to time-consuming processes (release procedure, ...)
  • Carsten did work related to provenance: https://github.com/cehbrecht/provenance-demo
    • ESMValTool uses the same provenance tool/library/standard
    • Difficult to set up a scheme that would be appropriate for our use case as it is a very flexible library
    • Not the right solution to compare the workflows, rather an intensive log file that only allows the reproduction

3. Project updates: relevant Ouranos/CRIM projects [Trevor/David/Tom]
   Trevor:

• changes to Travis-CI have caused some issues
• we are looking forward to a new release of clisops - to bring in the rtree dependency change that is now on master
  David:
• xESMF developments:
  • compute spatial averages over polygon
  • timeline for xESMF to include xESMF as a dependency in clisops: March (Prototype), June (Production)
• Changes in Travis CI have to be worked on
• Pascal committed changes that need to be respected by a clisops release
  Tom
• New funding from the Canadian Government
  • have had an invite to submit proposals for geospatial architectures
  • open point of collaboration
• use of Weaver: Carsten and Francis have demonstrated how Weaver can expose the "rook" WPS via a RESTful interface.

4. c3s_34e experience with Dask [Ag/Elle]

• Elle presents the update on our dask/xarray tests:
  https://docs.google.com/presentation/d/1jMOhTNh96RmPhVFbR0F-yK9g2ScfBXyavK0sAcv_HTQ/edit?usp=sharing

5. Proposed adaptive masking approach for regridding [Martin]

• Proposed approach handle masks in regridding (in 34e and xESMF)
  • Adaptive masking with threshold
  • further collaboration with Stéphane
  • Jupyter notebooks to be shared via Github

6. Discussion:

• Subsetting issue (Elle)
  • lines 863++ of subset.py needs discussion
  • causees problems when subsetting longitude
  • fix might be related to reversing the axis
• release procedure
  • addressed above
• managing test datasets
  • addressed above
• other activities
  • addressed above

7. Agree on next steps and next meeting

• Next meeting: Thursday 21st January