Add Parthenon frontend

[pgrete]

PR Summary

Add yt frontend for codes based on the Parthenon AMR framework (https://github.com/parthenon-hpc-lab/parthenon) including AthenaPK (https://github.com/parthenon-hpc-lab/athenapk), Phoebus (https://github.com/lanl/phoebus) or KHARMA (https://github.com/AFD-Illinois/kharma)
Note that the mesh structure in Parthenon is heavily derived from Athena++, so that's why we used the Athena++ frontend as baseline.

We've been using this frontend for quite some time now (without any recent major changes) so we consider it stable enough to open it for review and merge to yt proper.

At this point I'm looking for

• general feedback
• feedback on how to generally handle/differentiate between Parthenon (as framework) and downstream code related functions/fields/... (the current version of this frontend is geared towards AthenaPK and I'd like to abstract this)
• (assuming a test dataset is desired), how to upload/handle/provide the dataset

PR Checklist

• New features are documented, with docstrings and narrative docs
• Adds a test for any bugs fixed. Adds tests for new features.

[matthewturk]

This is awesome! Thank you for contributing it. I'll provide a review ASAP.

[neutrinoceros]

Congratz @pgrete ! I'll make a first pass for QA and will try to focus on stuff that isn't already reported by linters (see pre-commit.ci's report).

[neutrinoceros]

• feedback on how to generally handle/differentiate between Parthenon (as framework) and downstream code related functions/fields/... (the current version of this frontend is geared towards AthenaPK and I'd like to abstract this)

I think we have one example of a framework-as-a-dataformat already: the boxlib frontend. In my experience, t's indeed the one for which automatic determination of exact format is hardest to implement and maintain, and the reason why we added a hint keyword argument to yt.load. I hope we'll be able to have a cleaner distinction between formats with this frontend but if the binary data format is to be identical I think one practical approach is to reserve some header in binary files to encode some unique key (like the name of the code) that maps to a known format. Otherwise it may be impossible to guess.

At this point, and I'm not saying that to discourage this pull request, you may want to consider distributing the frontend as an extension package, which can help if frequent contributions are expected and you want to be free from our relatively slow release cycle.

• (assuming a test dataset is desired), how to upload/handle/provide the dataset

yes that would be a necessary step for us to merge this PR.
The process requires a companion PR to our website repo. The most recent example of such a PR would be yt-project/website#115

[pgrete]

@forrestglines what's the intention behind this logic, i.e., why creating the gamma_attrs rathern than checking if AdiabaticIndex is already in Params?
(same below for He_mass_fraction)

[forrestglines]

It's due to the actual parameter being Hydro/AdiabaticIndex in AthenaPK which will be different for the other Parthenon codes.

If this logic is too opaque, we should instead look for Hydro/AdiabaticIndex for AthenaPK and implement the adiabatic index as needed for the other codes.

[forrestglines]

As it's supposed to work right now, an adiabatic index passed in via gamma= will take priority over an adiabatic index defined in the parameters of the Parthenon simulation, then 5./3. if nothing else is found.

[forrestglines]

The He_mass_fraction piece isn't working since at some point we stopped outputing He_mass_fraction and started including mbar_over_kb instead. We could back out He_mass_fraction from mbar_over_kb instead.

[forrestglines]

I'm trying to fix reading in He_mass_fraction from mbar_over_kb but I'm getting nonsensical results. I'll have to look at it later with fresh eyes. I'll push what I'm working with in case someone can see what I'm doing wrong.

[pgrete]

• (assuming a test dataset is desired), how to upload/handle/provide the dataset

yes that would be a necessary step for us to merge this PR. The process requires a companion PR to our website repo. The most recent example of such a PR would be yt-project/website#115

Regarding the dataset, am I assuming correctly that the hdf5 file should not use (default) compression, i.e., deflate?

[matthewturk]

@pgrete I think as long as it doesn't require hdf5plugin it should be OK to use deflate.

[neutrinoceros]

I think that the reason CI is now failing is that #4344 was merged in between runs. Add test_outputs.py to the list of files that pytest should ignore to resolve this.

[pgrete]

I think that the reason CI is now failing is that #4344 was merged in between runs. Add test_outputs.py to the list of files that pytest should ignore to resolve this.

I'm currently figuring out some useful tests to add (with the appropriate data file).

[pgrete]

Looks like I'm stuck.
How can I actually run tests?

I first thought that the test are answer tests so that I should call them using a class (like TestTipsy described here https://yt-project.org/doc/developing/testing.html#how-to-run-the-answer-tests) but then I realized that the current test_outputs.py does not define a class (or I don't see where it does) -- similar to other test_outputs.py of other frontends.

Then I tried pytest (after reinstalling the project to fix dependencies) and only got

$ pytest yt/frontends/parthenon/tests
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.10.9, pytest-7.2.2, pluggy-1.0.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/pgrete/src/yt, configfile: pyproject.toml
plugins: typeguard-2.13.3, anyio-3.6.2
collected 0 items                                                                                                                                                                            

=================================================================================== no tests ran in 0.01s ====================================================================================

so what am I missing?

[pgrete]

I'm a little stuck here.
I realized that the failing test was run with py38 so I thought that my python environment may be too new (also wrt the nose PR for python >3.10).

So I installed a clean env with python 3.8 and the "full" version of yt pip install -e .[full]

Trying the test now results in:

$ pytest yt/frontends/parthenon/tests/test_outputs.py 
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.8.16, pytest-7.2.2, pluggy-1.0.0 -- /home/pgrete/miniconda3/envs/py3.8/bin/python
cachedir: .pytest_cache
rootdir: /home/pgrete/src/yt, configfile: pyproject.toml
collected 0 items / 1 error                                                                                                                                                                  

=========================================================================================== ERRORS ===========================================================================================
_______________________________________________________________ ERROR collecting yt/frontends/parthenon/tests/test_outputs.py ________________________________________________________________
yt/frontends/parthenon/tests/test_outputs.py:11: in <module>
    from yt.utilities.answer_testing.framework import (
yt/utilities/answer_testing/framework.py:24: in <module>
    from nose.plugins import Plugin
../../miniconda3/envs/py3.8/lib/python3.8/site-packages/nose/__init__.py:1: in <module>
    from nose.core import collector, main, run, run_exit, runmodule
../../miniconda3/envs/py3.8/lib/python3.8/site-packages/nose/core.py:12: in <module>
    from nose.loader import defaultTestLoader
../../miniconda3/envs/py3.8/lib/python3.8/site-packages/nose/loader.py:21: in <module>
    from nose.importer import Importer, add_path, remove_path
../../miniconda3/envs/py3.8/lib/python3.8/site-packages/nose/importer.py:12: in <module>
    from imp import find_module, load_module, acquire_lock, release_lock
../../miniconda3/envs/py3.8/lib/python3.8/imp.py:31: in <module>
    warnings.warn("the imp module is deprecated in favour of importlib; "
E   DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
================================================================================== short test summary info ===================================================================================
ERROR yt/frontends/parthenon/tests/test_outputs.py - DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================================== 1 error in 0.29s ======================================================================================

What am I missing?

[neutrinoceros]

Sorry for the late reply. GenericArrayTest is a subclass of AnswerTestingTest and as such, it cannot be run with pytest yet (we are still in the middle of the migration process).
As a result, your tests can only be run with Python 3.8 or 3.9 (since nose isn't compatible with Python 3.10 and later).

[pgrete]

So I'd call python tests/nose_runner.py (as that's the command being run by the py38 GitHub check)?
And, if yes, can I just run the Parthenon frontend or do I need to run all tests?

[neutrinoceros]

test/nose_runner.py is a connivence script. I'm sure it's possible to run only a fraction of tests.
I'm actually very confused seeing the docs recommend using pytest for that
https://yt-project.org/docs/dev/developing/testing.html#how-to-run-the-answer-tests

so clearly I'm not the expert here. Does the documentation help ? if not, maybe @Xarthisius is would be of better guidance.

[pgrete]

I figured it out (for reference nosetests -v --with-answer-testing --local --local-dir $HOME/src/yt-test-data --answer-store --answer-name=local-parthenon yt.frontends.parthenon). Going back to the old doc helped.

I just added a plain Parthenon 2D test (which of course will fail given that the data is not available online).

What do the next steps look like in general (in terms of when to upload/open a PR for the reference files) and the combined review?

With respect to this PR, I plan to make the following two additions:

• a 3D example (of the 2D type above) but with dx != dy != dz
• a 3D AthenaPK example (incl changes to work with a code hint) with support for units

Do you recommend any other additional tests?
Is sth else required to complete this PR?

[neutrinoceros]

What do the next steps look like in general (in terms of when to upload/open a PR for the reference files) and the combined review?

AFAICT it seems mature enough that submitting datasets for upload now wouldn't be inappropriate.

Do you recommend any other additional tests?

I think that frontend tests should be as focused as possible on things that might go wrong specifically with the data format at hand. Tests exposing problems encountered during development are also very welcome, but don't feel like you need to be 100% exhaustive.

Is sth else required to complete this PR?

Eventually we'll want to have 2 approvals from maintainers, but you're close to getting mine already, since my initial comments were all addressed ! I'll need to review again, and I'm ready to do so as soon as needed. Feel free to ping me if you'd rather get my review before dataset are uploaded, otherwise I'll just wait for tests to go green !

[pgrete]

In the process of adding data files of two other codes I'm currently wondering
a) if (and how) yt might handle data from general relativistic simulations, i.e., data that might carry a metric around
b) if (and how) multi material simulation data is supported (e.g., different equations of state for different fields)

[matthewturk]

@pgrete For a), this should be feasible (@chrishavlin has done some on-the-same-line-of-thought work with integrating pressure to get spatial coordinates) but might need some collaborative thinking, which I'm up for. For b) this should be feasible by even in a worst case using multiple "field types". Either way, both are high priorities, so let's try!

[pgrete]

I updated quite a bit of the field logic (moving things to known_other_fields above) and am now wondering if the registered/resolved fields are potentially already available here.
This would simplify logic here (and at several places below) wrt to just checking if ("gas", "momentum_density_x") is known rather than lookups for our own names in self.field_list.

[forrestglines]

Just pushed some quick changes so that coordinate ordering for cylindrical and spherical coordinates follows Athena++'s conventions, which AthenaPK will inherit.

(This is the last functional addition we had planned for this PR)

[neutrinoceros]

@forrestglines I suspect this line is the only reason for your latest commit, so I should point out that you should just use Geometry.POLAR here, so you don't need to specify axis_order at all.

[forrestglines]

Thanks @neutrinoceros that would certainly allow a bit of code cleanup. Unfortunately changing to Geometry.POLAR is breaking my scripts which use fixed resolution buffers. For example, this code to make a slice plot works with both Geometry.POLAR and the current solution with Geometry.CYLINDRICAL:

ds = yt.load(filename)

slc = yt.SlicePlot(ds, "z", ("gas", "density"))

but this code below only works with my current solution with Geometry.CYLINDRICAL:

ds = yt.load(filename)

slc = yt.SlicePlot(ds, "z", ("gas", "density"))
frb = slc.data_source.to_frb(width=(4,"cm"),resolution=512,center=(0,0))
frb["r"] 

with an error being thrown by pixelize_cylinder

File ~/installs-chicoma/miniconda3/envs/py39/lib/python3.9/site-packages/yt/visualization/fixed_resolution.py:600, in CylindricalFixedResolutionBuffer._generate_image_and_mask(self, item)
    597 @override
    598 def _generate_image_and_mask(self, item) -> None:
    599     buff = np.zeros(self.buff_size, dtype="f8")
--> 600     mask = pixelize_cylinder(
    601         buff,
    602         self.data_source["r"],
    603         self.data_source["dr"],
    604         self.data_source["theta"],
    605         self.data_source["dtheta"],
    606         self.data_source[item].astype("float64"),
    607         self.radius,
    608         return_mask=True,
    609     )
    610     self.data[item] = ImageArray(
    611         buff, units=self.data_source[item].units, info=self._get_info(item)
    612     )
    613     self.mask[item] = mask

File ~/installs-chicoma/miniconda3/envs/py39/lib/python3.9/site-packages/yt/utilities/lib/pixelization_routines.pyx:552, in yt.utilities.lib.pixelization_routines.pixelize_cylinder()

ValueError: Buffer has wrong number of dimensions (expected 2, got 1)

Suggestions? Is this something to be fixed in the frontend or in yt? I can provide the data file for this cylindrical dataset. The dataset will be included with our tests

[neutrinoceros]

Hard to tell for sure without the data file, but I can have a look if you can share it.

[forrestglines]

Thanks! I've pushed the small changes to switch from Geometry.CYLINDRICAL to Geometry.POLAR.

Here's the file I was testing on (with an .txt extension due to github's limitations)

disk_cyl.prim.00000.phdf.txt

[neutrinoceros]

thank you ! I can reproduce locally, which is a great start. I'll try to debug this today.

[neutrinoceros]

Actually it's a yt bug, I can also reproduce it with AMRVAC data

import yt

ds = yt.load_sample("bw_polar_2d")
slc = yt.SlicePlot(ds, "z", ("gas", "density"))
frb = slc.data_source.to_frb(width=(4, "cm"), resolution=512, center=(0, 0))
frb["r"]

I'll open a dedicated issue.

[forrestglines]

I saw issue #4790, the bug is not related to 2D datasets. I also explored this with a 3D dataset and got the same error.

disk_cyl.3d.prim.00000.phdf.txt

[forrestglines]

Fantastic that we have a bug fix started with this so soon. While this did eliminate the error with FRB, the polar coordinates are still not behaving like the cylindrical coordinates. In some comparison plots between Athena++ and AthenaPK using FRB's of a slice generated with the same code, I get plots looking like this:

My Parthenon frontend (using Geometry.POLAR with your new bugfix) is only getting one quadrant while the Athena++ frontend (using Geometry.CYLINDRICAL) is getting the full disk as expected.

Let me work on a better reproducer and perhaps one using an existing frontend in YT. I'm not convinced either way that this is now the Parthenon frontend's bug or Polar's bug but it's still a problem that changing from cylindrical to polar changes my plots.

[chrishavlin]

just FYI, i hit the button for the tests to run, but you'll likely get unrelated failures after your merge with main (e.g., #4785 ).

[forrestglines]

I've pushed @neutrinoceros 's bug fix to this PR although I'm still working on Polar. You can advise on how you'd like me to fix the git history later on if the merges get messy.

[Xarthisius]

@yt-fido test this please

[Xarthisius]

This broken doc build doesn't look like our doing, should we just merge main? (I'm more asking reviewers if a simple merge from main would be better than another rebase)

Don't worry about it. It's not blocking this PR. I re-triggered docs build and it will likely pass this time.

[neutrinoceros]

Not sure that's a deterministic error. Let me try again:
@yt-fido test this please

[neutrinoceros]

Docs built alright but this time regular tests failed (also flaky)
@yt-fido test this please

[neutrinoceros]

All green now ! I hope I'll find time to review this week, but with the imminent release of a major numpy release candidate I may not. Thank you for your patience !

[chrishavlin]

Generally looks good to me! Left a number of questions and comments, but nothing too critical.

[neutrinoceros]

Sorry for taking so long to get my review in. I note that there's also another, mostly unaddressed (from what I can see) review from @chrishavlin. Both need to be addressed before we can merge this.
Thank you for your patience !

[pgrete]

Alright, I think I've addressed all review comments (@neutrinoceros @chrishavlin ) pending potential replies by you to the unresolved ones.
I also updated the branch to current main.

From my point of view this should be ready to go (assuming all tests pass).

What are the chances this PR will make it to the 4.4 release?

[jzuhone]

@pgrete I will also take a look at this tomorrow and I think that we should make this a priority for the 4.4 release.

[neutrinoceros]

I would normally oppose stacking more PRs (especially large ones) to a release's milestone at the last minute, but this one has been waiting for so long that it makes sense to try and get it in.

[pgrete]

I would normally oppose stacking more PRs (especially large ones) to a release's milestone at the last minute, but this one has been waiting for so long that it makes sense to try and get it in.

That'd be awesome. We've been using this branch for quite some time now without issues and given that it doesn't change anything in the yt core, I think it should not pose any potential harm to the release code base as a whole.

[neutrinoceros]

Indeed. Though, if for any reason we cannot make it, you guys could also consider packaging this frontend as an extension

[chrishavlin]

Looks great from my perspective, thanks for addressing my comments @pgrete (edit: apologies, tagged the wrong person at first!!)

[chrishavlin]

also +1 for making this a priority for 4.4.

[pgrete]

"@pgrete dismissed @chrishavlin 's stale review via dd42468" <- I don't know where/how that came from. I just pushed a one line change removing a "Todo comment" I already addressed yesterday.

[BenWibking]

"@pgrete dismissed @chrishavlin 's stale review via dd42468" <- I don't know where/how that came from. I just pushed a one line change removing a "Todo comment" I already addressed yesterday.

This can be caused by the branch protection settings: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches#require-pull-request-reviews-before-merging.

Optionally, you can choose to dismiss stale pull request approvals when commits are pushed that affect the diff in the pull request. GitHub records the state of the diff at the point when a pull request is approved. This state represents the set of changes that the reviewer approved. If the diff changes from this state (for example, because a contributor pushes new changes to the pull request branch or clicks Update branch, or because a related pull request is merged into the target branch), the approving review is dismissed as stale, and the pull request cannot be merged until someone approves the work again.

[chrishavlin]

@pgrete ya, no worries, that's cause of yt's repo config. I'll hit approve again, and you can consider this my blanket approval if it gets dismissed again :)

[neutrinoceros]

Alright, I think all my comments have been addressed, and with 2 other approvals I feel like there's no reason to hold this back any longer, so let's get it in ! Thank you @pgrete and @forrestglines for your dedication and patience with the process. We'll be trying to get yt 4.4 out the door very shortly, so stay tuned !

[pgrete]

Thanks for pulling this in on short notice! I'm very happy to update our downstream notes to pull the main (and future yt 4.4) version rather than a custom branch.