BUG: Fix parsing of MANGA cube

[pllim]

Description

This pull request is to fix loading MANGA cube in Cubeviz. Since Brian C said it works on Jdaviz 3.10 series, probably no need to backport.

The problem is the IVAR extension (loaded to uncert) has no BUNIT, so it falls back to u.count and now Cubeviz complains about that unit. This patch forces uncert cube to follow flux cube unit if u.count (the fallback unit) is detected.

Question: Is this assumption safe? IVAR is inverse variance, so theoretically it should be inversion of flux unit squared, right? But if we do that, will spectral extraction fails because the uncert unit then still does not match flux unit?

Change log entry

• Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
  list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
  should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• Do the proposed changes follow the STScI Style Guides?
• Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• Did the CI pass? If not, are the failures related?
• Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
• After merge, any internal documentations need updating (e.g., JIRA, Innerspace)? 🐱

[codecov]

Codecov Report

Attention: Patch coverage is 58.33333% with 5 lines in your changes missing coverage. Please review.

Project coverage is 88.89%. Comparing base (3de1b08) to head (eacd85d).
Report is 144 commits behind head on main.

Files with missing lines	Patch %	Lines
jdaviz/configs/cubeviz/plugins/parsers.py	58.33%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3115      +/-   ##
==========================================
- Coverage   88.91%   88.89%   -0.02%     
==========================================
  Files         111      111              
  Lines       17365    17372       +7     
==========================================
+ Hits        15440    15443       +3     
- Misses       1925     1929       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pllim]

If we cannot find a safe assumption here, the correct thing to do is indeed throwing exception because the given data does not have BUNIT defined in its IVAR extension. The data provider should have clearly defined a BUNIT if the data has physical unit.

[pllim]

Cami said to not load the uncert cube at all in this situation. I will push new commit.

[havok2063]

I think it's a fine assumption for MaNGA, but not sure about in general. You can have a look at the MaNGA docs for cubes at https://www.sdss4.org/dr17/manga/manga-data/data-model/#3-DReductionPipelineOutput and https://data.sdss.org/datamodel/files/MANGA_SPECTRO_REDUX/DRPVER/PLATE4/stack/manga-CUBE.html for more info on the structure.

We can't change the MaNGA data now, and I think it's too much to ask users to edit the MaNGA data into some custom object to get it to load into Jdaviz. I don't believe BUNIT is a required FITS header keyword for extensions.

I don't like the idea of not loading the uncertainty viewer, and presumably the spectral view won't load an extracted spectrum as well. This would effectively make MaNGA data very confusing to use in Cubeviz without lots of context provided back to the user by Jdaviz on how to load. There should be some reasonable defaults and fall backs in place to get the data to load on input, when BUNIT is not supplied.

[camipacifici]

Thank you, Brian. I was hoping you would chime in. Not loading the uncertainty extension is just a patch until a better fix can be in.
I have not looked at the documentation yet, but sounds like the changes should go in the specutils parser for MaNGA. What do you think?

[pllim]

We have no special parser for MANGA. It loads as any other multi-extension FITS would.

[pllim]

To start having special cases is a slippery slope. Do you really need uncert cube for MANGA analysis?

[havok2063]

We do have specific data loaders for MaNGA in specutils. See https://github.com/astropy/specutils/blob/main/specutils/io/default_loaders/manga.py. I haven't been following all the changes to the spectral extraction plugin, so if someone can tell me what needs to be updated, I'm happy to update the parsers for MaNGA.

I'm guessing it's adding a unit here? https://github.com/astropy/specutils/blob/f5e708c3efd0ded0dd32b1bc2fc79e7b187a240d/specutils/io/default_loaders/manga.py#L117

[pllim]

I actually don't think it is going through Spectrum1D.read at all. I traced it to here:

jdaviz/jdaviz/configs/cubeviz/plugins/parsers.py

Lines 219 to 221 in 3de1b08

	def _parse_hdulist(app, hdulist, file_name=None,
	flux_viewer_reference_name=None,
	uncert_viewer_reference_name=None):

Maybe because we wanted the uncert to be in its own viewer, and hence it has to be a separate Spectrum1D object, rather than being attached to the flux Spectrum1D object? (I was not on this team when it was written originally.)

I think your ask to go through specutils reader might be a bigger refactor to the whole parser.

[pllim]

Even if we force the logic to go through Cubeviz Spectrum1D object parser, it is problematic because it blindly assumes flux unit (which is wrong for inverse variance):

jdaviz/jdaviz/configs/cubeviz/plugins/parsers.py

Line 420 in 3de1b08

flux = u.Quantity(val.array, file_obj.flux.unit)

[pllim]

I did confirmed 2 things loading the cube using code on main without this patch (also this patch won't fix that problem anyway):

1. uncert does load and spectral extraction works if you use sp = Spectrum1D.read(filename) first and then pass that into Cubeviz like this: cubeviz.load_data(sp) (so, don't pass in filename directly)
2. But, the uncert unit in Cubeviz is wrong.

[pllim]

I think I have a better fix. Closing this.

[pllim]

Please see #3119