Strange key added to map.meta '\x00\x00\x00\x00\x00\x00\x00\x00'

[amunozj]

Description

Trying to convert meta to fits_header using Map.fits_header fails because a very odd key, but this key is not in the original header. It is added somewhere in the construction of the map.

Expected behavior

For the meta not to have this key

Actual behavior

Original header:

SIMPLE  =                    T  / FITS STANDARD                                 
BITPIX  =                  -32 / FITS BITS/PIXEL                                
NAXIS   =                    2  / NUMBER OF AXES                                
NAXIS1  =                  512  /                                               
NAXIS2  =                  512  /                                               
BUNIT   = 'GAUSS'                                                               
ORIGIN  = 'MT. WILSON'                                                          
TELESCOP= '150 ft'                                                              
OBSERVER= 'SBEW RET'                                                            
OBJECT  = 'SUN'                                                                 
DATA-TYP= 'MAGNETOGRAM'                                                         
DATE-OBS= '1981.02.01_18:08:00_TAI'                                             
TAVG-UT =              19.2890  / Avg UT during obs                             
CARROT  =            1704.7341  / Carr rotation                                 
CENLON  =              95.7238  / Carr long of cent med                         
B0      =            -0.106516  / Sun tilt(rad)                                 
X0      =             256.5000  / image x center                                
Y0      =             256.5000  / image y center                                
R0      =             256.0000  / image radius                                  
LAMBDA  =              5250.20  / Wavelength(Angstroms)                         
GV      =             3.000000  / Lande g-factor                                
DXB_IMG =               3.8035  / image x-arcsec/pixel                          
DYB_IMG =               3.8035  / image y-arcsec/pixel                          
DXB_OBS =               9.7679  / obs avg x-arcsec/pixel                        
DYB_OBS =              12.6425  / obs avg y-arcsec/pixel                        
REMARK1 = 'TSAFARG   .M'                                                        
REMARK2 =                                                                       
PSEPMA  =             0.000000  / port separation in milliangstroms             
ICHK    =                    0  / int. ratio flag, 0=no, 1=yes, 2=1./ratio      
MRKMOD  =                  256  / hrdwre 0=datex,256=ael,1280-1283=fm,1284=24ch 

CUNIT1  = 'arcsec  '                                                            
CUNIT2  = 'arcsec  '                                                            
CDELT1  =               3.8035                                                  
CDELT2  =               3.8035                                                  
CRVAL1  =                  0.0                                                  
CRVAL2  =                  0.0                                                  
RSUN_OBS=              973.696                                                  
CROTA2  =                  0.0                                                  
CRPIX1  =                256.0                                                  
CRPIX2  =                256.0                                                  
T_OBS   = '1981.02.01_18:08:00_TAI'                                             
DATE_OBS= '1981.02.01_18:08:00_TAI'                                             
RSUN_REF=            696000000                                                  
CTYPE1  = 'HPLN-TAN'                                                            
CTYPE2  = 'HPLT-TAN'  

Meta:

MetaDict([('simple', True),
          ('bitpix', -32),
          ('naxis', 2),
          ('naxis1', 512),
          ('naxis2', 512),
          ('bunit', 'GAUSS'),
          ('origin', 'MT. WILSON'),
          ('telescop', '150 ft'),
          ('observer', 'SBEW RET'),
          ('object', 'SUN'),
          ('data-typ', 'MAGNETOGRAM'),
          ('date-obs', '1981-01-31T00:00:00'),
          ('tavg-ut', 19.289),
          ('carrot', 1704.7341),
          ('cenlon', 95.7238),
          ('b0', -0.106516),
          ('x0', 256.5),
          ('y0', 256.5),
          ('r0', 256.0),
          ('lambda', 5250.2),
          ('gv', 3.0),
          ('dxb_img', 3.8035),
          ('dyb_img', 3.8035),
          ('dxb_obs', 9.7679),
          ('dyb_obs', 12.6425),
          ('remark1', 'TSAFARG   .M'),
          ('remark2', None),
          ('psepma', 0.0),
          ('ichk', 0),
          ('mrkmod', 256),
          ('\x00\x00\x00\x00\x00\x00\x00\x00',
           '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'),
          ('cunit1', 'arcsec'),
          ('cunit2', 'arcsec'),
          ('cdelt1', 3.8035),
          ('cdelt2', 3.8035),
          ('crval1', 0.0),
          ('crval2', 0.0),
          ('rsun_obs', 973.696),
          ('crota2', 0.0),
          ('crpix1', 256.0),
          ('crpix2', 256.0),
          ('t_obs', '1981.02.01_18:08:00_TAI'),
          ('date_obs', '1981-01-31T00:00:00'),
          ('rsun_ref', 696000000),
          ('ctype1', 'HPLN-TAN'),
          ('ctype2', 'HPLT-TAN')])

Offending key:

'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'),

Error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-35-1d693c749a93> in <module>
----> 1 MWOmap.fits_header

/opt/anaconda3/lib/python3.7/site-packages/sunpy/map/mapbase.py in fits_header(self)
    713         A `~astropy.io.fits.Header` representation of the ``meta`` attribute.
    714         """
--> 715         return sunpy.io.fits.header_to_fits(self.meta)
    716 
    717 # #### Miscellaneous #### #

/opt/anaconda3/lib/python3.7/site-packages/sunpy/io/fits.py in header_to_fits(header)
    211 
    212         else:
--> 213             fits_header.append(fits.Card(k, v))
    214 
    215     if isinstance(key_comments, dict):

/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/card.py in __init__(self, keyword, value, comment, **kwargs)
    189             # value
    190             if keyword is not None:
--> 191                 self.keyword = keyword
    192             if value is not None:
    193                 self.value = value

/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/card.py in keyword(self, keyword)
    267                             keyword), VerifyWarning)
    268             else:
--> 269                 raise ValueError('Illegal keyword name: {!r}.'.format(keyword))
    270             self._keyword = keyword
    271             self._modified = True

ValueError: Illegal keyword name: '\x00\x00\x00\x00\x00\x00\x00\x00'.

Steps to Reproduce

Download this file:
ftp://howard.astro.ucla.edu/pub/obs/fits/1981a/MP00_5250MWO_00_0810201_1868.fits

    MWO_fits = fits.open('MP00_5250MWO_00_0810201_1868.fits', cache=False)
    MWO_fits.verify('fix')
    dataMWO = MWO_fits[0].data

    # Adding missing header information
    header = MWO_fits[0].header
    header['CUNIT1'] = 'arcsec'
    header['CUNIT2'] = 'arcsec'
    header['CDELT1'] = header['DXB_IMG']
    header['CDELT2'] = header['DYB_IMG']
    header['CRVAL1'] = 0.0
    header['CRVAL2'] = 0.0
    header['RSUN_OBS'] = (header['R0'])*header['DXB_IMG']
    header['CROTA2'] = 0.0
    header['CRPIX1'] = header['X0']-0.5
    header['CRPIX2'] = header['Y0']-0.5
    header['RSUN_REF']= 696000000
    header['CTYPE1']  = 'HPLN-TAN'                                                            
    header['CTYPE2']  = 'HPLT-TAN'                          

    # Assemble Sunpy map
    MWOmap = sunpy.map.Map(MWO_fits[0].data, header)

    # Turn Meta into header
    MWOmap.fits_header

System Details

• SunPy Version: 1.01
• Astropy Version: 3.2.1
• Python Version: 3.7.3
• OS information: linux debian

[amunozj]

hmm, now I see that there is a space in the header. An empty line. Maybe it has to do with that.

[Cadair]

So I don't have a lot of time to look into this, but it seems that this FITS file is just broken.

Reading the header with less shows:

MRKMOD  =                  256  / hrdwre 0=datex,256=ael,1280-1283=fm,1284=24ch ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@                                                                                END

That sequence of ^@ as far as I know should be whitespace if it padding to the END to make the header the correct length of bytes.

From a quick review of sunpy.io.fits.read we are just reading straight out of astropy here. You might need to move this up to astropy.

[jmason86]

Maybe the same issue?

instrument = a.Instrument('LASCO')
detector = a.Detector('C2')
wave = a.Wavelength(30 * u.nm, 31 * u.nm)
result = Fido.search(a.Time('2015-06-18T00:00:00', '2015-06-18T00:00:10'), instrument, detector)
downloaded_files = Fido.fetch(result)
map = sunpy.map.Map(downloaded_files)
map.fits_header

results in this traceback:

Traceback (most recent call last):
  File "/Users/jmason86/anaconda3/envs/cme_speed/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-ff7118f7559c>", line 1, in <module>
    maps[0].fits_header
  File "/Users/jmason86/anaconda3/envs/cme_speed/lib/python3.7/site-packages/sunpy/map/mapbase.py", line 715, in fits_header
    return sunpy.io.fits.header_to_fits(self.meta)
  File "/Users/jmason86/anaconda3/envs/cme_speed/lib/python3.7/site-packages/sunpy/io/fits.py", line 213, in header_to_fits
    fits_header.append(fits.Card(k, v))
  File "/Users/jmason86/anaconda3/envs/cme_speed/lib/python3.7/site-packages/astropy/io/fits/card.py", line 193, in __init__
    self.value = value
  File "/Users/jmason86/anaconda3/envs/cme_speed/lib/python3.7/site-packages/astropy/io/fits/card.py", line 333, in value
    'ASCII or non-printable characters.'.format(value))
ValueError: FITS header values must contain standard printable ASCII characters; 'offset_bias.pro\t1.37, 12/18/14, 631.118V15 18 Jun 1997 MAKE_FITS_HDR2' contains characters not representable in ASCII or non-printable characters.

This is what map.meta returns

MetaDict([('simple', True),
          ('bitpix', 16),
          ('naxis', 2),
          ('naxis1', 1024),
          ('naxis2', 1024),
          ('filename', '22551989.fts'),
          ('fileorig', '150618_000207.img'),
          ('date', '2015/10/30 05:38:23.081'),
          ('date-obs', '2015/06/18T00:00:05.794'),
          ('time-obs', '00:00:05.794'),
          ('p1col', 20),
          ('p1row', 1),
          ('p2col', 1043),
          ('p2row', 1024),
          ('version', 2),
          ('exptime', 25.0977),
          ('exp0', 23.0),
          ('expcmd', 23.0),
          ('exp1', 1.88428),
          ('exp2', 2.37305),
          ('exp3', 2.09766),
          ('telescop', 'SOHO'),
          ('instrume', 'LASCO'),
          ('detector', 'C2'),
          ('readport', 'C'),
          ('sumrow', 0),
          ('sumcol', 0),
          ('lebxsum', 1),
          ('lebysum', 1),
          ('shuttr', 0),
          ('lamp', 0),
          ('filter', 'Orange'),
          ('polar', 'Clear'),
          ('lp_num', 'Normal'),
          ('os_num', 4136),
          ('imgctr', 1622),
          ('imgseq', 0),
          ('comprssn', 'XZ%'),
          ('hcomp_sf', 64),
          ('mid_date', 57191),
          ('mid_time', 18.3428),
          ('platescl', 11.9),
          ('offset', 631.118),
          ('image_ct', 1622),
          ('seq_num', 0),
          ('obt_time', 1813276840.79),
          ('r1col', 20),
          ('r1row', 1),
          ('r2col', 1043),
          ('r2row', 1024),
          ('effport', 'C'),
          ('rectify', 'TRUE'),
          ('datamin', 633.0),
          ('datamax', 16384.0),
          ('datazer', 0),
          ('datasat', 0),
          ('dataavg', 2444.65),
          ('datasig', 1164.91),
          ('datap01', 710),
          ('datap10', 1192),
          ('datap25', 1804),
          ('datap75', 2938),
          ('datap90', 3496),
          ('datap95', 3900),
          ('datap98', 4782),
          ('datap99', 7476),
          ('crpix1', 511.2),
          ('crpix2', 507.5),
          ('crval1', 0.0),
          ('crval2', 0.0),
          ('crota1', 7.62989),
          ('crota2', 7.62989),
          ('ctype1', 'SOLAR-X'),
          ('ctype2', 'SOLAR-Y'),
          ('cunit1', 'arcsec'),
          ('cunit2', 'arcsec'),
          ('cdelt1', 11.9),
          ('cdelt2', 11.9),
          ('history',
           'offset_bias.pro\t1.37, 12/18/14, 631.118V15 18 Jun 1997 MAKE_FITS_HDR2'),
          ('comment', ''),
          ('waveunit', None)])

[nabobalis]

If I do the example as above (on current master), I get no error now.

[nabobalis]

If I try to open the original file, I get this:

MapMetaValidationError: Image coordinate units for axis 1 not present in metadata.
Image coordinate units for axis 2 not present in metadata.
See https://docs.sunpy.org/en/stable/code_ref/map.html#fixing-map-metadata` for instructions on how to add missing metadata.