Fix `with_spectral_unit` #944

Job	Run time
(Allowed Failure) Python 3.12 with remote data and dev version of key dependencies	4m 55s
Code style checks	11s
Python 3.11 with astropy data and coverage	2m 1s
Python 3.12 (Windows)	2m 32s
Python 3.12 (MacOS X)	2m 20s
Python 3.10	1m 33s
Python 3.9	1m 40s
Python 3.8 with oldest supported version of key dependencies	1m 46s
	16m 58s

Provide feedback