ValueError: No supported files found

[arthur-e]

How I installed PyCoxMunk:

virtualenv --system-site-packages /usr/local/python-env/PyCoxMunk
source /usr/local/python-env/PyCoxMunk/bin/activate

# NOTE: I am in the repository when I install
cd /usr/local/dev/pycoxmunk
pip install pycoxmunk

What I did:

I installed Jupyter Notebook:

pip install notebook

And launched it, navigating to the Examples folder. I opened Simple_SLSTR.ipynb and ran through to the third cell:

# Create a scene by searching for files
scn = Scene(find_files_and_readers(base_dir=indir, reader='slstr_l1b'))
# Load the bands
scn.load(bnames)

# Additionally load the solar and satellite angles
# This saves computing them internally
scn.load(['solar_zenith_angle',
          'solar_azimuth_angle',
          'satellite_zenith_angle',
          'satellite_azimuth_angle'])

I am getting the following error and traceback:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[3], line 2
      1 # Create a scene by searching for files
----> 2 scn = Scene(find_files_and_readers(base_dir=indir, reader='slstr_l1b'))
      3 # Load the bands
      4 scn.load(bnames)

File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/readers/__init__.py:484, in find_files_and_readers(start_time, end_time, base_dir, reader, sensor, filter_parameters, reader_kwargs, missing_ok, fs)
    481     raise ValueError("Sensor '{}' not supported by any readers".format(sensor))
    483 if not (reader_files or missing_ok):
--> 484     raise ValueError("No supported files found")
    485 return reader_files

ValueError: No supported files found

And yet, I have the following files in the indir:

$ ls /home/arthur/Downloads/PyCoxMunk/SLSTR/
cartesian_an.nc   flags_bo.nc     indices_fn.nc      S4_quality_bo.nc   S6_radiance_bo.nc
cartesian_ao.nc   flags_fn.nc     indices_fo.nc      S4_radiance_an.nc  S7_BT_in.nc
cartesian_bn.nc   flags_fo.nc     indices_in.nc      S4_radiance_ao.nc  S7_BT_io.nc
cartesian_bo.nc   flags_in.nc     indices_io.nc      S4_radiance_bn.nc  S7_quality_in.nc
cartesian_fn.nc   flags_io.nc     met_tx.nc          S4_radiance_bo.nc  S7_quality_io.nc
cartesian_fo.nc   geodetic_an.nc  S1_quality_an.nc   S5_quality_an.nc   S8_BT_in.nc
cartesian_in.nc   geodetic_ao.nc  S1_quality_ao.nc   S5_quality_ao.nc   S8_BT_io.nc
cartesian_io.nc   geodetic_bn.nc  S1_radiance_an.nc  S5_quality_bn.nc   S8_quality_in.nc
cartesian_tx.nc   geodetic_bo.nc  S1_radiance_ao.nc  S5_quality_bo.nc   S8_quality_io.nc
F1_BT_fn.nc       geodetic_fn.nc  S2_quality_an.nc   S5_radiance_an.nc  S9_BT_in.nc
F1_BT_fo.nc       geodetic_fo.nc  S2_quality_ao.nc   S5_radiance_ao.nc  S9_BT_io.nc
F1_quality_fn.nc  geodetic_in.nc  S2_radiance_an.nc  S5_radiance_bn.nc  S9_quality_in.nc
F1_quality_fo.nc  geodetic_io.nc  S2_radiance_ao.nc  S5_radiance_bo.nc  S9_quality_io.nc
F2_BT_in.nc       geodetic_tx.nc  S3_quality_an.nc   S6_quality_an.nc   time_an.nc
F2_BT_io.nc       geometry_tn.nc  S3_quality_ao.nc   S6_quality_ao.nc   time_bn.nc
F2_quality_in.nc  geometry_to.nc  S3_radiance_an.nc  S6_quality_bn.nc   time_in.nc
F2_quality_io.nc  indices_an.nc   S3_radiance_ao.nc  S6_quality_bo.nc   viscal.nc
flags_an.nc       indices_ao.nc   S4_quality_an.nc   S6_radiance_an.nc  xfdumanifest.xml
flags_ao.nc       indices_bn.nc   S4_quality_ao.nc   S6_radiance_ao.nc
flags_bn.nc       indices_bo.nc   S4_quality_bn.nc   S6_radiance_bn.nc

Which are the data extracted from this ZIP archive downloaded from the Copernicus hub (URL):

S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.zip

I couldn't find satpy documentation on find_files_and_readers() but the source code gives me the impression I should be able to find a list of the expected files somewhere...

---

My Python environment:

$ pip freeze -r requirements.txt 
numpy==1.23.5
scipy==1.10.0
pandas==1.5.3
dask==2023.1.0
pyorbital==1.7.3
pyresample==1.26.0.post0
satpy==0.39.0
click==8.1.3
numba==0.56.4
xarray==2023.1.0

[simonrp84]

Hi,
Thanks for raising this. It looks like you unzipped the files into a folder named SLSTR. Satpy requires files to be unzipped into a folder with the same name (without extension) as the zip file itself, so in this case the folder name should be:
S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004

Could you try placing the files into a folder with that name and seeing if it works? If this fixes it, I'll add an additional comment in the notebook specifying how to unzip the data. Let me know how you get on!

[arthur-e]

Hi @simonrp84, I made the change to the folder name:

$ ls /home/arthur/Downloads/PyCoxMunk/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/
cartesian_an.nc   flags_an.nc     geometry_tn.nc     S2_radiance_an.nc  S5_quality_bo.nc   S8_BT_in.nc
cartesian_ao.nc   flags_ao.nc     geometry_to.nc     S2_radiance_ao.nc  S5_radiance_an.nc  S8_BT_io.nc
cartesian_bn.nc   flags_bn.nc     indices_an.nc      S3_quality_an.nc   S5_radiance_ao.nc  S8_quality_in.nc
cartesian_bo.nc   flags_bo.nc     indices_ao.nc      S3_quality_ao.nc   S5_radiance_bn.nc  S8_quality_io.nc
cartesian_fn.nc   flags_fn.nc     indices_bn.nc      S3_radiance_an.nc  S5_radiance_bo.nc  S9_BT_in.nc
cartesian_fo.nc   flags_fo.nc     indices_bo.nc      S3_radiance_ao.nc  S6_quality_an.nc   S9_BT_io.nc
cartesian_in.nc   flags_in.nc     indices_fn.nc      S4_quality_an.nc   S6_quality_ao.nc   S9_quality_in.nc
cartesian_io.nc   flags_io.nc     indices_fo.nc      S4_quality_ao.nc   S6_quality_bn.nc   S9_quality_io.nc
cartesian_tx.nc   geodetic_an.nc  indices_in.nc      S4_quality_bn.nc   S6_quality_bo.nc   time_an.nc
F1_BT_fn.nc       geodetic_ao.nc  indices_io.nc      S4_quality_bo.nc   S6_radiance_an.nc  time_bn.nc
F1_BT_fo.nc       geodetic_bn.nc  met_tx.nc          S4_radiance_an.nc  S6_radiance_ao.nc  time_in.nc
F1_quality_fn.nc  geodetic_bo.nc  S1_quality_an.nc   S4_radiance_ao.nc  S6_radiance_bn.nc  viscal.nc
F1_quality_fo.nc  geodetic_fn.nc  S1_quality_ao.nc   S4_radiance_bn.nc  S6_radiance_bo.nc  xfdumanifest.xml
F2_BT_in.nc       geodetic_fo.nc  S1_radiance_an.nc  S4_radiance_bo.nc  S7_BT_in.nc
F2_BT_io.nc       geodetic_in.nc  S1_radiance_ao.nc  S5_quality_an.nc   S7_BT_io.nc
F2_quality_in.nc  geodetic_io.nc  S2_quality_an.nc   S5_quality_ao.nc   S7_quality_in.nc
F2_quality_io.nc  geodetic_tx.nc  S2_quality_ao.nc   S5_quality_bn.nc   S7_quality_io.nc

However, I'm getting the same error as before.

I set a trace with ipdb in the corresponding satpy code, in the configs_for_reader() loop (ca. L476).

    473     for reader_configs in configs_for_reader(reader):
    474         (reader_instance, loadables, this_sensor_supported) = _get_loadables_for_reader_config(
    475                 base_dir, reader, sensor, reader_configs, reader_kwargs, fs)
    476         import ipdb
    477         ipdb.set_trace()#FIXME

I'm seeing:

>>> list(configs_for_reader(reader))
[['/usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/etc/readers/slstr_l1b.yaml']]

I looked through this YAML file but I don't really understand it.

>>> reader_instance
<satpy.readers.yaml_reader.FileYAMLReader object at 0x7f4d28427160>

>>> loadables
set()

>>> this_sensor_supported
False

And in the call to _get_loadables_for_reader_config():

>>> base_dir
'/home/arthur/Downloads/PyCoxMunk/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004'

>>> reader
'slstr_l1b'

>>> sensor is None
True

>>> reader_configs
['/usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/etc/readers/slstr_l1b.yaml']

>>> reader_kwargs
{'filter_parameters': {}}

>>> fs is None
True

[arthur-e]

I'm reading this satpy issue as well and I can also write:

• I have 98 files in the SL1RBT data directory

And:

>>> from satpy.utils import check_satpy; check_satpy()

Readers
=======
abi_l1b:  ok
abi_l1b_scmi:  ok
abi_l2_nc:  ok
acspo:  cannot find module 'satpy.readers.acspo' (No module named 'netCDF4')
agri_fy4a_l1:  cannot find module 'satpy.readers.agri_l1' (No module named 'h5py')
agri_fy4b_l1:  cannot find module 'satpy.readers.agri_l1' (No module named 'h5py')
ahi_hrit:  ok
ahi_hsd:  ok
ahi_l1b_gridded_bin:  ok
ami_l1b:  cannot find module 'satpy.readers.ami_l1b' (No module named 'pyspectral')
amsr2_l1b:  cannot find module 'satpy.readers.amsr2_l1b' (No module named 'h5py')
amsr2_l2:  cannot find module 'satpy.readers.amsr2_l2' (No module named 'h5py')
amsr2_l2_gaasp:  ok
amsub_l1c_aapp:  cannot find 'FrequencyDoubleSideBand' in the module 'satpy.readers.aapp_mhs_amsub_l1c'
ascat_l2_soilmoisture_bufr:  cannot find module 'satpy.readers.ascat_l2_soilmoisture_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n           Error: ', ModuleNotFoundError("No module named 'eccodes'")))
atms_l1b_nc:  cannot find module 'satpy.readers.atms_l1b_nc' (No module named 'netCDF4')
avhrr_l1b_aapp:  ok
avhrr_l1b_eps:  ok
avhrr_l1b_gaclac:  cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt:  cannot find module 'satpy.readers.hrpt' (No module named 'geotiepoints')
avhrr_l1c_eum_gac_fdr_nc:  ok
caliop_l2_cloud:  cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
clavrx:  cannot find module 'satpy.readers.clavrx' (No module named 'netCDF4')
cmsaf-claas2_l2_nc:  cannot find module 'satpy.readers.cmsaf_claas2' (No module named 'netCDF4')
electrol_hrit:  ok
fci_l1c_nc:  cannot find module 'satpy.readers.fci_l1c_nc' (No module named 'netCDF4')
fci_l2_nc:  ok
generic_image:  cannot find module 'satpy.readers.generic_image' (No module named 'rasterio')
geocat:  cannot find module 'satpy.readers.geocat' (No module named 'netCDF4')
ghi_l1:  cannot find module 'satpy.readers.ghi_l1' (No module named 'h5py')
ghrsst_l2:  ok
glm_l2:  ok
goes-imager_hrit:  ok
goes-imager_nc:  ok
gpm_imerg:  cannot find module 'satpy.readers.gpm_imerg' (No module named 'h5py')
grib:  cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib:  cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
hy2_scat_l2b_h5:  cannot find module 'satpy.readers.hy2_scat_l2b_h5' (No module named 'h5py')
iasi_l2:  cannot find module 'satpy.readers.iasi_l2' (No module named 'h5py')
iasi_l2_so2_bufr:  cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n           Error: ', ModuleNotFoundError("No module named 'eccodes'")))
ici_l1b_nc:  cannot find module 'satpy.readers.ici_l1b_nc' (No module named 'geotiepoints')
insat3d_img_l1b_h5:  cannot find module 'satpy.readers.insat3d_img_l1b_h5' (No module named 'datatree' It can be installed with the xarray-datatree package.)
jami_hrit:  ok
li_l2_nc:  cannot find module 'satpy.readers.li_l2_nc' (No module named 'netCDF4')
maia:  cannot find module 'satpy.readers.maia' (No module named 'h5py')
meris_nc_sen3:  ok
mersi2_l1b:  cannot find module 'satpy.readers.mersi2_l1b' (No module named 'pyspectral')
mhs_l1c_aapp:  ok
mimicTPW2_comp:  cannot find module 'satpy.readers.mimic_TPW2_nc' (No module named 'netCDF4')
mirs:  ok
modis_l1b:  cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
modis_l2:  cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf')
msi_safe:  cannot find module 'satpy.readers.msi_safe' (No module named 'rioxarray')
msu_gsa_l1b:  cannot find module 'satpy.readers.msu_gsa_l1b' (No module named 'h5py')
mtsat2-imager_hrit:  ok
mviri_l1b_fiduceo_nc:  ok
mwi_l1b_nc:  cannot find module 'satpy.readers.ici_l1b_nc' (No module named 'geotiepoints')
mws_l1b_nc:  cannot find module 'satpy.readers.mws_l1b' (No module named 'netCDF4')
nucaps:  cannot find module 'satpy.readers.nucaps' (No module named 'netCDF4')
nwcsaf-geo:  ok
nwcsaf-msg2013-hdf5:  cannot find module 'satpy.readers.nwcsaf_msg2013_hdf5' (No module named 'h5py')
nwcsaf-pps_nc:  ok
oceancolorcci_l3_nc:  cannot find module 'satpy.readers.oceancolorcci_l3_nc' (No module named 'netCDF4')
olci_l1b:  ok
olci_l2:  ok
omps_edr:  cannot find module 'satpy.readers.omps_edr' (No module named 'h5py')
safe_sar_l2_ocn:  ok
sar-c_safe:  cannot find module 'satpy.readers.sar_c_safe' (No module named 'rasterio')
satpy_cf_nc:  ok
scatsat1_l2b:  cannot find module 'satpy.readers.scatsat1_l2b' (No module named 'h5py')
seadas_l2:  cannot find module 'satpy.readers.seadas_l2' (No module named 'pyhdf')
seviri_l1b_hrit:  ok
seviri_l1b_icare:  cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf')
seviri_l1b_native:  ok
seviri_l1b_nc:  ok
seviri_l2_bufr:  cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
seviri_l2_grib:  cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
slstr_l1b:  ok
slstr_l2:  cannot find module 'satpy.readers.slstr_l2' (No module named 'satpy.readers.slstr_l2')
smos_l2_wind:  cannot find module 'satpy.readers.smos_l2_wind' (No module named 'netCDF4')
tropomi_l2:  cannot find module 'satpy.readers.tropomi_l2' (No module named 'netCDF4')
vaisala_gld360:  ok
vii_l1b_nc:  cannot find module 'satpy.readers.vii_l1b_nc' (No module named 'geotiepoints')
vii_l2_nc:  cannot find module 'satpy.readers.vii_l2_nc' (No module named 'geotiepoints')
viirs_compact:  cannot find module 'satpy.readers.viirs_compact' (No module named 'h5py')
viirs_edr_active_fires:  cannot find module 'satpy.readers.viirs_edr_active_fires' (No module named 'netCDF4')
viirs_edr_flood:  cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
viirs_l1b:  cannot find module 'satpy.readers.viirs_l1b' (No module named 'netCDF4')
viirs_sdr:  cannot find module 'satpy.readers.viirs_sdr' (No module named 'h5py')
virr_l1b:  cannot find module 'satpy.readers.virr_l1b' (No module named 'pyspectral')

Writers
=======
awips_tiled:  ok
cf:  cannot find module 'satpy.writers.cf_writer' (Ensure that the netCDF4 or h5netcdf package is installed.)
geotiff:  cannot find module 'satpy.writers.geotiff' (No module named 'rasterio')
mitiff:  ok
ninjogeotiff:  cannot find module 'satpy.writers.ninjogeotiff' (No module named 'rasterio')
ninjotiff:  cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
simple_image:  ok

Extras
======
cartopy:  No module named 'cartopy'
geoviews:  No module named 'geoviews'

But it seems that: slstr_l1b: ok .

---

Following that same satpy issue, I tried instead:

import glob
from satpy import Scene, find_files_and_readers
from pycoxmunk import PyCoxMunk
import numpy as np

from satpy.utils import debug_on
debug_on()

import warnings
warnings.filterwarnings('ignore')

# Set which bands we wish to load
bnames = ['S1', 'S2', 'S3']

# Set the directory containing the SLSTR data dir
# This folder should contain one or more subfolders with the SLSTR data (ending .SEN3)
# To save memory and CPU time, only use a few granules maximum for this.
indir = '/home/arthur/Downloads/PyCoxMunk/inputs'

# Set the directory where we'll save the output
outdir = '/home/arthur/Downloads/PyCoxMunk/'

files = glob.glob(f'{indir}/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/*')

And I have a new error! Well, a variation on the older error. Progress.

>>> scn = Scene(filenames = files, reader = "slstr_l1b")
[DEBUG: 2023-01-25 08:00:49 : satpy.readers.yaml_reader] Reading ('/usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/etc/readers/slstr_l1b.yaml',)
[WARNING: 2023-01-25 08:00:49 : satpy.readers.yaml_reader] No filenames found for reader: slstr_l1b
[WARNING: 2023-01-25 08:00:49 : satpy.readers] Don't know how to open the following files: {'/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/met_tx.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/S5_quality_bn.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/F2_BT_io.nc',
...
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[6], line 1
----> 1 scn = Scene(filenames = files, reader = "slstr_l1b")

File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/scene.py:133, in Scene.__init__(self, filenames, reader, filter_parameters, reader_kwargs)
    130 if filenames:
    131     filenames = convert_remote_files_to_fsspec(filenames, storage_options)
--> 133 self._readers = self._create_reader_instances(filenames=filenames,
    134                                               reader=reader,
    135                                               reader_kwargs=cleaned_reader_kwargs)
    136 self._datasets = DatasetDict()
    137 self._wishlist = set()

File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/scene.py:154, in Scene._create_reader_instances(self, filenames, reader, reader_kwargs)
    149 def _create_reader_instances(self,
    150                              filenames=None,
    151                              reader=None,
    152                              reader_kwargs=None):
    153     """Find readers and return their instances."""
--> 154     return load_readers(filenames=filenames,
    155                         reader=reader,
    156                         reader_kwargs=reader_kwargs)

File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/readers/__init__.py:579, in load_readers(filenames, reader, reader_kwargs)
    576         break
    578 _check_remaining_files(remaining_filenames)
--> 579 _check_reader_instances(reader_instances)
    580 return reader_instances

File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/readers/__init__.py:618, in _check_reader_instances(reader_instances)
    616 def _check_reader_instances(reader_instances):
    617     if not reader_instances:
--> 618         raise ValueError("No supported files found")
    619     if not any(list(r.available_dataset_ids) for r in reader_instances.values()):
    620         raise ValueError("No dataset could be loaded. Either missing "
    621                          "requirements (such as Epilog, Prolog) or none of the "
    622                          "provided files match the filter parameters.")

ValueError: No supported files found

So, it seems that despite slstr_l1b: ok in the output of check_satpy(), that reader is not really "ok"?

Also, I've since installed netCDF4 (and h5py) in this virtual environment, since it seemed that it would be impossible to open *.nc files otherwise. That was prior to getting the ValueError above, however.

Are these supposed to be *.SEN3 files, instead, as a comment in Simple_SLSTR.ipynb implies?

[simonrp84]

Hmm, this is curious. I'll download those files myself and see what problems I run into...

[simonrp84]

Ok, I tried this myself and both find_files_and_readers and glob.glob work for setting up the Scene.
However, I'm using a conda-based install, whereas you've been using pip. The satpy developers suggest that there may be a conflict as you're in the pycoxmunk directory but not installing the files from that directory and that this may result in different / conflicting libraries being used.
Could you please re-try the installation you give in your initial message, but replace pip install pycoxmunk with pip install .?

[arthur-e]

pip install . did not work. I don't usually use conda because it is so slow, but I just installed miniconda to try it out...

conda activate
conda create -n PyCoxMunk
conda activate PyCoxMunk
conda install -c conda-forge pycoxmunk

Then, running the code from before, I'm getting the same error. I also get the error when running Simple_SLSTR.ipynb after conda install notebook. Here's the output from conda list:

$ conda list
# packages in environment at /usr/local/dev/miniconda3/envs/PyCoxMunk:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
affine                    2.4.0              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
blosc                     1.21.3               hafa529b_0    conda-forge
bokeh                     2.4.3              pyhd8ed1ab_3    conda-forge
boost-cpp                 1.78.0               h75c5d50_1    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cairo                     1.16.0            ha61ee94_1014    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
cfitsio                   4.2.0                hd9d235c_0    conda-forge
cftime                    1.6.2           py310hde88566_1    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
configobj                 5.0.8              pyhd8ed1ab_0    conda-forge
cryptography              39.0.0          py310h34c0648_0    conda-forge
curl                      7.87.0               hdc1c0ab_0    conda-forge
cytoolz                   0.12.0          py310h5764c6d_1    conda-forge
dask                      2023.1.0           pyhd8ed1ab_0    conda-forge
dask-core                 2023.1.0           pyhd8ed1ab_0    conda-forge
distributed               2023.1.0           pyhd8ed1ab_0    conda-forge
docutils                  0.19            py310hff52083_1    conda-forge
donfig                    0.7.0              pyhd8ed1ab_1    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
expat                     2.5.0                h27087fc_0    conda-forge
fasteners                 0.17.3             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.1               hc2a2eb6_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
freexl                    1.0.6                h166bdaf_1    conda-forge
fsspec                    2023.1.0           pyhd8ed1ab_0    conda-forge
geos                      3.11.1               h27087fc_0    conda-forge
geotiff                   1.7.1                h7157cca_5    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
h5py                      3.8.0           nompi_py310h0311031_100    conda-forge
hdf4                      4.2.15               h9772cbc_5    conda-forge
hdf5                      1.12.2          nompi_h4df4325_101    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
json-c                    0.16                 hc379101_0    conda-forge
kealib                    1.5.0                ha7026e8_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lcms2                     2.14                 hfd0df8a_1    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libcurl                   7.87.0               hdc1c0ab_0    conda-forge
libdeflate                1.17                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgdal                   3.6.2                h10cbb15_3    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libglib                   2.74.1               h606061b_1    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.4                h166bdaf_0    conda-forge
libkml                    1.3.0             h37653c0_1015    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libllvm11                 11.1.0               he0ac6c6_5    conda-forge
libnetcdf                 4.8.1           nompi_h261ec11_106    conda-forge
libnghttp2                1.51.0               hff17c54_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.1                 hb675445_3    conda-forge
librttopo                 1.1.0               ha49c73b_12    conda-forge
libspatialite             5.0.1               h7c8129e_22    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libtiff                   4.5.0                h6adf6a1_2    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzip                    1.9.2                hc929e4a_1    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
llvmlite                  0.39.1          py310h58363a5_1    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.2.0           py310h0cfdcf0_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markupsafe                2.1.2           py310h1fa729e_0    conda-forge
msgpack-python            1.0.4           py310hbf28c38_1    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
netcdf4                   1.6.2           nompi_py310h55e1e36_100    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.82                 he02c5a1_0    conda-forge
numba                     0.56.4          py310ha5257ce_0    conda-forge
numcodecs                 0.11.0          py310heca2aa9_1    conda-forge
numpy                     1.23.5          py310h53a5b5f_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.0.7                h0b41bf4_1    conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
pandas                    1.5.3           py310h9b08913_0    conda-forge
partd                     1.3.0              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pillow                    9.4.0           py310h4927cde_0    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pooch                     1.6.0              pyhd8ed1ab_0    conda-forge
poppler                   22.12.0              h091648b_1    conda-forge
poppler-data              0.4.11               hd8ed1ab_0    conda-forge
postgresql                15.1                 h3248436_3    conda-forge
proj                      9.1.0                h8ffa02c_1    conda-forge
psutil                    5.9.4           py310h5764c6d_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pycoxmunk                 1.0.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pykdtree                  1.3.6           py310hde88566_2    conda-forge
pyopenssl                 23.0.0             pyhd8ed1ab_0    conda-forge
pyorbital                 1.7.3              pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyproj                    3.4.1           py310hfc24d34_0    conda-forge
pyresample                1.26.0.post0    py310h769672d_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pyspectral                0.12.3             pyhd8ed1ab_0    conda-forge
python                    3.10.8          h4a9ceb5_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-geotiepoints       1.5.1           py310h0a54255_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytz                      2022.7.1           pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py310h5764c6d_5    conda-forge
rasterio                  1.3.4           py310hfc14bbd_0    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.2             pyhd8ed1ab_0    conda-forge
satpy                     0.39.0             pyhd8ed1ab_0    conda-forge
scipy                     1.10.0          py310h8deb116_0    conda-forge
setuptools                66.1.1             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.9                hbd366e4_2    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.40.0               h4ff8645_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
tiledb                    2.13.2               hd532e3d_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.2             py310h5764c6d_1    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
trollimage                1.20.0             pyhd8ed1ab_0    conda-forge
trollsift                 0.5.0              pyhd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzcode                    2022g                h166bdaf_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
urllib3                   1.26.14            pyhd8ed1ab_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xarray                    2023.1.0           pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.4                h55805fa_1    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zarr                      2.13.3             pyhd8ed1ab_0    conda-forge
zict                      2.2.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstd                      1.5.2                h3eb15da_6    conda-forge

[kameshvinjamuri]

hi @arthur-e , try the following

from satpy import Scene, find_files_and_readers
from satpy.utils import debug_on
debug_on()

if __name__ == '__main__':
    files = find_files_and_readers(base_dir="/home/**/slstr/", reader='slstr_l1b')

    scn = Scene(filenames=files, reader='slstr_l1b')
    my_channel = 'S1'
    scn.load([my_channel])
    #scn[my_channel].data

    import matplotlib.pyplot as plt
    plt.figure()
    scn[my_channel].plot.imshow(cmap='viridis')
    plt.show()

[kameshvinjamuri]

@arthur-e @simonrp84
i guess the problem is with solar zenith angles. it is because it is in a different stripe (tie point grid). but the commands above by default use the stripe<1> i.e 'a'

[arthur-e]

Hi @kameshvinjamuri, I saved your Python script as test.py and ran it:

$ python test.py 
/usr/local/dev/miniconda3/envs/PyCoxMunk/lib/python3.10/site-packages/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in entry_points().get(name, []):
[DEBUG: 2023-01-25 13:16:32 : satpy.readers.yaml_reader] Reading ('/usr/local/dev/miniconda3/envs/PyCoxMunk/lib/python3.10/site-packages/satpy/etc/readers/slstr_l1b.yaml',)
Traceback (most recent call last):
  File "/usr/local/dev/PyCoxMunk/test.py", line 6, in <module>
    files = find_files_and_readers(base_dir="/home/**/slstr/", reader='slstr_l1b')
  File "/usr/local/dev/miniconda3/envs/PyCoxMunk/lib/python3.10/site-packages/satpy/readers/__init__.py", line 484, in find_files_and_readers
    raise ValueError("No supported files found")
ValueError: No supported files found

I gather than Sentinel-3 has multiple overpasses ("stripes"?). Is there a way to specify which "stripe" to use? I am looking at the satpy Quickstart and I don't see a relevant example.

[kameshvinjamuri]

@arthur-e can you repeat test.py with an exclusive satpy environment using conda ( i mean conda create new env, conda install satpy).

[kameshvinjamuri]

@arthur-e slstr have different stripe for different bands. most of the 500m bands are available in all stripes namely a, b, c, and tie point. for example, some of the fire channels at 1000 m resolution have only b/c stripe. a tie point stripe is something that is sampled for every 16 points (unfortunately solar zenith angles are only at tie point). we should interpolate to the resolution chosen. I suggest the slstr level 1 ATBD. (things mentioned here might change in the future with the new data product versions)

[arthur-e]

@kameshvinjamuri I have tried it in a fresh conda environment:

conda create -c conda-forge -n satpy python satpy
conda activate satpy

import glob
from satpy import Scene, find_files_and_readers
import numpy as np

from satpy.utils import debug_on
debug_on()

import warnings
warnings.filterwarnings('ignore')

# Set which bands we wish to load
bnames = ['S1', 'S2', 'S3']

# Set the directory containing the SLSTR data dir
# This folder should contain one or more subfolders with the SLSTR data (ending .SEN3)
# To save memory and CPU time, only use a few granules maximum for this.
indir = '/home/arthur/Downloads/PyCoxMunk/inputs'

# Set the directory where we'll save the output
outdir = '/home/arthur/Downloads/PyCoxMunk/'

# Re-downloaded this granule using direct URL and it now has ".SEN3" appended to the folder name
files = glob.glob(f'{indir}/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/*')

scn = Scene(filenames = files, reader = "slstr_l1b")

And this is the output I obtained:

[DEBUG: 2023-01-25 14:50:52 : satpy.readers.yaml_reader] Reading ('/usr/local/dev/miniconda3/envs/satpy/lib/python3.11/site-packages/satpy/etc/readers/slstr_l1b.yaml',)
[DEBUG: 2023-01-25 14:50:52 : satpy.readers.yaml_reader] Assigning to slstr_l1b: ['/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S1_radiance_ao.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S6_radiance_ao.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S2_radiance_ao.nc', 

...

[WARNING: 2023-01-25 14:50:54 : satpy.readers] Don't know how to open the following files: {'/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S6_quality_bn.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/indices_ao.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S2_quality_ao.nc',

...

[simonrp84]

That looks like it was successful, and that the scene was created.

The stripes etc should not be important in this case, by the way, and you don't need to specify them when creating or loading the Scene.

[arthur-e]

@simonrp84 I think I now understand what the issue is. In Examples/Simple_SLSTR.ipynb there is written:

scn = Scene(find_files_and_readers(base_dir=indir, reader='slstr_l1b'))

However, what ultimately worked when I created a new environment was:

files = glob.glob(f'{indir}/*/*')
scn = Scene(filenames = files, reader = "slstr_l1b")

So, it seems that Scene also needs to know what reader to use. In Examples/Simple_SLSTR.ipynb, this would amount to:

scn = Scene(find_files_and_readers(base_dir=indir, reader='slstr_l1b'), reader = 'slstr_l1b')

Or, for clarity:

file_list = find_files_and_readers(base_dir = indir, reader = 'slstr_l1b')
scn = Scene(file_list, reader = 'slstr_l1b')

So, I created a new virtualenv and installed PyCoxMunk in editable mode using pip (working with my preferred installation approach for now):**

cd PyCoxMunk
pip install -e .

I found that netCDF4 and h5py were not installed as part of the PyCoxMunk installation:

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
https://docs.xarray.dev/en/stable/user-guide/io.html 
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html

So I also added a note to the Notebook about needing to install netCDF4 for the example.

I put all this in a Pull Request. See PR #4. Glad we got this figured out! If PyCoxMunk users have no prior experience with satpy, I think these changes will be essential to avoid confusion.