The debug run reports an error, but there is no problem running after stopping at the breakpoint #2907
Comments
|
@haiyangdaozhang thanks for reaching out to us! Now why that does not happen systematically, I don't know... but maybe NaNs aren't very stable? |
|
@mraspaud If you look higher up in their error traceback you'll see: and This would usually mean you're providing different amounts of geolocation versus data files, but I find it extremely odd (impossible?) to have negative chunk sizes. I'm not sure how that happened. Otherwise, I have no idea why a breakpoint or no breakpoint would change the results in your code unless we have a bug in how we're using dask. Other coding things to note @haiyangdaozhang: First: Could likely have the loop removed and just do Second: Could be: That is, you don't need the |
|
Thanks for the advice! This is indeed a very strange question. Maybe the interactive run introduces something new？ |
|
The error you're getting starts in the reader. The dynamic area definition stuff doesn't really matter as it is mainly complaining that no datasets were loaded so it can't resample anything or even create a "frozen" area definition. What type of files are you providing to the Scene? Are you sure you're providing the same number of granules for I and M bands? Where did you get your files from? The negative values in the chunk sizes makes me think that the files have invalid values for some of the information in them (ex. number of scans). Are the files you're providing aggregate files with more than one granule in them? What about with more than one band (ex. SVI01 and SVI02 in the same file)? |
|
|
|
How many files are you passing in To further debug this may require us getting access to your files to look at the metadata and try to reproduce the issue. |
|
Thank you for your help. I will put the data into Google Drive, which will take some time. Perhaps the snr is not good enough, causing quality issues with the received data? |
|
That may play a part later, but unless I'm missing something the errors are purely talking about the shape of the arrays in the files. Satpy takes various parameters from the metadata like number of rows per instrument scan and number of scans in the file and "chunks" the data into separate pieces. These separate chunks are then processed in parallel (using dask). For some reason the chunks being provided to dask from the VIIRS SDR reader include negative numbers. Again, these chunk sizes are computed from the size of the arrays and the metadata from the file. For them to be negative likely means that the metadata is invalid or that the reader in Satpy needs to be updated to handle "fill values" for some of these metadata items. For example, if number of scans is set to a fill value of |
|
These are the data I used. Thank you again! |
|
Thanks for the files. It took a bit for google to let me download all of them, but the following code worked for me on a PopOS Linux system with Python 3.12 and latest Satpy: from satpy import Scene; from glob import glob
scn = Scene(reader="viirs_sdr", filenames=glob("viirs_l1b_test/*.h5"))
scn.load(["ocean_color"])
projection = {'proj': 'latlong'}
area_def = create_area_def(area_id='User_define', projection=projection, resolution=0.02, units='degrees')
scn_new = scn.resample(area_def, resampler="nearest")
scn_new.save_datasets() |
|
Thank you！ I created Python 3.12 on Rocky, and it's working fine! |
|
Just curious, you said you were on Windows before, what version of Python were you running? |
|
Windows 11 & Python 3.9.13. |
|
Hm good find. Yeah seeing as the error was about dask array chunk sizes, it seems reasonable that the old version of dask is causing the issue. I'll close this as it seems like we figured it out. We can reopen if you would like. |
Describe the bug
I encountered a very strange problem. When I used debug or ran directly, an error was reported. However, I set a breakpoint on the last line. When I ran to the breakpoint and ran the last line （
scn_new.save_dataset(iproduct,filename=filename_product)） in the interactive interface, there was no error and an image was generated.The received image is not very good, there are some bad sectors, is it related to this?
To Reproduce
The error reported when running directly is as follows：
Environment Info:
The text was updated successfully, but these errors were encountered: