Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adapt the SEVIRI native format reader in Satpy to support remote reading #2863

Merged
merged 22 commits into from
Aug 14, 2024

Commits on Aug 12, 2024

  1. Add readers.utils.fromfile() for remote reading

    This function uses `readers.utils.generic_open()` and `np.frombuffer()` to achieve this.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    5401d28 View commit details
    Browse the repository at this point in the history
  2. Update AUTHORS.md

    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    7142a68 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8051d50 View commit details
    Browse the repository at this point in the history
  4. Adapt readers.seviri_l1b_native for remote reading

    In particular, the following functions/methods have been modified:
    - `has_archive_header()` now uses `readers.utils.generic_open()` instead of `open()`.
    - `read_header()` now uses `readers.utils.fromfile()` instead of `np.formfile()`.
    - `NativeMSGFileHandler._read_trailer()` now uses `readers.utils.fromfile()` instead of `np.formfile()`.
    - `NativeMSGFileHandler._get_memmap()` has been renamed to `NativeMSGFileHandler._get_array()` and now uses `readers.utils.fromfile()` instead of `np.memmap()`.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    16f2642 View commit details
    Browse the repository at this point in the history
  5. Update test_seviri_l1b_native with _get_array

    Reason: since `NativeMSGFileHandler._get_memmap()` has been renamed to `NativeMSGFileHandler._get_array()`.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    1f10690 View commit details
    Browse the repository at this point in the history
  6. Update mock.patch args in test_seviri_l1b_native to match changes i…

    …n `seviri_l1b_native`
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    7d516d8 View commit details
    Browse the repository at this point in the history
  7. Update test_seviri_l1b_native with tests for remote reading

    This includes generating an actual file on disk and attempt to read it.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    d01a519 View commit details
    Browse the repository at this point in the history
  8. Parametrize test_read_physical_seviri_nat_file to test zip files as…

    … well as plain files
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    0e5f3e7 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    15d17d6 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    df4a6db View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    43b846b View commit details
    Browse the repository at this point in the history
  12. Use a header which leads to a smaller seviri nat file on disk

    This concerns `test_read_physical_seviri_nat_file()`.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    816a2c7 View commit details
    Browse the repository at this point in the history
  13. Make the returned path posix compliant in compress_seviri_native_file

    This is to ensure that the remote reading tests can also run on Windows.
    Note: `fsspec` expects a POSIX path.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    d4976c3 View commit details
    Browse the repository at this point in the history
  14. Resolve warnings raised as a result of slow & order being pytest …

    …unknown marks
    
    We do not need these marks anymore, as we decreased the size of the generated seviri native file. As a result, the tests now run fast enough.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    db3ddb8 View commit details
    Browse the repository at this point in the history
  15. Resolve warnings raised as a failure in the orbit polynomial

    This warning is totally benign. It is caused as a result of the seviri native file that we create which is essentially filled with zeros.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    fbd02b5 View commit details
    Browse the repository at this point in the history
  16. Fix the issue with the docstring in amend_seviri_native_null_header()

    The issue was an unexpected indentation on the last line of the docstring!
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    f002b00 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    a7596d3 View commit details
    Browse the repository at this point in the history
  18. Use dask map_blocks() in _get_array()

    This includes:
    - Extracting the `_get_array()` method so that it is now a function in the module and not a class method.
    - Introduction of `NativeMSGFileHandler_make_dask_array_with_map_blocks()` method to utilize the dask `map_blocks()`.
    - Introduction of a new method, namely `NativeMSGFileHandler._number_of_visir_channels` to facilitate testing and mock patching.
    - Adapting the mock patches in tests accordingly.
    pkhalaj committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    80f3925 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Configuration menu
    Copy the full SHA
    42dfc05 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1c00de7 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b58823c View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Configuration menu
    Copy the full SHA
    d22d2b6 View commit details
    Browse the repository at this point in the history