Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anticipate filehandler sorting in GEOSegmentYAMLReader to have sorted handlers also with pad_data=False #2692

Conversation

ameraner
Copy link
Member

This PR fixes the issue reported in #2588 by making the segment sorting happen early during create_filehandlers, so that it acts for both pad_data=True and False.

The root cause was that the filehandler sorting by segment was done only in the default pad_data=True branch inside the padding mechanism of GEOSegmentYAMLReader. When calling e.g. scn.load(['ir_105'], pad_data=False, upper_right_corner='NE'), in the previous implementation the filehandlers were being sorted (in the base FileYAMLReader.create_filehandlers) only by start_time (which is always the same after #2420), and then alphabetically by filenames. This is problematic for FCI where sorting alphabetically means sorting by processing_time, and there are cases where the processing_time order does not correspond to the chunk order (due to parallel processing), causing the issue.

Note that even without #2420, some FCI segments have the same filename start_time, hence the bug would have happened anyway.

@ameraner
Copy link
Member Author

This snippet of code

scn = Scene(filenames=filenames, reader='fci_l1c_nc')
scn.load(['ir_105'], pad_data=False, upper_right_corner='NE')
scn.save_dataset('ir_105', filename='ir105_nopad.png')

produces this before this PR

image

and this after this PR

image

Copy link
Member

@pnuu pnuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

codecov bot commented Dec 15, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c5c18cf) 95.32% compared to head (36aa471) 95.35%.
Report is 29 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2692      +/-   ##
==========================================
+ Coverage   95.32%   95.35%   +0.03%     
==========================================
  Files         371      371              
  Lines       52441    52485      +44     
==========================================
+ Hits        49988    50049      +61     
+ Misses       2453     2436      -17     
Flag Coverage Δ
behaviourtests 4.15% <3.70%> (-0.01%) ⬇️
unittests 95.97% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

Pull Request Test Coverage Report for Build 7221665410

Warning: This coverage report may be inaccurate.

We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report.
To ensure accuracy in future PRs, please see these guidelines.
A quick fix for this PR: rebase it; your next report should be accurate.

  • 27 of 27 (100.0%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall first build on bugfix_add_filehandler_sorting_with_paddata_false at 95.922%

Totals Coverage Status
Change from base Build 7220123684: 95.9%
Covered Lines: 50175
Relevant Lines: 52308

💛 - Coveralls

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mraspaud mraspaud merged commit eed1c1c into pytroll:main Dec 15, 2023
18 of 19 checks passed
@ameraner ameraner deleted the bugfix_add_filehandler_sorting_with_paddata_false branch April 17, 2024 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging this pull request may close these issues.

FCI chunks/segments out of order if pad_data=False
4 participants