Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enabled passing of corrupt files to map #7018

Merged
merged 9 commits into from
Jun 2, 2023

Conversation

hannahc243
Copy link
Contributor

Fixes #3688 which allows passing of corrupt files to map

@hannahc243 hannahc243 requested a review from a team as a code owner May 23, 2023 12:03
@nabobalis nabobalis added map Affects the map submodule backport 5.0 labels May 23, 2023
@hayesla
Copy link
Member

hayesla commented May 23, 2023

Thanks for the PR @hannahc243 and congrats on your first contribution ! 🥳

@hannahc243 hannahc243 requested a review from a team as a code owner May 23, 2023 12:54
Copy link
Member

@ayshih ayshih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Some immediate thoughts:

  • Don't add a "bad" test file to the repository, much less one that is named to look like a real file! Such an approach is bound to cause confusion and breakages, as you've already discovered with the unrelated unit test that you had to separately fix. Instead, define a pytest fixture that creates a temporary "bad" test file for use in the unit tests.
  • Naming the keyword silence_errors makes me uneasy because it still emits warnings, so is definitely not silent. I'd probably name the keyword something like skip_errors, but others might have better suggestions.

@ayshih
Copy link
Member

ayshih commented May 23, 2023

Maybe allow errors?

@nabobalis
Copy link
Contributor

  • Don't add a "bad" test file to the repository, much less one that is named to look like a real file! Such an approach is bound to cause confusion and breakages, as you've already discovered with the unrelated unit test that you had to separately fix. Instead, define a pytest fixture that creates a temporary "bad" test file for use in the unit tests.

I don't see the harm in a bad test file, I was the one who told them to add it, the filename I think we can work on. It would be nice to have a file that is from a failed download, that we can use easily.

That unrelated unit test I don't think even needs to exist.

  • Naming the keyword silence_errors makes me uneasy because it still emits warnings, so is definitely not silent. I'd probably name the keyword something like skip_errors, but others might have better suggestions.

That silence_errors already exists in the Map API, I wanted to avoid adding a new keyword. If you prefer, we can add a new keyword that will handle this behaviour, but since "silence_errors" already does this but for hdu pairs that are passed, extending it to do it for files seems fine to me.

@ayshih
Copy link
Member

ayshih commented May 23, 2023

  • Don't add a "bad" test file to the repository, much less one that is named to look like a real file! Such an approach is bound to cause confusion and breakages, as you've already discovered with the unrelated unit test that you had to separately fix. Instead, define a pytest fixture that creates a temporary "bad" test file for use in the unit tests.

I don't see the harm in a bad test file, I was the one who told them to add it, the filename I think we can work on. It would be nice to have a file that is from a failed download, that we can use easily.

I think it'd be of extremely limited value to have in the repository, but I'd probably be fine if the name were simply changed to something like not_actually_a_fits_file.fits.

  • Naming the keyword silence_errors makes me uneasy because it still emits warnings, so is definitely not silent. I'd probably name the keyword something like skip_errors, but others might have better suggestions.

That silence_errors already exists in the Map API, I wanted to avoid adding a new keyword. If you prefer, we can add a new keyword that will handle this behaviour, but since "silence_errors" already does this but for hdu pairs that are passed, extending it to do it for files seems fine to me.

Well, then I additionally prefer to change the name of the keyword that already exists in the API. It doesn't make sense that a keyword called silence_errors does not actually silence anything, but instead replaces errors with (non-silent) warnings.

@nabobalis
Copy link
Contributor

nabobalis commented May 23, 2023

Well, then I additionally prefer to change the name of the keyword that already exists in the API. It doesn't make sense that a keyword called silence_errors does not actually silence anything, but instead replaces errors with (non-silent) warnings.

I was hoping you wouldn't say this but I agree.

Would you be ok with this being punted to another PR?

@ayshih
Copy link
Member

ayshih commented May 23, 2023

Yeah, punting is fine

Co-authored-by: Alasdair Wilson <[email protected]>
changelog/7018.feature.rst Outdated Show resolved Hide resolved
Copy link
Member

@hayesla hayesla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR @hannahc243 !

Looks great, just two suggestions

changelog/7018.feature.rst Outdated Show resolved Hide resolved
if kwargs.get("silence_errors"):
warn_user(msg)
return []
raise OSError(msg) from e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also add a message here to suggest to the user to use the silence_errors keyword to skip over invalid files?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is worth adding to my follow up PR renaming the keyword.

@nabobalis nabobalis added the No Backport A PR that isn't to be backported to any release branch. (To be used as a flag to other maintainers) label Jun 2, 2023
@nabobalis nabobalis added No Backport A PR that isn't to be backported to any release branch. (To be used as a flag to other maintainers) and removed No Backport A PR that isn't to be backported to any release branch. (To be used as a flag to other maintainers) labels Jun 2, 2023
@nabobalis
Copy link
Contributor

Thank you for the pull request @hannahc243!

@nabobalis nabobalis merged commit 97fe423 into sunpy:main Jun 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
map Affects the map submodule No Backport A PR that isn't to be backported to any release branch. (To be used as a flag to other maintainers)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide a keyword for map sequence to skip over bad files and continue loading.
6 participants