Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update all subprocess calls to use errors="backslashreplace" #1145

Merged
merged 4 commits into from
Mar 24, 2023

Conversation

rmartin16
Copy link
Member

@rmartin16 rmartin16 commented Mar 23, 2023

Changes

  • Update calls to subprocess to include errors="backslashreplace" to avoid UnicodeDecodeError exceptions
  • Limit the ability for the output streamer to ignore ValueError exceptions to only those arising from the process unexpectedly closing stdout.

Example of effect of errors="backslashreplace"

Docker| [gtk/stdout] ‘girepository-1.0/Atk-1.0.typelib’ -> ‘/app/Hello World.AppDir//usr/lib/girepository-1.0/Atk-1.0.typelib’                          subprocess.py:641
Docker| [gtk/stdout] \xe2\x80[gtk/stderr] ++ realpath --relative-to=/usr/lib64 /usr/lib64/girepository-1.0                                              subprocess.py:641
Docker| [gtk/stderr] + cp girepository-1.0 --archive --parents '--target-directory=/app/Hello World.AppDir//usr/lib' --verbose                          subprocess.py:641
Docker| \x98girepository-1.0/SoupGNOME-2.4.typelib’ -> ‘/app/Hello World.AppDir//usr/lib/girepository-1.0/SoupGNOME-2.4.typelib’                        subprocess.py:641
Docker| [gtk/stdout] ‘girepository-1.0/GdkPixbuf-2.0.typelib’ -> ‘/app/Hello World.AppDir//usr/lib/girepository-1.0/GdkPixbuf-2.0.typelib’              subprocess.py:641

Issues

PR Checklist:

  • All new features have been tested
  • All new features have been documented
  • I have read the CONTRIBUTING.md file
  • I will abide by the code of conduct

- The use of `readline()` in the output streamer can raise `ValueError`
  exceptions for reasons other than `stdout` unexpectedly closing. So,
  ensure other exceptions are properly caught and reported in the log.
@rmartin16
Copy link
Member Author

So, I thought through specific tests for this functionality....and I can't think of any that wouldn't simply be testing whether subprocess respects the errors setting or not (...presumably it does). In principle, this change should just affect whether subprocess raises UnicodeDecodeError for bytes it cannot decode....or prints them in hex (like in my example above).

If others have tests they think would be valuable, let me know and I'll add them.

@rmartin16 rmartin16 marked this pull request as ready for review March 23, 2023 23:21
Copy link
Member

@freakboy3742 freakboy3742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me. One minor tweak to the formatting, mostly in service of making Black easier to deal with, but otherwise good to go.

I have a minor reservation about the sub_kw optimisation - it's a local test configuration we need to remember to use, and I don't necessarily consider verbosity a flaw in tests if the verbosity is explicit. However, I don't feel strongly enough to revert this myself (or ask you to revert it), and I can see some value in having a common definition so it's obvious and explicit when we're not using default values.

As for testing - I can't think of any obvious additional tests either. As you note, any testing of subprocess output really falls into the "testing that subprocess does what it says it does", which doesn't really improve quality.

@rmartin16
Copy link
Member Author

Tbh, the consolidation was more about avoiding having to update 100 tests again in the future. I think this was the third time I've done it because something in Subprocess was changed.

@freakboy3742
Copy link
Member

I figured as much :-) You will insist on working on gnarly bits of code, though... so you have no-one to blame by yourself :-P

@freakboy3742 freakboy3742 merged commit 788dd35 into beeware:main Mar 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants