Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception thrown when output is not configured as utf-8 #212

Closed
DonBraulio opened this issue Aug 5, 2020 · 7 comments
Closed

Exception thrown when output is not configured as utf-8 #212

DonBraulio opened this issue Aug 5, 2020 · 7 comments

Comments

@DonBraulio
Copy link

Describe the bug
Running a program with a rich progressbar inside a docker container, I got the following exception:

  File "/usr/local/lib/python3.6/dist-packages/rich/progress.py", line 150, in track
    sequence, total=total, description=description, update_period=update_period
  File "/usr/local/lib/python3.6/dist-packages/rich/progress.py", line 667, in track
    with self:
  File "/usr/local/lib/python3.6/dist-packages/rich/progress.py", line 627, in __enter__
    self.start()
  File "/usr/local/lib/python3.6/dist-packages/rich/progress.py", line 596, in start
    self.refresh()
  File "/usr/local/lib/python3.6/dist-packages/rich/progress.py", line 812, in refresh
    self.console.print(Control(""))
  File "/usr/local/lib/python3.6/dist-packages/rich/console.py", line 448, in __exit__
    self._exit_buffer()
  File "/usr/local/lib/python3.6/dist-packages/rich/console.py", line 426, in _exit_buffer
    self._check_buffer()
  File "/usr/local/lib/python3.6/dist-packages/rich/console.py", line 993, in _check_buffer
    self.file.write(text)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 16-55: ordinal not in range(128)

The problem can be easily solved in my case by setting the encoding properly:

export PYTHONIOENCODING=utf8

After that, the script runs perfectly fine with the beautiful rich progressbar.

However, setting this env var in some environments might not be possible or have other undesired consequences. It would be great if there was some kind of failback mode, or a WARNING with a suggestion to set this var, instead of failing without further information.

To Reproduce
Use the README progressbar example, let's call this file progress.py:

from rich.progress import track
for step in track(range(100)):
    pass

Run it with ascii encoding vs. utf8 and check the difference:

PYTHONIOENCODING=ascii python progress.py  # Exception thrown
PYTHONIOENCODING=utf8 python progress.py  # Works

Platform

  • rich==5.0.0
  • Probably works on any python3 installation. Reproduced on Python 3.6.10 on OSX Catalina, and Python 3.6.9 on Ubuntu Linux native and the same version on Linux for tegra (ARM) docker image.
@hedyhli
Copy link
Contributor

hedyhli commented Aug 5, 2020

I have also gotten this exception before when doing this on a raspberry pi, same error message. But what I did was I used "string".encode('utf-8')

@willmcgugan
Copy link
Collaborator

This has come up a few times and I've advised set PYTHONIOENCODING=utf8 which invariably fixes it.

My understanding is that when Python doesn't have a LANG env var set, it will default to ascii encoding. But Rich can't know if the encoding is a result of misconfiguration or intentional. And you'll get the same encoding error with the builtin print.

A fallback is possible for many things, ascii tables, progress bars etc. But there's not much I can do if the user prints CJK or emoji. Given that the problem is a result of a misconfiguration I'm a little reluctant to spend the effort.

A warning is a good idea, and I'll consider that. But I'm also thinking about wrapping that exception, and re-raising it with a helpful message about encodings.

@joaqo
Copy link
Contributor

joaqo commented Aug 6, 2020

Hi, I am substituting click for rich in an open source library, and hit this issue when running on docker.

What do you think is the best way to catch this exception so we can direct the user on how to solve it, or even avoid it altogether? I would have complete control over what is printed on the screen or how the code runs, I just want to make the transition from click to rich as easy as possible on users using docker.

@willmcgugan
Copy link
Collaborator

Catching the exception will be easy. There is only a single line in Console that actually writes to stdout. I can catch the exception there and re-raise it with some helpful info.

There's not much I can do to avoid it entirely that doesn't potentially break something else.

@willmcgugan
Copy link
Collaborator

This is what happens if you get a unicode error now:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-75: ordinal not in range(128)
*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***

Hopefully that will save time for developers running Rich code in docker / CI etc.

Still considering other options for this issue...

@joaqo
Copy link
Contributor

joaqo commented Aug 10, 2020

That looks awesome, more than good enough for my case. Thanks!

@DonBraulio
Copy link
Author

This is great @willmcgugan ! I'll take the chance to thank you for the awesome work on this library 🙌 🙌 🙌
Feel free to close this issue when you want to.

vladsavelyev added a commit to MultiQC/MultiQC that referenced this issue May 4, 2024
* Add types to report module

* Break up the main multiqc run function

* Initial setup for interactive use. WIP

* Allow multiple DOI

* Use Pydantic for plots and datasets

* Comlete Pydantic for all plot types, implement show_plot, support adding to a run

* Add script to print module plotting stats

* Support table. Add dt to plot model

* Salmon: remove non-standard fields from intermediate data

* No flat plot width for show function

* Add pydantic dep

* Fix heatmap and scatter

* Fix table_object.py when keys are different

* Fix creating violin

* Fix violin undefined check in JS

* Do no set export_plots in dev mode

* More JS proper checks for null

* Validate goleft_indexcov

* Fix lingraph.plot input types

* Use decimal format for decimal values

* Table: try parse before applying format

* Allow format=None

* Table deep copy data before applying modify

* Table: fix applying format to int

* Fix quast module

* Table object: make a copy of data and remove unused keys

* Filter None from table

* Collect raw_data and formatted_data separately in table object

* Validate format string in table

* ccs: fix format string

* Fix warning messages in violin

* Fix format and modify

* Use different tmp dir for log to fix windows

* Deleting modify in violin no longer needed

* Add comment why we create a separate tmp dir for logging

* Fix merge artefacts

* Gitignore notebooks

* Init rich console in log.py

* Global rich_console

* Named params to logger

* Do not create second rich console

* Fix

* Log to stdout

* Do not print logo in notebooks

* Use tqdm for search bar

* Progress bar in notebooks: fit screen and use stdout

* Print MultiQC intro without Rich

* Add tqdm

* Fix

* Revert "Add tqdm"

This reverts commit 2f5bfea.

* Revert "Print MultiQC intro without Rich"

This reverts commit fdf196a.

* Revert "Progress bar in notebooks: fit screen and use stdout"

This reverts commit fd1b8fc.

* Revert "Use tqdm for search bar"

This reverts commit 0d3c189.

* Revert "Do not print logo in notebooks"

This reverts commit 882f1b7.

* Use Rich console for logging (#2457)

* Use rich console for logging. Drop coloredlogs

* Apply patch for JUPYTER_HTML_FORMAT

* Set legacy_windows to fix CI?

* Do not use emoji in terminal

* Add get_module_data and get_general_stats_data interactive functions

* Add write_report and add_custom_content_section

* Add logger.propagate=False and remove logger argument from init_log

* logger.propagate = False only if is_running_in_notebook

* Use Rich in terminal, coloredlogs in notebooks

* Add tqdm into pyproject

* Split up multiqc.py into submodules

* Avoid emoji on windows

* Logger names to correspond to module files

* Do not use rich progress bar when unicode is not supported by the terminal, see Textualize/rich#212

* Support overriding config from command line or interactively: default all flags to None, defaults are only set in config_defaults.yaml

* Do not print the magnifier if no_unicode

* Do not print intro quen quiet

* Support passing temporary config parameters interactively to parse_logs and write_report

* Properly reset config and file search between parse_logs

* Add showcase notebook

* Further separate module run, directory creation, and disk write

* Fix

* Test CI output without s_fn

* Logging: remove name from the root logger

* Try rich console softwrap

* Fix

* Use logger params from config module instead of directly

* Try stderr=True for rich console

* Set COLUMNS in CI

* Update example notebook

* Disable progress bar in CI

* Fix ignoring samples

* Wrap API functions in copy_function_signature

* Do no clean up tmp dir within interactive session

* Test broken module

* Fix copy_fucntion_signature wrappers for older python

* Revert samtools assertion

* Use coloredlogs for PYCHARM_HOSTED and CI as it doesnt update the COLUMNS env var. Add line breaks to intro

* Do not print emoji if unicode is not supported (fixes windows CI)

* Redundant soft_wrap option to Rich console (ignored, and crops if it worked anyway)

* Override duplicated modules and warn about it

* Assertion if plot is not Plot or str

* Add custom content module to module outputs

* Fix dump layout to json

* Update notebook

* Add warning about table keys header mismatch and update notebook

* Do not return RunResult, plus fixes

* Move notebook to MultiQC_Notebook

* Move dump_json into util_functions.py

* Move api functions into multiqc/core/interactive.py

* Logging: print logger name, not module name

* Add load_config public function, plus add config.reset

* Refactoring

* COLUMNS in CI no needed as we do not use Rich in CI

* Call render_and_export_plots in list_plots

* Updates

* Improve adding section and multiqc.load_config functionality

* Do not import IPython globally

* Support sections with name=None (e.g. custom content)

* Move files around

* Make ClConfig a Pydantic model

* Fixes and expose ClConfig

* Move search patterns

* Point to dev branch of website in CI for moving search patterns

* Interactive functions to be configured with params rather than ClConfig

* Fix type hint

* Apply ignore sample in add_data_source

* Add unit test

* Make table empty message a debug message

* Suppoer flat flag in show_plot

* Fix import rich console

* Fix loading user configs

* Pass config.kwargs

* Make config.analysis_dir absolute

* Remove violin debug messages

* Rename pydantic models
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants