Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Writing JSON to STDOUT #1591

Closed
thehappydinoa opened this issue Oct 15, 2021 · 10 comments · Fixed by #3468
Closed

[BUG] Writing JSON to STDOUT #1591

thehappydinoa opened this issue Oct 15, 2021 · 10 comments · Fixed by #3468

Comments

@thehappydinoa
Copy link

Describe the bug
A clear and concise description of what the bug is.

I have been attempting to use rich to output a very large JSON into STDOUT.

To Reproduce
A minimal code example that reproduces the problem would be a big help if you can provide it. If the issue is visual in nature, consider posting a screenshot.

test_rich.py

from rich.console import Console

console = Console()
console.print_json(data=[{"hello": {"big wide": "world"}} for _ in range(50000)])
# Please note that my data is a decent bit larger
$ python test_rich.py | jq
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 14, column 9
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "my_module/cli/__init__.py", line 21, in main
    args.func(args)
  File "my_module/cli/commands/search.py", line 143, in cli_search
    write_file(results, **write_args)
  File "my_module/cli/utils.py", line 115, in write_file
    _write_screen(results_list)
  File "my_module/cli/utils.py", line 82, in _write_screen
    console.print_json(data=search_results)
  File "./.venv/lib/python3.9/site-packages/rich/console.py", line 1643, in print_json
    self.print(json_renderable)
  File "./.venv/lib/python3.9/site-packages/rich/console.py", line 1615, in print
    self._buffer.extend(new_segments)
  File "./.venv/lib/python3.9/site-packages/rich/console.py", line 825, in __exit__
    self._exit_buffer()
  File "./.venv/lib/python3.9/site-packages/rich/console.py", line 784, in _exit_buffer
    self._check_buffer()
  File ./.venv/lib/python3.9/site-packages/rich/console.py", line 1872, in _check_buffer
    self.file.write(text)
BrokenPipeError: [Errno 32] Broken pipe

Platform
What platform (Win/Linux/Mac) are you running on? What terminal software are you using?

macOS, I have tried the normal Terminal, iTerm 2, and VSCode Integrated Terminal.

Diagnose

$ python -m rich.diagnose
╭─────────────────────── <class 'rich.console.Console'> ───────────────────────╮
│ A high level console interface.                                              │
│                                                                              │
│ ╭──────────────────────────────────────────────────────────────────────────╮ │
│ │ <console width=80 ColorSystem.TRUECOLOR>                                 │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│     color_system = 'truecolor'                                               │
│         encoding = 'utf-8'                                                   │
│             file = <_io.TextIOWrapper name='<stdout>' mode='w'               │
│                    encoding='utf-8'>                                         │
│           height = 40                                                        │
│    is_alt_screen = False                                                     │
│ is_dumb_terminal = False                                                     │
│   is_interactive = True                                                      │
│       is_jupyter = False                                                     │
│      is_terminal = True                                                      │
│   legacy_windows = False                                                     │
│         no_color = False                                                     │
│          options = ConsoleOptions(                                           │
│                        size=ConsoleDimensions(width=80, height=40),          │
│                        legacy_windows=False,                                 │
│                        min_width=1,                                          │
│                        max_width=80,                                         │
│                        is_terminal=True,                                     │
│                        encoding='utf-8',                                     │
│                        max_height=40,                                        │
│                        justify=None,                                         │
│                        overflow=None,                                        │
│                        no_wrap=False,                                        │
│                        highlight=None,                                       │
│                        markup=None,                                          │
│                        height=None                                           │
│                    )                                                         │
│            quiet = False                                                     │
│           record = False                                                     │
│         safe_box = True                                                      │
│             size = ConsoleDimensions(width=80, height=40)                    │
│        soft_wrap = False                                                     │
│           stderr = False                                                     │
│            style = None                                                      │
│         tab_size = 8                                                         │
│            width = 80                                                        │
╰──────────────────────────────────────────────────────────────────────────────╯
$ python -m rich._windows
platform="Darwin"
WindowsConsoleFeatures(vt=False, truecolor=False)
$ pip freeze | grep rich
rich @ file:///Users/aidan/Library/Caches/pypoetry/artifacts/a2/2b/30/30eb6b7558c858219ff0094233e8dd6414d7733fe81ef92169292d774b/rich-10.12.0-py3-none-any.whl
@willmcgugan
Copy link
Collaborator

Works fine here. The broken pipe suggests it is something to do with the mechanics of piping the data in to JQ.

Are you using a custom shell?

What happens if you don't pipe it in to jq?

@thehappydinoa
Copy link
Author

I am using zsh but I have had issues in bash too.

@willmcgugan
Copy link
Collaborator

Since I can't reproduce it, can you help me narrow down what the trigger is? What happens if you output a just 10 objects? Does it break if you don't pipe it in to jq?

Does your three line example break for you? I can see that the traceback has some additional code.

@willmcgugan
Copy link
Collaborator

@thehappydinoa Any more information. Will close in a week if I can't reproduce this.

@thehappydinoa
Copy link
Author

Hi @willmcgugan I have been unable to reproduce. My apologies but the best thing for my use case was to be able to revert to standard print statements. Thank you for all the help.

@github-actions
Copy link

github-actions bot commented Nov 6, 2021

Did I solve your problem?

Consider sponsoring the ongoing work on Rich and Textual.

Or buy me a coffee to say thanks.

Will McGugan

@guacs
Copy link

guacs commented Oct 20, 2023

Hi @willmcgugan, this issue still persists and I think I have a minimal reproduction.

# file: foo.py
from rich.console import Console

console = Console()

console.print("foo")
console.print("bar")
console.print("baz")

Running the following bash command reproduces the error: python foo.py | grep -q "foo".

A few points to note:

  • the -q flag is needed for the reproduction
  • this is documented in the official Python docs here

Traceback:

Traceback (most recent call last):
  File "/home/guacs/open-source/pdm/trial.py", line 7, in <module>
    console.print("bar")
  File "/home/guacs/open-source/pdm/.venv/lib/python3.11/site-packages/rich/console.py", line 1673, in print
    with self:
  File "/home/guacs/open-source/pdm/.venv/lib/python3.11/site-packages/rich/console.py", line 865, in __exit__
    self._exit_buffer()
  File "/home/guacs/open-source/pdm/.venv/lib/python3.11/site-packages/rich/console.py", line 823, in _exit_buffer
    self._check_buffer()
  File "/home/guacs/open-source/pdm/.venv/lib/python3.11/site-packages/rich/console.py", line 2065, in _check_buffer
    self.file.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

EDIT: added traceback

@willmcgugan willmcgugan reopened this Oct 20, 2023
@lmmx
Copy link

lmmx commented Dec 16, 2023

Hmm yes just saw this regress after I switched to rich, but it can be remedied in the interim for integration in library code for example (who would otherwise see the same error if using regular print statements) with the following context manager.

pipes.py

import os
import sys

__all__ = ("pipe_cleanup",)


def pipe_cleanup() -> None:
    """Handle BrokenPipeError: redirect output to devnull"""
    devnull = os.open(os.devnull, os.O_WRONLY)
    os.dup2(devnull, sys.stdout.fileno())
    sys.exit(1)  # Python exits with error code 1 on EPIPE

error_handlers.py

from contextlib import AbstractContextManager
from types import TracebackType

from .pipes import pipe_cleanup

__all__ = ("SuppressBrokenPipeError",)


class SuppressBrokenPipeError(AbstractContextManager):
    def __exit__(
        self,
        __exc_type: type[BaseException] | None,
        __exc_value: BaseException | None,
        __traceback: TracebackType | None,
    ) -> bool | None:
        if __exc_type is BrokenPipeError:
            return pipe_cleanup()
        else:
            return super().__exit__(__exc_type, __exc_value, __traceback)

I incorporated this into my library like so:

io.py

from contextlib import contextmanager

from rich.console import Console

from .error_handlers import SuppressBrokenPipeError

__all__ = ("FugitConsole", "fugit_console")


class FugitConsole:
    console: Console
    use_pager: bool
    page_with_styles: bool

    def __init__(self, use_pager: bool = False, page_with_styles: bool = True):
        self.console: Console = Console()
        self.use_pager: bool = use_pager
        self.page_with_styles: bool = page_with_styles

    @contextmanager
    def pager_available(self):
        """Uses console pagination if `DisplayConfig` switched this setting on."""
        if self.use_pager:
            with self.console.pager(styles=self.page_with_styles):
                yield self
        else:
            yield self

    def print(self, output: str, end="", style=None) -> None:
        """
        Report output through the rich console, but don't style at all if rich was set to
        no_color (so no bold, italics, etc. either), and avoid broken pipe errors when
        piping to `head` etc.
        """
        with_style = style if fugit_console.no_color else None
        with SuppressBrokenPipeError():
            fugit_console.console.print(output, end=end, style=with_style)


"""
Global `rich.console.Console` instance modified by a model validator upon initialisation
of `fugit.interfaces.display.DisplayConfig` or its subclass, the main `DiffConfig` model.
"""
fugit_console = FugitConsole()

I can then test that piping output of my package's CLI entrypoint to | head doesn't truncate with a BrokenPipeError afterwards.

lmmx added a commit to lmmx/rich that referenced this issue Dec 16, 2023
@lmmx
Copy link

lmmx commented Dec 16, 2023

Opened a PR that fixes it:

Copy link

I hope we solved your problem.

If you like using Rich, you might also enjoy Textual

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants