Dogwrap fails to event with unicode in stderr, stdout when using buffer_outs option

[ronindesign]

[invalid] Real issue from -b (buffer_opts) with stderr,stdout redirect

Environment

• Shared Hosting (cPanel)
• Python 2.7.13

~$ ./.localpython/Python-2.7.12/python --version
Python 2.7.13

• Dog/Dogwrap v0.14.0

~$ ./.local/bin/dog --version
dog 0.14.0
~$ ./.local/bin/dogwrap --version
dogwrap 0.14.0

Context
Using dogwrap via cron to run a PHP cli script:
/home/MYUSER/.local/bin/dogwrap -n "My Cron" -k APIKEY -b --submit_mode all --notify_error="Notifying @MYEMAIL1 @MYEMAIL2" --tags=cron,test "php -q cron.php" >/dev/null 2>&1

The cron runs every 15 minutes. When there are no tasks, a small summary string of (ascii) text is sent to stdout from PHP cli (dogwrap correctly captures it, and I see it in my events view):

My Automation Task
===================================
...
 [OK] Completed

However, each and every time cron.php does have a task to do, dogwrap fails to send the event to datadog server. The dogwrap command doesn't seem to exit or crash, it simply finishes and prints buffered output from stdout and stderr like everything is fine, but the event never seems to send. The output in these cases, seen by removing null route >/dev/null is:

0/1 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0% < 1 sec/< 1 sec 16.0 MiB
 1/1 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 2 secs/2 secs 34.0 MiB
My Automation Task
=================================
...
 [OK] Completed

Thus, the only seemingly significant difference I can see between the outputs is the use of extended character encoding via UTF-8. Specifically the use of:
(UTF-8) Light Shade (U+2591) | ASCII: 9617
https://www.utf8icons.com/character/9617/light-shade
(UTF-8) Dark Shade ( U+2593) | ASCII: 9619
https://www.utf8icons.com/character/9619/dark-shade

I believe this is likely a unicode encoding issue.

Without redirecting stdout, there is no crash, everything prints fine in my console (where sys.stdout.encoding=UTF-8), and the event is successfully transmitted.
If I test with some unicode, and redirect to a file instead of to /dev/null, I see an exceptions:

Traceback (most recent call last):
  File "/home/static/.local/bin/dogwrap", line 11, in <module>
    load_entry_point('datadog==0.14.0', 'console_scripts', 'dogwrap')()
  File "/home/static/.local/lib/python2.7/site-packages/datadog-0.14.0-py2.7.egg/datadog/dogshell/wrap.py", line 316, in main
    print >> sys.stderr, stderr.strip()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2591' in position 30: ordinal not in range(128)

So maybe there is an issue with unicode support, conversion, handling, etc?
I mostly use Python 3+ these days, but, if I understand correctly, most datadog related python is based on version 2.7.x where unicode support is handled differently? (I.e. in 3+ a str is unicode by default, but no so in 2.7.x I think?)

• + Unicode output in stdout, stderr (with/without IO redirection).
• + Better consistency when handling unicode.
• ~ Would checking for I/O redirection be useful? Any unintended side effects?
• + Tags including unicode are ignored? Ex: --tags=test░tag, tag does not appear on event.

According to dogwrap --help:

-b, --buffer_outs     displays the stderr and stdout of the command only
                        once it has returned (the command outputs remains
                        buffered in dogwrap meanwhile)

It seems like the buffer_outs option causes different path for string printing, that doesn't include / support unicode encoding. I believe the culprit here in datadogpy.dogshell.wrap:

    if options.buffer_outs:
        print >> sys.stderr, stderr.strip()
        print >> sys.stdout, stdout.strip()

Following the state of stdout, it is concatenated with a unicode string, changing its type to unicode as well. However, it is never encoded back before printing.

Pull Request for minimal fix for just this localized issue: #203

[ronindesign]

Looks like unicode might be the issue, possibly duplicates: #55

[ronindesign]

Final: This issue is now invalid. The problem doesn't appear to be unicode related at all.

At this point, it appears there is an issue with the -b (buffer_outs) option on dogwrap when redirecting stdout,stderr to, maybe in general, but at least, /dev/null

Closing and reopening a new ticket that focuses on this directly without all the noise in this ticket.

[ronindesign]

Figured out it really was unicode issue. PR Submitted: #203

[yannmh]

I believe this is closed with #203. Thanks again @ronindesign