[dogstatsd] support unicode tags 🔣

[yannmh]

Using unicode with DogStatsD tags causes UnicodeDecodeError
exceptions.

Do not raise on unicode tags. Fix #132.

[truthbk]

that's right 💯

[degemer]

👍

[Julian]

This doesn't seem correct to me. I'm trying to diagnose a related issue (I'll update when I do), but from a very quick read, IMHO #132 should be wontfixed and this backed out. tags should either be unicode or bytes, and the current behavior asks for bytes. This changeset will break existing tags.

[Julian]

OK, it looks like the current behavior actually isn't consistent about which it's after, so I'm not sure what's intended. I can survey what the behavior is on different versions I guess after finding my own issue. This patchset is certainly assuming that everything passed to dogstatsd is unicode though (or accidentally-ASCII-encoded-bytes-on-py2).

[yannmh]

Hi @Julian,
I am not sure to understand exactly what issue you are facing. Could you give us more details about it please ? For instance

• Steps to reproduce
• Results you received versus the ones you would expect

Thanks!

[Julian]

@yannmh I've had to switch to debugging something else, I'll try to circle back and provide more detail, but this patchset changed something from calling str to calling unicode. That's not really safe -- the former fails when you call it on non-ASCII representable unicode, the latter fails in the inverse case, on non-ASCII representable bytes. So either the string literal, and the surrounding code here, are misleading, and should be unicode objects (and probably the documentation updated to say that tags should always be unicode objects now), or something has changed here.

You can reproduce by e.g. passing in a tag of "foo:שלום", i.e. bytes, instead of u"foo:שלום". It did look like however that the former is already broken because the DogStatsd object for some reason wants to call packet.encode("utf-8") on everything going over the socket, instead of just asking for bytes -- I'm not really sure what the intention is -- it seems likely that it's to accept unicode everywhere (that's not the decision I would have made, but maybe it's the one that exists already). I'll try to provide some more details if I get some free time to read through what's here.