-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
error while decoding serialized histogram produced by rust version #29
Comments
traceback 1: This looks like an issue with the compressed data (what is base 64 encoded). traceback 2 and 3: uncompressed histogram is not a valid/supported format as far as I know If you can provide an example of rust generated histoblob (base64 compressed) that fails decoding in python, I can have a closer look. |
The data was the raw set of bytes for the histogram with no base64 encoding. The Rust encoder for compressed histograms does not appear to do base64 encoding. See https://github.com/HdrHistogram/HdrHistogram_rust/blob/89ea97afdfa543a6b7a0ebc8c7d03eddf66affb3/src/serialization/v2_deflate_serializer.rs#L75-L133
The Rust code is able to produce uncompressed histograms though. See https://github.com/HdrHistogram/HdrHistogram_rust/blob/89ea97afdfa543a6b7a0ebc8c7d03eddf66affb3/src/serialization/v2_serializer.rs#L67-L115
The Rust side of the project is here: https://github.com/tdyas/pants/blob/9f4e51cb0bc0293e56c7fa6376f7530d008ceaf5/src/rust/engine/workunit_store/src/lib.rs#L730-L756 On the Python side, I need to encode base64.b64encode on the raw bytes to go from the raw bytes to base64-encoding. Then the Python decoder works.
I have not. Maybe this is a bug in the Rust encoder where it fails to base64 encode?
I have not. The Python code is the part of the project that uploads histograms out of the Pants build tool into a server for histograms collected in the Rust engine. |
There are 4 kinds of encoding: V0, V1, V2, V2+Deflate. The Rust implementation currently supports the latter two. |
maybe we can discuss this over https://gitter.im/HdrHistogram/HdrHistogram ? |
@tdyas it looks like the only path that would work is if you generate on Rust side using hdrhistogram::serialization::V2DeflateSerializer (and without base64) (was not clear above which format you were using when you say b64_wrap=False did not work) To move forward, can you send an example of Rust generated compressed histogram (base64 version works) and I can have a look on my side why the python decode fails. |
Here is the failure with a compressed blob with a single observation (and the success once it has been base64 encoded). The value was produced by V2DeflateSerializer in the Rust library.
Here is the failure with an uncompressed blob produced by V2Serializer in the Rust library:
|
Yes I got the backtraces but I really need to get a hold on the buffer you pass to decode() so I can try to reproduce on my computer and decode it manually.
The "encoded" buffer, You can either print directly the result of hdrhistogram::serialization::V2DeflateSerializer with base64 |
The buffers are in there as Python
and:
|
And here they are converted to base64:
and:
|
ok here's what I found on the decode of a rust V2 compressed histogram.
However the non base 64 compressed (rust_compressed) fails.
As you can see the rust compressed buffer is 8 bytes too long (start of buffer), which explains why the deflate fails/. |
That's the v2 compressed cookie and the length. 0x1f is 31, which is the length of the buffer. |
The code path in the Lines 353 to 355 in 6462abf
|
I am generating histograms in Rust and am deserializing in Python using the HDR Histogram libraries for Rust and Python. The Rust code produces a byte array with the encoded histogram which ends up as a
bytes
instance in Python. (The project is a Python program that integrates with a Rust library via thecpython
crate.)It appears that the Python library is only able to decode the encoded histogram if Rust encodes using
hdrhistogram::serialization::V2DeflateSerializer
and further encodes it using base64 (via Python'sbase64.b64encode
).Without the base64 encoding, decoding with
histogram = HdrHistogram.decode(encoded_histogram, b64_wrap=False)
results in this error:Using uncompressed encoding (via
hdrhistogram::serialization::V2Serializer
in Rust) and base64 encoding in Python results in this error:And using uncompressed encoding without base64 results in:
The text was updated successfully, but these errors were encountered: