Buffer reuse

[fegies]

Avoid reallocating the Protocol message buffer by introducing a Trait that abstracts over the concrete stream type and accepts a mutable buffer.

Furthermore, the ByteConsumer Iterator impl is now based on the FramedRead.

Will update the PR with profiling data once I am back at work with access to the test dataset

There are some open questions:

• Does it make sense to update the Converter struct to use the new Trait also?
• Should the minor version of the stream-delimit subcrate be bumped? (there should be strictly new features)

EDIT:

Time after this PR:
$ time ./pq --msgtype simmo.TranshipmentCandidate --fdsetdir ./fdset --stream i32be < candidates.proto.ld > /dev/null
real 1m12.450s
user 1m11.133s
sys 0m1.047s

For comparison: achieved Result from previous PR:
$ time ./pq --msgtype simmo.TranshipmentCandidate --fdsetdir ./fdset --stream i32be < candidates.proto.ld > /dev/null
real 1m19.370s
user 1m14.457s
sys 0m4.568s

[fegies]

Not sure if its fine to just shunt all errors into the InvalidData kind

[fegies]

Here is a flamegraph after:

And here before (with the saved part crudely marked)

Looking at the graphs, it would seem there are still significant savings to be had in the serde_protobuf library, though I am not sure if this is feasible with justifiable effort

[sevagh]

Does it make sense to update the Converter struct to use the new Trait also?

I'll have to review this PR more closely so I can have an answer for this. It could take more time.

Should the minor version of the stream-delimit subcrate be bumped? (there should be strictly new features)

Unless you really want to, you don't need to worry about version bumps - after merging all the PRs, I can take care of the version bumps for the next release.

[fegies]

So, I had a thorough look at serde-protobuf, and found some unfortunate things:

• because of the protobuf spec, the reader should only keep the last entry for a given key. This requires at least buffering the entire message or deserializing and overwriting duplicate subobjects as the parser descends the structure.
• serde-protobuf chooses the latter, leading to a tree of keys being constructed internally for every message and submessage
• because the parser structure does not have any lifetime references to the parsed data, its not possible to buffer the message and descend lazily, or reuse sets or maps without major effort.
• this is complicated because it uses the protobuf crate, with the CodedInputStream as an abstration over a BufferedStream -> no zero-copy here possible either.

On the pq-side, I did not find any area where perf could be easily improved. (aside from changing the protobuf library).

So, aside from addressing comments from your side, I'd consider this PR series finished.

Cheers!

[sevagh]

Verified on my end with pmap and the new PR has much less memory used. Tests passing. Thanks for another great contribution.