rust-protobuf version 3

[stepancheg]

Issues to be fixed to release version 3

• Make most of Message functions non-virtual to reduce generated code size #339
• Remove RepeatedField #503
• Remove serde support from rust-protobuf-3 #519
• Dynamic Message #498
• Message field design #520
• Rename carllerche-bytes codegen option name to tokio-bytes #599
• Move protoc-bin-vendored into a separate repository #601
• Codegen option to create module hierarchy #438
• Support Field Presence (proto3) #513
• Change generated code for version 3 release #521
• Carefully work with integer overflow in serialization and deserialization #530
• Use underscore to escape keywords #618

Version 3 alpha versions are published. docs.rs

Open questions:

• Is current proto file placing OK? I. e. all files become first-level modules regardless of input directory structure and package declaration. I. e. foo/bar.proto becomes mod bar
• Do we need to rename .proto member to rust idiomatic names? I. e. do we need to rename enum variant FOO to Foo?
• Should we use raw identifiers for reserved names? Currently proto field type becomes field_type in generated code. Perhaps it should be r#type.
• What about MessageField type? It is convenient, but isn't it too much mental overhead to work with?
• Should we get rid of hack around figuring out how place messages in oneofs? They are boxed when there's a loop, and unboxed otherwise. I'm in favor of unconditionally making them boxed. Maybe later provide codegen option to store then unboxed.
• Is codegen customize callback API convenient?
• Shouldn't we pull JSON/text format parsing and serialization out of protobuf crate? For JSON we would be able to use third-party JSON parser more easily
• should we restrict generated code only to current version of rust-protobuf? Or guarantee forward compatibility with generated code as rust-protobuf v2 did? Not guaranteeing compatibility would make future changes around reflection much easier.

[saethlin]

I'm looking at a ~22% perf regression from 2.18.1 to the current master due to the changes in the semantics of Clear. Previously, I could use that trait to reuse all of a message's internal Vecs, but not anymore. It looks like this is caused by the change from SingularField to Option, which seems pretty deliberate but the docs for MessageField::clear still document the old behavior, which now contradicts the implementation:

rust-protobuf/protobuf/src/message_field.rs

Line 154 in 0f0956b

/// Clear this object, but do not call destructor of underlying data.

Is re-establishing the old Clear semantics a TODO item? Or is this some consequence of proto3 😕

[stepancheg]

@saethlin thank you for the feedback!

We can more or less easily restore old Clear semantics of MessageField (because it is already a new type).

But the issue is with RepeatedField. Current version of rust-protobuf 3 unconditionally uses Vec for repeated fields, and respectively does not preserve allocated objects. It was done because for users it is not always convenient to use RepeatedField and convert it to and from Vec.

So trade-off here is between:

• significantly faster performance when clear is used, and it is used rarely
• mental and syntax overhead working with RepeatedField

I honestly don't know what would be the right solution.

[saethlin]

Would it be too much mental overhead to have Clear declare that it only retains the first level of indirection? Personally I don't mind the mental model being too odd because this is entirely an optimization. But maybe the opinion of someone who actually uses proto3 would matter more.

[stepancheg]

Would it be too much mental overhead to have Clear declare that it only retains the first level of indirection?

You meant, only singular nested messages preserved, but not repeated nested messages? It can be done, but it will be only partial benefit, and still some complication: MessageField Option<Box<T>> field is public now. If it meant to preserve allocation, field should be private, somewhat less convenient.

BTW is protobuf really meant to be super-fast? I suspect for really good performance, probably flatbuffers or capnproto should be used (where "deserialization" is no-op).

But maybe the opinion of someone who actually uses proto3 would matter more.

The issue is the same in proto2 and proto3. Singular and repeated message fields work the same in proto2 and proto3.