JSON representation of ProtoBufJS 'bytes' type

[barbareek]

In my project i am dynamically generating json structure from input fields:
Structure of my sample project is as follows:

1. create a form entry for all protobuf fields
2. all primitive types are displayed as text fields
   Now all work fine for string,uint,int etc
   But how to i represent bytes type. i tried pushing a object with bytes something like 3000 into {0: 0, 1: 0, 2: 0, 3: 0, 4: 11, 5: 184} also i tried setting it to [0,0,0,11,184] all resulting in errors like
   [Error: Cannot wrap buffer of type object, Object]
   ,[Error: Illegal value for Message.Field .protocol.ServerRequest.device_id: 3,1,0,0 (no array expected)] respectively

how exactly can we represent bytes in json

[dcodeIO]

Internally, bytes fields are represented by a ByteBuffer object, which in turn wraps either an ArrayBuffer in the browser or a node Buffer under node.js.

As these types are not valid in JSON, handling them requires special care. You might either convert them to hex or base64 encoding as a string and convert it back later on or develop a custom conversion.

Let's say you want to display the contents of a bytes field in a form as text. You'd then call str = ByteBuffer#toString('hex') to convert it for display, and ByteBuffer.fromString(str, 'hex') to convert it back to bytes.

[barbareek]

Well my server side code is

var obj = JSON.parse(json_from_client, null, 4);
// get actual message object
var classObj = new class(obj);
// get proto bytes
var buffer = classObj.encode();

since i am directly using the encoding on JSON from client , now do i need to parse through the JSON and for all bytes type fields do i need to convert them back from hex string to node buffer.

[dcodeIO]

I see. What you should be able to do currently is using an array of octets for a buffer field, which later on should properly be wrapped by a bytebuffer. For example, if you receive the following JSON:

{
   "id": "someid",
   "data": [1,2,3]
}

And data is defined as a bytes type, creating a message object from it should work. This isn't optimal though as the JSON string will probably become quite large for large data. Manually converting to base64 on the client, then converting back to a bytebuffer on the server prior to constructing the message object would be more efficient network-wise. Depends on your use case I guess.

[barbareek]

Thanks a lot , i will try this server side conversion (many of my bytes field belong to repeated messages type so may be i have to iterate them)

[apirila]

I have similar issue and I think that the current implementation is not optimal.

Currently it wraps the data to ByteBuffer with utf-8 encoding. If the results is "bytes", which indicates binary data, then the utf-8 encoding is probably not very suitable. The default encoding should be either base64 or binary. (in Field.verifyValue) The other possibility is to have the a parameter

[dcodeIO]

See: Changes in ProtoBuf.js 3.8

This will break any applications already using Message#toRaw(true) with binary data present or string assignments to bytes fields when switching to 3.8.0.