Support of google.protobuf.Any

[woigl]

I started using proto3, but it seems that "google.protobuf.Any" datatype is not yet supported. Please let me know what I am doing wrong, ori if that is still not yet implemented.

This is my proto file:

syntax = "proto3";

package vnd.proto;

message Test1 {
    uint32              key = 1;
    google.protobuf.Any value   = 2;
}

This is my code used to read the proto file:

var ProtoBuf = dcodeIO.ProtoBuf;

//Synchronously
var builder = ProtoBuf.loadProtoFile("test.proto"),
    proto = builder.build("vnd.proto");
    console.log(proto);

This is the error I receive:
Error: unresolvable type reference in Message.Field .vnd.proto.Test1.number: google.protobuf.Any

Thanks in Advance.

[dcodeIO]

This is not supported currently. Can you provide a link to the documentation passage on this?

[woigl]

Sure, here comes the link: https://developers.google.com/protocol-buffers/docs/proto3#any

[dcodeIO]

Thanks! What you can do already is to include google/protobuf/any.proto in your project, and import it within the file in question. Basically, it's just:

syntax = "proto3";
package google.protobuf;
message Any {
  string type_url = 1;
  bytes value = 2;
}

You are then able to work with the Any type manually until a better mechanism is in place in protobuf.js.

[woigl]

Thanks for the idea.
What timeframe you assume for the implementation of the Any support in protobuf.js?

[woigl]

I would highly appreciate, if someone could schedule this as a further enhancement.

[giorgosera]

👍 Need this too.

[giorgosera]

It's not very clear to me what's the optimal way to use the workaround mentioned above. I have a minimal working project below:

myservice.proto

syntax = "proto3";  

package myservice;

message Any {
  string type_url = 1;
  bytes value = 2;
}

message Request {
    string id = 1;
}

message SampleA {
    string fieldA = 1;
}

message SampleB {
    string fieldB = 1;
}

message Response {
    string id = 1;
    repeated Any samples = 2;
}

service MyService {
  rpc Get (Request) returns (Response);
}

samples.js

const ProtoBuf = require('protobufjs');
const builder = ProtoBuf.loadProtoFile('./myservice.proto');
const Any = builder.build('Any');
const SampleA = builder.build('SampleA');
const SampleB = builder.build('SampleB');
const Response = builder.build('Response');

const pack = (message, prefix) => {
  return new Any({
    type_url: path.join((prefix || ''), message.toString()),
    value: message.encode()
  });
};

const unpack = (message) => {
  const b = builder.build(message.type_url.split('.')[1]);
  return b.decode(message.value);
};

// Pack and create a response
const sampleA = new SampleA({fieldA: 'testa'});
const sampleB = new SampleA({fieldA: 'testa'});
const response = new Response({
  id: '1234',
  samples: [pack(sampleA), pack(sampleB)]
});


// Unpack
const samples = response.samples.map((sample) => {
    return unpack(sample);
  });

console.log(samples);

This works and I get back what I expected. However, I have a couple of questions:

1. Is there a better way to get the full name of a message to construct the type_url. I used .toString() in my code but I had a look at the implementations of other languages and messages seem to have some kind of getter for the name.
2. When encoding and decoding an Any message should I use .encode and .decode or is there a better way?

I appreciate your input on this!

[dcodeIO]

Closing this for now.

protobuf.js 6.0.0

Feel free to send a pull request if this is still a requirement.

[sp3c1]

is this working now?

[kheyse-werk]

It doesn't seem to work as expected yet: #764

[psaelango]

loadProtoFile and build are not working in the latest version (matrix-io/matrix-creator-malos#53). So can someone please provide example to use google.protobuf.any.

[txiaocao]

+1

[jianfenkezhan]

+1

[live680g]

+1

+1

[live680g]

It's likely that I find ways to solve the Support of google.protobuf.Any problem.

Below are the detail ways.

for example, i want to serialize a JS Object depending .proto file like this.

syntax = "proto3";
import "google/protobuf/any.proto";
package request

message Message {
  int32 version = 6;
  google.protobuf.Any body = 7;
}
message Heartbeat {
    int64 timestamp = 1;
    int32 expires = 2;
}

javascript snippet

// serializition
const msg = new proto.request.Message()
const hb = new proto.request.Heartbeat()
hb.setTimestamp('12312421412414')
const any = new proto.google.protobuf.Any()
any.pack(hb.serializeBinary(), 'request.Heartbeat')
msg.setBody(any)
console.log(msg.serializeBinary())

// deserializition. input: byte stream
const msgInstance = proto.request.Message.deserializeBinary(bytes)
const msg = msgInstance.toObject()
let bodyInstance = null
const bodyPacked = msgInstance.getBody()
if (bodyPacked) {
    bodyInstance = bodyPacked.unpack(proto.request.Heartbeat.deserializeBinary, 'request.Heartbeat')
}
console.log({
    ...msg,
    body: bodyInstance && bodyInstance.toObject()
})

[live680g]

I used google proto buf official solution instead of protobuf.js npm package, I also didn't find the solution to Support of google.protobuf.Any problem.

[live680g]

Finally, I used protobuf.js npm package to solve the Support of google.protobuf.Any problem.

1. Firstly, generated static code pbjs -t static-module -w commonjs -o compiled.js file1.proto file2.proto, pbts -o compiled.d.ts compiled.js
2. Secondly, import compiled.js file

import { request } './compiled';

// serializition
const message = request.Message.create({ version: 1 });
const heartbeat = request.Heartbeat.create({ timestamp: '123',  expires: 123}); 
message.body = {
  type_url: 'type.googleapis.com/dispatcher.Heartbeat',
  value: request.Heartbeat.encode(heartbeat).finish()
};
const buffer = request.Message.encode(message).finish();

// deserializition
const message = request.Message.decode(buffer);
const heartbeat = request.Heartbeat.decode(message.body.value);

[gaochongGithub]

helped me a lot, thanks a lot

[ansemjo]

While evaluating protobuf.js for a project, I also stumbled upon this issue in search for information how to use google.protobuf.Any and whether there are any pack/unpack methods. So I'll leave this info here, even if the issue is closed.

For packing a Message in a somewhat typesafe way in TypeScript, you could use this snippet:

interface SomeProtoType {
  new (...args: any[]): any;
  getTypeUrl(prefix: string): string;
  encode(message: InstanceType<this>): pbWriter;
}

function packAny<T extends SomeProtoType>(proto: T, instance: InstanceType<T> | ConstructorParameters<T>[0]) {
  return <proto.google.protobuf.IAny>{
    type_url: proto.getTypeUrl("prefix"),
    value: proto.encode(instance).finish(),
  };
};

// usage example:
import { google, HelloRequest } from "./messages";
let any: google.protobuf.IAny = packAny(HelloRequest, { name: "Alice" });

This is conceptually similar to what bufbuild does with its schema arguments. For unpacking, I resorted to a switch statement on the .type_url property for testing. Though you should be able to write something with lookup, which unpacks the correct type automatically, if it is known.

I also found wrappers for the google.protobuf.Any type in the code but I couldn't figure out how you're supposed to use those yet:

protobuf.js/src/wrappers.js

Lines 39 to 102 in 0a0cdb6

	wrappers[".google.protobuf.Any"] = {
	fromObject: function(object) {
	// unwrap value type if mapped
	if (object && object["@type"]) {
	// Only use fully qualified type name after the last '/'
	var name = object["@type"].substring(object["@type"].lastIndexOf("/") + 1);
	var type = this.lookup(name);
	/* istanbul ignore else */
	if (type) {
	// type_url does not accept leading "."
	var type_url = object["@type"].charAt(0) === "." ?
	object["@type"].slice(1) : object["@type"];
	// type_url prefix is optional, but path seperator is required
	if (type_url.indexOf("/") === -1) {
	type_url = "/" + type_url;
	}
	return this.create({
	type_url: type_url,
	value: type.encode(type.fromObject(object)).finish()
	});
	}
	}
	return this.fromObject(object);
	},
	toObject: function(message, options) {
	// Default prefix
	var googleApi = "type.googleapis.com/";
	var prefix = "";
	var name = "";
	// decode value if requested and unmapped
	if (options && options.json && message.type_url && message.value) {
	// Only use fully qualified type name after the last '/'
	name = message.type_url.substring(message.type_url.lastIndexOf("/") + 1);
	// Separate the prefix used
	prefix = message.type_url.substring(0, message.type_url.lastIndexOf("/") + 1);
	var type = this.lookup(name);
	/* istanbul ignore else */
	if (type)
	message = type.decode(message.value);
	}
	// wrap value if unmapped
	if (!(message instanceof this.ctor) && message instanceof Message) {
	var object = message.$type.toObject(message, options);
	var messageName = message.$type.fullName[0] === "." ?
	message.$type.fullName.slice(1) : message.$type.fullName;
	// Default to type.googleapis.com prefix if no prefix is used
	if (prefix === "") {
	prefix = googleApi;
	}
	name = prefix + messageName;
	object["@type"] = name;
	return object;
	}
	return this.toObject(message, options);
	}
	};