A low-level, fast, ultra-lightweight (3KB gzipped) JavaScript library for decoding and encoding protocol buffers, a compact binary format for structured data serialization. Works both in Node and the browser. Supports lazy decoding and detailed customization of the reading/writing code.
This library is extremely fast — much faster than native JSON.parse
/JSON.stringify
and the protocol-buffers module.
Here's a result from running a real-world benchmark on Node v6.5
(decoding and encoding a sample of 439 vector tiles, 22.6 MB total):
- pbf decode: 387ms, or 57 MB/s
- pbf encode: 396ms, or 56 MB/s
- protocol-buffers decode: 837ms, or 26 MB/s
- protocol-buffers encode: 4197ms, or 5 MB/s
- JSON.parse: 1540ms, or 125 MB/s (parsing an equivalent 77.5 MB JSON file)
- JSON.stringify: 607ms, or 49 MB/s
Install pbf
and compile a JavaScript module from a .proto
file:
$ npm install -g pbf
$ pbf example.proto > example.js
Then read and write objects using the module like this:
var Pbf = require('pbf');
var Example = require('./example.js').Example;
// read
var pbf = new Pbf(buffer);
var obj = Example.read(pbf);
// write
var pbf = new Pbf();
Example.write(obj, pbf);
var buffer = pbf.finish();
Alternatively, you can compile a module directly in the code:
var compile = require('pbf/compile');
var schema = require('protocol-buffers-schema');
var proto = schema.parse(fs.readFileSync('example.proto'));
var Test = compile(proto).Test;
If you use webpack
as your module bundler, you can use pbf-loader
to load .proto files directly. It returns a compiled module ready to be used.
Given you already configured your webpack.config.js
, the code above would look like:
var Pbf = require('pbf');
var proto = require('./example.proto');
var Test = proto.Test;
var data = new Pbf(buffer).readFields(readData, {});
function readData(tag, data, pbf) {
if (tag === 1) data.name = pbf.readString();
else if (tag === 2) data.version = pbf.readVarint();
else if (tag === 3) data.layer = pbf.readMessage(readLayer, {});
}
function readLayer(tag, layer, pbf) {
if (tag === 1) layer.name = pbf.readString();
else if (tag === 3) layer.size = pbf.readVarint();
}
var pbf = new Pbf();
writeData(data, pbf);
var buffer = pbf.finish();
function writeData(data, pbf) {
pbf.writeStringField(1, data.name);
pbf.writeVarintField(2, data.version);
pbf.writeMessage(3, writeLayer, data.layer);
}
function writeLayer(layer, pbf) {
pbf.writeStringField(1, layer.name);
pbf.writeVarintField(2, layer.size);
}
Node and Browserify:
npm install pbf
Making a browser build:
npm install
npm run build-dev # dist/pbf-dev.js (development build)
npm run build-min # dist/pbf.js (minified production build)
CDN link: https://unpkg.com/[email protected]/dist/pbf.js
Create a Pbf
object, optionally given a Buffer
or Uint8Array
as input data:
// parse a pbf file from disk in Node
var pbf = new Pbf(fs.readFileSync('data.pbf'));
// parse a pbf file in a browser after an ajax request with responseType="arraybuffer"
var pbf = new Pbf(new Uint8Array(xhr.response));
Pbf
object properties:
pbf.length; // length of the underlying buffer
pbf.pos; // current offset for reading or writing
Read a sequence of fields:
pbf.readFields(function (tag) {
if (tag === 1) pbf.readVarint();
else if (tag === 2) pbf.readString();
else ...
});
It optionally accepts an object that will be passed to the reading function for easier construction of decoded data,
and also passes the Pbf
object as a third argument:
var result = pbf.readFields(callback, {})
function callback(tag, result, pbf) {
if (tag === 1) result.id = pbf.readVarint();
}
To read an embedded message, use pbf.readMessage(fn[, obj])
(in the same way as read
).
Read values:
var value = pbf.readVarint();
var str = pbf.readString();
var numbers = pbf.readPackedVarint();
For lazy or partial decoding, simply save the position instead of reading a value, then later set it back to the saved value and read:
var fooPos = -1;
pbf.readFields(function (tag) {
if (tag === 1) fooPos = pbf.pos;
});
...
pbf.pos = fooPos;
pbf.readMessage(readFoo);
Scalar reading methods:
readVarint(isSigned)
(passtrue
if you expect negative varints)readSVarint()
readFixed32()
readFixed64()
readSFixed32()
readSFixed64()
readBoolean()
readFloat()
readDouble()
readString()
readBytes()
skip(value)
Packed reading methods:
readPackedVarint(arr, isSigned)
(appends read items toarr
)readPackedSVarint(arr)
readPackedFixed32(arr)
readPackedFixed64(arr)
readPackedSFixed32(arr)
readPackedSFixed64(arr)
readPackedBoolean(arr)
readPackedFloat(arr)
readPackedDouble(arr)
Write values:
pbf.writeVarint(123);
pbf.writeString("Hello world");
Write an embedded message:
pbf.writeMessage(1, writeObj, obj);
function writeObj(obj, pbf) {
pbf.writeStringField(obj.name);
pbf.writeVarintField(obj.version);
}
Field writing methods:
writeVarintField(tag, val)
writeSVarintField(tag, val)
writeFixed32Field(tag, val)
writeFixed64Field(tag, val)
writeSFixed32Field(tag, val)
writeSFixed64Field(tag, val)
writeBooleanField(tag, val)
writeFloatField(tag, val)
writeDoubleField(tag, val)
writeStringField(tag, val)
writeBytesField(tag, buffer)
Packed field writing methods:
writePackedVarint(tag, val)
writePackedSVarint(tag, val)
writePackedSFixed32(tag, val)
writePackedSFixed64(tag, val)
writePackedBoolean(tag, val)
writePackedFloat(tag, val)
writePackedDouble(tag, val)
Scalar writing methods:
writeVarint(val)
writeSVarint(val)
writeSFixed32(val)
writeSFixed64(val)
writeBoolean(val)
writeFloat(val)
writeDouble(val)
writeString(val)
writeBytes(buffer)
Message writing methods:
writeMessage(tag, fn[, obj])
writeRawMessage(fn[, obj])
Misc methods:
realloc(minBytes)
- pad the underlying buffer size to accommodate the given number of bytes; note that the size increases exponentially, so it won't necessarily equal the size of data writtenfinish()
- make the current buffer ready for reading and return the data as a buffer slicedestroy()
- dispose the buffer
For an example of a real-world usage of the library, see vector-tile-js.
If installed globally, pbf
provides a binary that compiles proto
files into JavaScript modules. Usage:
$ pbf <proto_path> [--no-write] [--no-read] [--browser]
The --no-write
and --no-read
switches remove corresponding code in the output.
The --browser
switch makes the module work in browsers instead of Node.
The resulting module exports each message by name with the following methods:
read(pbf)
- decodes an object from the givenPbf
instancewrite(obj, pbf)
- encodes an object into the givenPbf
instance (usually empty)
The resulting code is clean and simple, so feel free to customize it.
- Significantly improved performance when decoding large strings in the browser.
- Improved decoding to be able to parse repeated fields even if they were specified as packed, and vise versa.
- Improved packed encoding to skip empty arrays (previously, it would write a tag).
- Fixed an off-by-one data corruption bug when writing a message larger than 0x10000000 bytes.
- Added support for Protocol Buffer 3 maps to proto compiler.
- Fixed an error appearing in some versions of IE11 and old Android browsers.
- Fixed compiling repeated packed enum fields.
- Fixed a regression that broke compiling repeated enum fields with defaults.
- Fixed a regression that broke decoding of packed fields with a tag that didn't fit into one byte.
- Fixed a regression that broke encoding of long strings.
This release include tons of compatibility/robustness fixes, and a more reliable Node implementation. Decoding performance is expected to get up to ~15% slower than v2.0 in Node (browsers are unaffected), but encoding got faster by ~15% in return.
- Breaking: changed Node implementation to use
Uint8Array
instead ofBuffer
internally (and produce corresponding result onfinish()
), making it fully match the browser implementation for consistency and simplicity. - Fixed
writeVarint
to write0
when givenNaN
or other non-number to avoid producing a broken Protobuf message. - Changed
readPacked*
methods signature to accept an optionalarr
argument to append the results to (to support messages with repeated fields that mix packed/non-packed encoding). - Added an optional
isSigned
argument toreadVarint
that enables proper reading of negative varints. - Deprecated
readVarint64()
(it still works, but it's recommended to be changed toreadVarint(true)
). - Faster string encoding.
- Breaking: Full support for defaults field values (both implicit and explicit); they're now included in the decoded JSON objects.
- Fixed reading of repeated fields with mixed packed/non-packed encoding for compatibility.
- Fixed proto3 compiler to use packed by default for repeated scalar fields.
- Fixed reading of negative varint types.
- Fixed packed fields to decode into
[]
if they're not present. - Fixed nested message references handling.
- Fixed
packed=false
being interpreted as packed. - Added a comment to generated code with pbf version number.
- Fixed a regression with
writeVarint
that affected certain numbers.
- Significantly improved the proto compiler, which now produces a much safer reading/writing code.
- Added the ability to compile a read/write module from a protobuf schema directly in the code.
- Proto compiler: fixed name resolutions and collisions in schemas with nested messages.
- Proto compiler: fixed broken top-level enums.
- Fixed a regression with
writeVarint
that affected certain numbers.
- Improved read and write performance (both ~15% faster).
- Improved generated code for default values.
- Added support for
syntax
keyword proto files (by updatingresolve-protobuf-schema
dependency).
- Added
writeRawMessage
method for writing a message without a tag, useful for creating pbfs with multiple top-level messages.
- Added
readVarint64
method for proper decoding of negativeint64
-encoded values.
- Fixed pbf proto compile tool generating broken writing code.
- Added
pbf
binary that compiles.proto
files intoPbf
-based JavaScript modules.
- Changed
writeMessage
signature to(tag, fn, obj)
(see example in the docs) for a huge encoding performance improvement. - Replaced
readPacked
andwritePacked
methods that accept type as a string withreadPackedVarint
, etc. for each type (better performance and simpler API).
- 5x faster encoding in Node (vector tile benchmark).
- 40x faster encoding and 3x faster decoding in the browser (vector tile benchmark).
- Significantly improved
readPacked
andwritePacked
performance (the tile reading benchmark is now 70% faster).
Brings tons of improvements and fixes over the previous version (0.0.2
).
Basically makes the library complete.
- Improved performance of both reading and writing.
- Made the browser build 3 times smaller.
- Added convenience
readFields
andreadMessage
methods for a much easier reading API. - Added reading methods:
readFloat
,readBoolean
,readSFixed32
,readSFixed64
. - Added writing methods:
writeUInt64
,writeSFixed32
,writeSFixed64
. - Improved
readDouble
andreadString
to use native Buffer methods under Node. - Improved
readString
andwriteString
to use HTML5TextEncoder
andTextDecoder
where available. - Made
Pbf
buffer
argument optional. - Added extensive docs and examples in the readme.
- Added an extensive test suite that brings test coverage up to 100%.
- Renamed
readBuffer
/writeBuffer
toreadBytes
/writeBytes
. - Renamed
readUInt32
/writeUInt32
toreadFixed32
/writeFixed32
, etc. - Renamed
writeTaggedVarint
towriteVarintField
, etc. - Changed
writePacked
signature from(type, tag, items)
to(tag, type, items)
.
- Fixed
readVarint
to handle varints bigger than 6 bytes. - Fixed
readSVarint
to handle number bigger than2^30
. - Fixed
writeVarint
failing on some integers. - Fixed
writeVarint
not throwing an error on numbers that are too big. - Fixed
readUInt64
always failing. - Fixed writing to an empty buffer always failing.