- 
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
FlatBuffers is an efficient cross platform serialization library for C++, Java, C#, Go, Python and JavaScript (C, PHP & Ruby in progress). It was originally created at Google for game development and other performance-critical applications.
-It is available as Open Source on GitHub under the Apache license, v2 (see LICENSE.txt).
-Why use FlatBuffers?
--
-
- Access to serialized data without parsing/unpacking - What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility). -
- Memory efficiency and speed - The only memory needed to access your data is that of the buffer. It requires 0 additional allocations (in C++, other languages may vary). FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the benchmarks for details. -
- Flexible - Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don't have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don't, and how you design data structures. -
- Tiny code footprint - Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details. -
- Strongly typed - Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you. -
Convenient to use - Generated C++ code allows for terse access & construction code. Then there's optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers).
-Java and Go code supports object-reuse. C# has efficient struct based accessors.
-
-- Cross platform code with no dependencies - C++ code will work with any recent gcc/clang and VS2010. Comes with build files for the tests & samples (Android .mk files, and cmake for all other platforms). -
Why not use Protocol Buffers, or .. ?
-Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/ unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocation. The code is an order of magnitude bigger, too. Protocol Buffers has neither optional text import/export nor schema language features like unions.
-But all the cool kids use JSON!
-JSON is very readable (which is why we use it as our optional text format) and very convenient when used together with dynamically typed languages (such as JavaScript). When serializing data from statically typed languages, however, JSON not only has the obvious drawback of runtime inefficiency, but also forces you to write more code to access data (counterintuitively) due to its dynamic-typing serialization system. In this context, it is only a better choice for systems that have very little to no information ahead of time about what data needs to be stored.
-Read more about the "why" of FlatBuffers in the white paper.
-Who uses FlatBuffers?
--
-
- Cocos2d-x, the #1 open source mobile game engine, uses it to serialize all their game data. -
- Facebook uses it for client-server communication in their Android app. They have a nice article explaining how it speeds up loading their posts. -
- Fun Propulsion Labs at Google uses it extensively in all their libraries and games. -
Usage in brief
-This section is a quick rundown of how to use this system. Subsequent sections provide a more in-depth usage guide.
--
-
- Write a schema file that allows you to define the data structures you may want to serialize. Fields can have a scalar type (ints/floats of all sizes), or they can be a: string; array of any type; reference to yet another object; or, a set of possible objects (unions). Fields are optional and have defaults, so they don't need to be present for every object instance. -
- Use
flatc(the FlatBuffer compiler) to generate a C++ header (or Java/C#/Go/Python.. classes) with helper classes to access and construct serialized data. This header (saymydata_generated.h) only depends onflatbuffers.h, which defines the core functionality.
- - Use the
FlatBufferBuilderclass to construct a flat binary buffer. The generated functions allow you to add objects to this buffer recursively, often as simply as making a single function call.
- - Store or send your buffer somewhere! -
- When reading it back, you can obtain the pointer to the root object from the binary buffer, and from there traverse it conveniently in-place with
object->field().
-
In-depth documentation
--
-
- How to build the compiler and samples on various platforms. -
- How to use the compiler. -
- How to write a schema. -
- How to use the generated C++ code in your own programs. -
- How to use the generated Java/C# code in your own programs. -
- How to use the generated Go code in your own programs. -
- Support matrix for platforms/languages/features. -
- Some benchmarks showing the advantage of using FlatBuffers. -
- A white paper explaining the "why" of FlatBuffers. -
- A description of the internals of FlatBuffers. -
- A formal grammar of the schema language. -
Online resources
--
-
- GitHub repository -
- Landing page -
- FlatBuffers Google Group -
- FlatBuffers Issues Tracker -
- Independent implementations & tools:
-
-
- FlatCC Alternative FlatBuffers parser, code generator and runtime all in C. -
- - Videos:
-
-
- Colt's DevByte. -
- GDC 2015 Lightning Talk. -
- FlatBuffers for Go. -
- Evolution of FlatBuffers visualization. -
- - Useful documentation created by others: - -
| t |
|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
Comparing against other serialization solutions, running on Windows 7 64bit. We use the LITE runtime for Protocol Buffers (less code / lower overhead), Rapid JSON (one of the fastest C++ JSON parsers around), and pugixml, also one of the fastest XML parsers.
-We also compare against code that doesn't use a serialization library at all (the column "Raw structs"), which is what you get if you write hardcoded code that just writes structs. This is the fastest possible, but of course is not cross platform nor has any kind of forwards / backwards compatibility.
-We compare against Flatbuffers with the binary wire format (as intended), and also with JSON as the wire format with the optional JSON parser (which, using a schema, parses JSON into a binary buffer that can then be accessed as before).
-The benchmark object is a set of about 10 objects containing an array, 4 strings, and a large variety of int/float scalar values of all sizes, meant to be representative of game data, e.g. a scene format.
-| FlatBuffers (binary) | Protocol Buffers LITE | Rapid JSON | FlatBuffers (JSON) | pugixml | Raw structs | |
|---|---|---|---|---|---|---|
| Decode + Traverse + Dealloc (1 million times, seconds) | 0.08 | 302 | 583 | 105 | 196 | 0.02 |
| Decode / Traverse / Dealloc (breakdown) | 0 / 0.08 / 0 | 220 / 0.15 / 81 | 294 / 0.9 / 287 | 70 / 0.08 / 35 | 41 / 3.9 / 150 | 0 / 0.02 / 0 |
| Encode (1 million times, seconds) | 3.2 | 185 | 650 | 169 | 273 | 0.15 |
| Wire format size (normal / zlib, bytes) | 344 / 220 | 228 / 174 | 1475 / 322 | 1029 / 298 | 1137 / 341 | 312 / 187 |
| Memory needed to store decoded wire (bytes / blocks) | 0 / 0 | 760 / 20 | 65689 / 4 | 328 / 1 | 34194 / 3 | 0 / 0 |
| Transient memory allocated during decode (KB) | 0 | 1 | 131 | 4 | 34 | 0 |
| Generated source code size (KB) | 4 | 61 | 0 | 4 | 0 | 0 |
| Field access in handwritten traversal code | typed accessors | typed accessors | manual error checking | typed accessors | manual error checking | typed but no safety |
| Library source code (KB) | 15 | some subset of 3800 | 87 | 43 | 327 | 0 |
Some other serialization systems we compared against but did not benchmark (yet), in rough order of applicability:
--
-
- Cap'n'Proto promises to reduce Protocol Buffers much like FlatBuffers does, though with a more complicated binary encoding and less flexibility (no optional fields to allow deprecating fields or serializing with missing fields for which defaults exist). It currently also isn't fully cross-platform portable (lack of VS support). -
- msgpack: has very minimal forwards/backwards compatibility support when used with the typed C++ interface. Also lacks VS2010 support. -
- Thrift: very similar to Protocol Buffers, but appears to be less efficient, and have more dependencies. -
- YAML: a superset of JSON and otherwise very similar. Used by e.g. Unity. -
- C# comes with built-in serialization functionality, as used by Unity also. Being tied to the language, and having no automatic versioning support limits its applicability. -
- Project Anarchy (the free mobile engine by Havok) comes with a serialization system, that however does no automatic versioning (have to code around new fields manually), is very much tied to the rest of the engine, and works without a schema to generate code (tied to your C++ class definition). -
|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
There are project files for Visual Studio and Xcode that should allow you to build the compiler flatc, the samples and the tests out of the box.
Alternatively, the distribution comes with a cmake file that should allow you to build project/make files for any platform. For details on cmake, see http://www.cmake.org. In brief, depending on your platform, use one of e.g.:
cmake -G "Unix Makefiles" -cmake -G "Visual Studio 10" -cmake -G "Xcode" -
Then, build as normal for your platform. This should result in a flatc executable, essential for the next steps. Note that to use clang instead of gcc, you may need to set up your environment variables, e.g. CC=/usr/bin/clang CXX=/usr/bin/clang++ cmake -G "Unix Makefiles".
Optionally, run the flattests executable to ensure everything is working correctly on your system. If this fails, please contact us!
Note that you MUST be in the root of the FlatBuffers distribution when you run 'flattests' (and the samples), or it will fail to load its files.
-Building should also produce two sample executables, sample_binary and sample_text, see the corresponding .cpp file in the samples directory.
There is an android directory that contains all you need to build the test executable on android (use the included build_apk.sh script, or use ndk_build / adb etc. as usual). Upon running, it will output to the log if tests succeeded or not.
There is usually no runtime to compile, as the code consists of a single header, include/flatbuffers/flatbuffers.h. You should add the include folder to your include paths. If you wish to be able to load schemas and/or parse text into binary buffers at runtime, you additionally need the other headers in include/flatbuffers. You must also compile/link src/idl_parser.cpp (and src/idl_gen_text.cpp if you also want to be able convert binary to text).
For applications on Google Play that integrate this library, usage is tracked. This tracking is done automatically using the embedded version string (flatbuffer_version_string), and helps us continue to optimize it. Aside from consuming a few extra bytes in your application binary, it shouldn't affect your application at all. We use this information to let us know if FlatBuffers is useful and if we should continue to invest in it. Since this is open source, you are free to remove the version string but we would appreciate if you would leave it in.
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
Usage:
flatc [ GENERATOR OPTIONS ] [ -o PATH ] [ -I PATH ] [ -S ] FILES... - [ -- FILES...] -
The files are read and parsed in order, and can contain either schemas or data (see below). Data files are processed according to the definitions of the most recent schema specified.
--- indicates that the following files are binary files in FlatBuffer format conforming to the schema indicated before it.
Depending on the flags passed, additional files may be generated for each file processed:
-For any schema input files, one or more generators can be specified:
--
-
--cpp,-c: Generate a C++ header for all definitions in this file (asfilename_generated.h).
---java,-j: Generate Java code.
---csharp,-n: Generate C# code.
---go,-g: Generate Go code.
---python,-p: Generate Python code.
---javascript,-s: Generate JavaScript code.
---php: Generate PHP code.
-
For any data input files:
--
-
--binary,-b: If data is contained in this file, generate afilename.bincontaining the binary flatbuffer (or a different extension if one is specified in the schema).
---json,-t: If data is contained in this file, generate afilename.jsonrepresenting the data in the flatbuffer.
-
Additional options:
--
-
-o PATH: Output all generated files to PATH (either absolute, or relative to the current directory). If omitted, PATH will be the current directory. PATH should end in your systems path separator, e.g./or\.
--I PATH: when encounteringincludestatements, attempt to load the files from this path. Paths will be tried in the order given, and if all fail (or none are specified) it will try to load relative to the path of the schema file being parsed.
--M: Print make rules for generated files.
---strict-json: Require & generate strict JSON (field names are enclosed in quotes, no trailing commas in tables/vectors). By default, no quotes are required/generated, and trailing commas are allowed.
---defaults-json: Output fields whose value is equal to the default value when writing JSON text.
---no-prefix: Don't prefix enum values in generated C++ by their enum type.
---scoped-enums: Use C++11 style scoped and strongly typed enums in generated C++. This also implies--no-prefix.
---gen-includes: (deprecated), this is the default behavior. If the original behavior is required (no include statements) use--no-includes.
---no-includes: Don't generate include statements for included schemas the generated file depends on (C++).
---gen-mutable: Generate additional non-const accessors for mutating FlatBuffers in-place.
---gen-onefile: Generate single output file (useful for C#)
---gen-all: Generate not just code for the current schema files, but for all files it includes as well. If the language uses a single file for output (by default the case for C++ and JS), all code will end up in this one file.
---raw-binary: Allow binaries without a file_indentifier to be read. This may crash flatc given a mismatched schema.
---proto: Expect input files to be .proto files (protocol buffers). Output the corresponding .fbs file. Currently supports:package,message,enum, nested declarations,import(use-Ifor paths),extend,oneof,group. Does not support, but will skip without error:option,service,extensions, and most everything else.
---schema: Serialize schemas instead of JSON (use with -b). This will output a binary version of the specified schema that itself corresponds to the reflection/reflection.fbs schema. Loading this binary file is the basis for reflection functionality.
-
NOTE: short-form options for generators are deprecated, use the long form whenever possible.
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
Assuming you have written a schema using the above language in say mygame.fbs (FlatBuffer Schema, though the extension doesn't matter), you've generated a C++ header called mygame_generated.h using the compiler (e.g. flatc -c mygame.fbs), you can now start using this in your program by including the header. As noted, this header relies on flatbuffers/flatbuffers.h, which should be in your include path.
Writing in C++
-To start creating a buffer, create an instance of FlatBufferBuilder which will contain the buffer as it grows:
Before we serialize a Monster, we need to first serialize any objects that are contained there-in, i.e. we serialize the data tree using depth first, pre-order traversal. This is generally easy to do on any tree structures. For example:
-CreateString and CreateVector serialize these two built-in datatypes, and return offsets into the serialized data indicating where they are stored, such that Monster below can refer to them.
CreateString can also take an std::string, or a const char * with an explicit length, and is suitable for holding UTF-8 and binary data if needed.
CreateVector can also take an std::vector. The offset it returns is typed, i.e. can only be used to set fields of the correct type below. To create a vector of struct objects (which will be stored as contiguous memory in the buffer, use CreateVectorOfStructs instead.
To create a vector of nested objects (e.g. tables, strings or other vectors) collect their offsets in a temporary array/vector, then call CreateVector on that (see e.g. the array of strings example in test.cpp CreateFlatBufferTest).
Vec3 is the first example of code from our generated header. Structs (unlike tables) translate to simple structs in C++, so we can construct them in a familiar way.
We have now serialized the non-scalar components of of the monster example, so we could create the monster something like this:
-Note that we're passing 150 for the mana field, which happens to be the default value: this means the field will not actually be written to the buffer, since we'll get that value anyway when we query it. This is a nice space savings, since it is very common for fields to be at their default. It means we also don't need to be scared to add fields only used in a minority of cases, since they won't bloat up the buffer sizes if they're not actually used.
We do something similarly for the union field test by specifying a 0 offset and the NONE enum value (part of every union) to indicate we don't actually want to write this field. You can use 0 also as a default for other non-scalar types, such as strings, vectors and tables. To pass an actual table, pass a preconstructed table as mytable.Union() that corresponds to union enum you're passing.
Tables (like Monster) give you full flexibility on what fields you write (unlike Vec3, which always has all fields set because it is a struct). If you want even more control over this (i.e. skip fields even when they are not default), instead of the convenient CreateMonster call we can also build the object field-by-field manually:
We start with a temporary helper class MonsterBuilder (which is defined in our generated code also), then call the various add_ methods to set fields, and Finish to complete the object. This is pretty much the same code as you find inside CreateMonster, except we're leaving out a few fields. Fields may also be added in any order, though orderings with fields of the same size adjacent to each other most efficient in size, due to alignment. You should not nest these Builder classes (serialize your data in pre-order).
Regardless of whether you used CreateMonster or MonsterBuilder, you now have an offset to the root of your data, and you can finish the buffer using:
The buffer is now ready to be stored somewhere, sent over the network, be compressed, or whatever you'd like to do with it. You can access the start of the buffer with fbb.GetBufferPointer(), and it's size from fbb.GetSize().
Calling code may take ownership of the buffer with fbb.ReleaseBufferPointer(). Should you do it, the FlatBufferBuilder will be in an invalid state, and must be cleared before it can be used again. However, it also means you are able to destroy the builder while keeping the buffer in your application.
samples/sample_binary.cpp is a complete code sample similar to the code above, that also includes the reading code below.
Reading in C++
-If you've received a buffer from somewhere (disk, network, etc.) you can directly start traversing it using:
-monster is of type Monster *, and points to somewhere inside your buffer (root object pointers are not the same as buffer_pointer !). If you look in your generated header, you'll see it has convenient accessors for all fields, e.g.
These should all be true. Note that we never stored a mana value, so it will return the default.
To access sub-objects, in this case the Vec3:
If we had not set the pos field during serialization, it would be NULL.
Similarly, we can access elements of the inventory array:
-Mutating FlatBuffers
-As you saw above, typically once you have created a FlatBuffer, it is read-only from that moment on. There are however cases where you have just received a FlatBuffer, and you'd like to modify something about it before sending it on to another recipient. With the above functionality, you'd have to generate an entirely new FlatBuffer, while tracking what you modify in your own data structures. This is inconvenient.
-For this reason FlatBuffers can also be mutated in-place. While this is great for making small fixes to an existing buffer, you generally want to create buffers from scratch whenever possible, since it is much more efficient and the API is much more general purpose.
-To get non-const accessors, invoke flatc with --gen-mutable.
Similar to the reading API above, you now can:
-We use the somewhat verbose term mutate instead of set to indicate that this is a special use case, not to be confused with the default way of constructing FlatBuffer data.
After the above mutations, you can send on the FlatBuffer to a new recipient without any further work!
-Note that any mutate_ functions on tables return a bool, which is false if the field we're trying to set isn't present in the buffer. Fields are not present if they weren't set, or even if they happen to be equal to the default value. For example, in the creation code above we set the mana field to 150, which is the default value, so it was never stored in the buffer. Trying to call mutate_mana() on such data will return false, and the value won't actually be modified!
One way to solve this is to call ForceDefaults() on a FlatBufferBuilder to force all fields you set to actually be written. This of course increases the size of the buffer somewhat, but this may be acceptable for a mutable buffer.
Alternatively, you can use the more powerful reflection functionality:
-Reflection (& Resizing)
-If the above ways of accessing a buffer are still too static for you, there is experimental support for reflection in FlatBuffers, allowing you to read and write data even if you don't know the exact format of a buffer, and even allows you to change sizes of strings and vectors in-place.
-The way this works is very elegant, there is actually a FlatBuffer schema that describes schemas (!) which you can find in reflection/reflection.fbs. The compiler flatc can write out any schemas it has just parsed as a binary FlatBuffer, corresponding to this meta-schema.
Loading in one of these binary schemas at runtime allows you traverse any FlatBuffer data that corresponds to it without knowing the exact format. You can query what fields are present, and then read/write them after.
-For convenient field manipulation, you can include the header flatbuffers/reflection.h which includes both the generated code from the meta schema, as well as a lot of helper functions.
And example of usage for the moment you can find in test.cpp/ReflectionTest().
Storing maps / dictionaries in a FlatBuffer
-FlatBuffers doesn't support maps natively, but there is support to emulate their behavior with vectors and binary search, which means you can have fast lookups directly from a FlatBuffer without having to unpack your data into a std::map or similar.
To use it:
-
-
- Designate one of the fields in a table as they "key" field. You do this by setting the
keyattribute on this field, e.g.name:string (key). You may only have one key field, and it must be of string or scalar type.
- - Write out tables of this type as usual, collect their offsets in an array or vector. -
- Instead of
CreateVector, callCreateVectorOfSortedTables, which will first sort all offsets such that the tables they refer to are sorted by the key field, then serialize it.
- - Now when you're accessing the FlatBuffer, you can use
Vector::LookupByKeyinstead of justVector::Getto access elements of the vector, e.g.:myvector->LookupByKey("Fred"), which returns a pointer to the corresponding table type, ornullptrif not found.LookupByKeyperforms a binary search, so should have a similar speed tostd::map, though may be faster because of better caching.LookupByKeyonly works if the vector has been sorted, it will likely not find elements if it hasn't been sorted.
-
Direct memory access
-As you can see from the above examples, all elements in a buffer are accessed through generated accessors. This is because everything is stored in little endian format on all platforms (the accessor performs a swap operation on big endian machines), and also because the layout of things is generally not known to the user.
-For structs, layout is deterministic and guaranteed to be the same accross platforms (scalars are aligned to their own size, and structs themselves to their largest member), and you are allowed to access this memory directly by using sizeof() and memcpy on the pointer to a struct, or even an array of structs.
To compute offsets to sub-elements of a struct, make sure they are a structs themselves, as then you can use the pointers to figure out the offset without having to hardcode it. This is handy for use of arrays of structs with calls like glVertexAttribPointer in OpenGL or similar APIs.
It is important to note is that structs are still little endian on all machines, so only use tricks like this if you can guarantee you're not shipping on a big endian machine (an assert(FLATBUFFERS_LITTLEENDIAN) would be wise).
Access of untrusted buffers
-The generated accessor functions access fields over offsets, which is very quick. These offsets are not verified at run-time, so a malformed buffer could cause a program to crash by accessing random memory.
-When you're processing large amounts of data from a source you know (e.g. your own generated data on disk), this is acceptable, but when reading data from the network that can potentially have been modified by an attacker, this is undesirable.
-For this reason, you can optionally use a buffer verifier before you access the data. This verifier will check all offsets, all sizes of fields, and null termination of strings to ensure that when a buffer is accessed, all reads will end up inside the buffer.
-Each root type will have a verification function generated for it, e.g. for Monster, you can call:
if ok is true, the buffer is safe to read.
Besides untrusted data, this function may be useful to call in debug mode, as extra insurance against data being corrupted somewhere along the way.
-While verifying a buffer isn't "free", it is typically faster than a full traversal (since any scalar data is not actually touched), and since it may cause the buffer to be brought into cache before reading, the actual overhead may be even lower than expected.
-In specialized cases where a denial of service attack is possible, the verifier has two additional constructor arguments that allow you to limit the nesting depth and total amount of tables the verifier may encounter before declaring the buffer malformed. The default is Verifier(buf, len, 64 /* max depth */, 1000000, /* max tables */) which should be sufficient for most uses.
Text & schema parsing
-Using binary buffers with the generated header provides a super low overhead use of FlatBuffer data. There are, however, times when you want to use text formats, for example because it interacts better with source control, or you want to give your users easy access to data.
-Another reason might be that you already have a lot of data in JSON format, or a tool that generates JSON, and if you can write a schema for it, this will provide you an easy way to use that data directly.
-(see the schema documentation for some specifics on the JSON format accepted).
-There are two ways to use text formats:
-Using the compiler as a conversion tool
-This is the preferred path, as it doesn't require you to add any new code to your program, and is maximally efficient since you can ship with binary data. The disadvantage is that it is an extra step for your users/developers to perform, though you might be able to automate it.
flatc -b myschema.fbs mydata.json -
This will generate the binary file mydata_wire.bin which can be loaded as before.
Making your program capable of loading text directly
-This gives you maximum flexibility. You could even opt to support both, i.e. check for both files, and regenerate the binary from text when required, otherwise just load the binary.
-This option is currently only available for C++, or Java through JNI.
-As mentioned in the section "Building" above, this technique requires you to link a few more files into your program, and you'll want to include flatbuffers/idl.h.
Load text (either a schema or json) into an in-memory buffer (there is a convenient LoadFile() utility function in flatbuffers/util.h if you wish). Construct a parser:
Now you can parse any number of text files in sequence:
-This works similarly to how the command-line compiler works: a sequence of files parsed by the same Parser object allow later files to reference definitions in earlier files. Typically this means you first load a schema file (which populates Parser with definitions), followed by one or more JSON files.
As optional argument to Parse, you may specify a null-terminated list of include paths. If not specified, any include statements try to resolve from the current directory.
If there were any parsing errors, Parse will return false, and Parser::err contains a human readable error string with a line number etc, which you should present to the creator of that file.
After each JSON file, the Parser::fbb member variable is the FlatBufferBuilder that contains the binary buffer version of that file, that you can access as described above.
samples/sample_text.cpp is a code sample showing the above operations.
Threading
-Reading a FlatBuffer does not touch any memory outside the original buffer, and is entirely read-only (all const), so is safe to access from multiple threads even without synchronisation primitives.
-Creating a FlatBuffer is not thread safe. All state related to building a FlatBuffer is contained in a FlatBufferBuilder instance, and no memory outside of it is touched. To make this thread safe, either do not share instances of FlatBufferBuilder between threads (recommended), or manually wrap it in synchronisation primites. There's no automatic way to accomplish this, by design, as we feel multithreaded construction of a single buffer will be rare, and synchronisation overhead would be costly.
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
There's experimental support for reading FlatBuffers in Go. Generate code for Go with the -g option to flatc.
See go_test.go for an example. You import the generated code, read a FlatBuffer binary file into a []byte, which you pass to the GetRootAsMonster function:
Now you can access values like this:
-Note that whenever you access a new object like in the Pos example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), you can replace nil with a pointer to a Vec3 object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.
To access vectors you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access:
You can also construct these buffers in Go using the functions found in the generated code, and the FlatBufferBuilder class:
-Create strings:
-Create a table with a struct contained therein:
-Unlike C++, Go does not support table creation functions like 'createMonster()'. This is to create the buffer without using temporary object allocation (since the Vec3 is an inline component of Monster, it has to be created right where it is added, whereas the name and the inventory are not inline, and must be created outside of the table creation sequence). Structs do have convenient methods that allow you to construct them in one call. These also have arguments for nested structs, e.g. if a struct has a field a and a nested struct field b (which has fields c and d), then the arguments will be a, c and d.
Vectors also use this start/end pattern to allow vectors of both scalar types and structs:
-The generated method 'StartInventoryVector' is provided as a convenience function which calls 'StartVector' with the correct element size of the vector type which in this case is 'ubyte' or 1 byte per vector element. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front. Use the correct Prepend call for the type, or PrependUOffsetT for offsets. You then pass inv to the corresponding Add call when you construct the table containing it afterwards.
There are Prepend functions for all the scalar types. You use PrependUOffset for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate create function in-line, as shown above in the Monster example.
Once you're done constructing a buffer, you call Finish with the root object offset (mon in the example above). Your data now resides in Builder.Bytes. Important to note is that the real data starts at the index indicated by Head(), for Offset() bytes (this is because the buffer is constructed backwards). If you wanted to read the buffer right after creating it (using GetRootAsMonster above), the second argument, instead of 0 would thus also be Head().
Text Parsing
-There currently is no support for parsing text (Schema's and JSON) directly from Go, though you could use the C++ parser through cgo. Please see the C++ documentation for more on text parsing.
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
schema = include* ( namespace_decl | type_decl | enum_decl | root_decl | file_extension_decl | file_identifier_decl | attribute_decl | object )*
-include = include string_constant ;
namespace_decl = namespace ident ( . ident )* ;
attribute_decl = attribute string_constant ;
type_decl = ( table | struct ) ident metadata { field_decl+ }
enum_decl = ( enum | union ) ident [ : type ] metadata { commasep( enumval_decl ) }
root_decl = root_type ident ;
field_decl = ident : type [ = scalar ] metadata ;
type = bool | byte | ubyte | short | ushort | int | uint | float | long | ulong | double | string | [ type ] | ident
enumval_decl = ident [ = integer_constant ]
metadata = [ ( commasep( ident [ : single_value ] ) ) ]
scalar = integer_constant | float_constant
-object = { commasep( ident : value ) }
single_value = scalar | string_constant
-value = single_value | object | [ commasep( value ) ]
commasep(x) = [ x ( , x )* ]
file_extension_decl = file_extension string_constant ;
file_identifier_decl = file_identifier string_constant ;
integer_constant = -?[0-9]+ | true | false
float_constant = -?[0-9]+.[0-9]+((e|E)(+|-)?[0-9]+)?
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
This section is entirely optional for the use of FlatBuffers. In normal usage, you should never need the information contained herein. If you're interested however, it should give you more of an appreciation of why FlatBuffers is both efficient and convenient.
-Format components
-A FlatBuffer is a binary file and in-memory format consisting mostly of scalars of various sizes, all aligned to their own size. Each scalar is also always represented in little-endian format, as this corresponds to all commonly used CPUs today. FlatBuffers will also work on big-endian machines, but will be slightly slower because of additional byte-swap intrinsics.
-On purpose, the format leaves a lot of details about where exactly things live in memory undefined, e.g. fields in a table can have any order, and objects to some extend can be stored in many orders. This is because the format doesn't need this information to be efficient, and it leaves room for optimization and extension (for example, fields can be packed in a way that is most compact). Instead, the format is defined in terms of offsets and adjacency only. This may mean two different implementations may produce different binaries given the same input values, and this is perfectly valid.
-Format identification
-The format also doesn't contain information for format identification and versioning, which is also by design. FlatBuffers is a statically typed system, meaning the user of a buffer needs to know what kind of buffer it is. FlatBuffers can of course be wrapped inside other containers where needed, or you can use its union feature to dynamically identify multiple possible sub-objects stored. Additionally, it can be used together with the schema parser if full reflective capabilities are desired.
-Versioning is something that is intrinsically part of the format (the optionality / extensibility of fields), so the format itself does not need a version number (it's a meta-format, in a sense). We're hoping that this format can accommodate all data needed. If format breaking changes are ever necessary, it would become a new kind of format rather than just a variation.
-Offsets
-The most important and generic offset type (see flatbuffers.h) is uoffset_t, which is currently always a uint32_t, and is used to refer to all tables/unions/strings/vectors (these are never stored in-line). 32bit is intentional, since we want to keep the format binary compatible between 32 and 64bit systems, and a 64bit offset would bloat the size for almost all uses. A version of this format with 64bit (or 16bit) offsets is easy to set when needed. Unsigned means they can only point in one direction, which typically is forward (towards a higher memory location). Any backwards offsets will be explicitly marked as such.
The format starts with an uoffset_t to the root object in the buffer.
We have two kinds of objects, structs and tables.
-Structs
-These are the simplest, and as mentioned, intended for simple data that benefits from being extra efficient and doesn't need versioning / extensibility. They are always stored inline in their parent (a struct, table, or vector) for maximum compactness. Structs define a consistent memory layout where all components are aligned to their size, and structs aligned to their largest scalar member. This is done independent of the alignment rules of the underlying compiler to guarantee a cross platform compatible layout. This layout is then enforced in the generated code.
-Tables
-Unlike structs, these are not stored in inline in their parent, but are referred to by offset.
-They start with an soffset_t to a vtable. This is a signed version of uoffset_t, since vtables may be stored anywhere relative to the object. This offset is substracted (not added) from the object start to arrive at the vtable start. This offset is followed by all the fields as aligned scalars (or offsets). Unlike structs, not all fields need to be present. There is no set order and layout.
To be able to access fields regardless of these uncertainties, we go through a vtable of offsets. Vtables are shared between any objects that happen to have the same vtable values.
-The elements of a vtable are all of type voffset_t, which is a uint16_t. The first element is the size of the vtable in bytes, including the size element. The second one is the size of the object, in bytes (including the vtable offset). This size could be used for streaming, to know how many bytes to read to be able to access all inline fields of the object. The remaining elements are the N offsets, where N is the amount of fields declared in the schema when the code that constructed this buffer was compiled (thus, the size of the table is N + 2).
All accessor functions in the generated code for tables contain the offset into this table as a constant. This offset is checked against the first field (the number of elements), to protect against newer code reading older data. If this offset is out of range, or the vtable entry is 0, that means the field is not present in this object, and the default value is return. Otherwise, the entry is used as offset to the field to be read.
-Strings and Vectors
-Strings are simply a vector of bytes, and are always null-terminated. Vectors are stored as contiguous aligned scalar elements prefixed by a 32bit element count (not including any null termination). Neither is stored inline in their parent, but are referred to by offset.
-Construction
-The current implementation constructs these buffers backwards (starting at the highest memory address of the buffer), since that significantly reduces the amount of bookkeeping and simplifies the construction API.
-Code example
-Here's an example of the code that gets generated for the samples/monster.fbs. What follows is the entire file, broken up by comments:
// automatically generated, do not modify
-
-#include "flatbuffers/flatbuffers.h"
-
-namespace MyGame {
-namespace Sample {
-Nested namespace support.
enum {
- Color_Red = 0,
- Color_Green = 1,
- Color_Blue = 2,
-};
-
-inline const char **EnumNamesColor() {
- static const char *names[] = { "Red", "Green", "Blue", nullptr };
- return names;
-}
-
-inline const char *EnumNameColor(int e) { return EnumNamesColor()[e]; }
-Enums and convenient reverse lookup.
enum {
- Any_NONE = 0,
- Any_Monster = 1,
-};
-
-inline const char **EnumNamesAny() {
- static const char *names[] = { "NONE", "Monster", nullptr };
- return names;
-}
-
-inline const char *EnumNameAny(int e) { return EnumNamesAny()[e]; }
-Unions share a lot with enums.
struct Vec3; -struct Monster; -
Predeclare all data types since circular references between types are allowed (circular references between object are not, though).
MANUALLY_ALIGNED_STRUCT(4) Vec3 {
- private:
- float x_;
- float y_;
- float z_;
-
- public:
- Vec3(float x, float y, float z)
- : x_(flatbuffers::EndianScalar(x)), y_(flatbuffers::EndianScalar(y)), z_(flatbuffers::EndianScalar(z)) {}
-
- float x() const { return flatbuffers::EndianScalar(x_); }
- float y() const { return flatbuffers::EndianScalar(y_); }
- float z() const { return flatbuffers::EndianScalar(z_); }
-};
-STRUCT_END(Vec3, 12);
-These ugly macros do a couple of things: they turn off any padding the compiler might normally do, since we add padding manually (though none in this example), and they enforce alignment chosen by FlatBuffers. This ensures the layout of this struct will look the same regardless of compiler and platform. Note that the fields are private: this is because these store little endian scalars regardless of platform (since this is part of the serialized data). EndianScalar then converts back and forth, which is a no-op on all current mobile and desktop platforms, and a single machine instruction on the few remaining big endian platforms.
struct Monster : private flatbuffers::Table {
- const Vec3 *pos() const { return GetStruct<const Vec3 *>(4); }
- int16_t mana() const { return GetField<int16_t>(6, 150); }
- int16_t hp() const { return GetField<int16_t>(8, 100); }
- const flatbuffers::String *name() const { return GetPointer<const flatbuffers::String *>(10); }
- const flatbuffers::Vector<uint8_t> *inventory() const { return GetPointer<const flatbuffers::Vector<uint8_t> *>(14); }
- int8_t color() const { return GetField<int8_t>(16, 2); }
-};
-Tables are a bit more complicated. A table accessor struct is used to point at the serialized data for a table, which always starts with an offset to its vtable. It derives from Table, which contains the GetField helper functions. GetField takes a vtable offset, and a default value. It will look in the vtable at that offset. If the offset is out of bounds (data from an older version) or the vtable entry is 0, the field is not present and the default is returned. Otherwise, it uses the entry as an offset into the table to locate the field.
struct MonsterBuilder {
- flatbuffers::FlatBufferBuilder &fbb_;
- flatbuffers::uoffset_t start_;
- void add_pos(const Vec3 *pos) { fbb_.AddStruct(4, pos); }
- void add_mana(int16_t mana) { fbb_.AddElement<int16_t>(6, mana, 150); }
- void add_hp(int16_t hp) { fbb_.AddElement<int16_t>(8, hp, 100); }
- void add_name(flatbuffers::Offset<flatbuffers::String> name) { fbb_.AddOffset(10, name); }
- void add_inventory(flatbuffers::Offset<flatbuffers::Vector<uint8_t>> inventory) { fbb_.AddOffset(14, inventory); }
- void add_color(int8_t color) { fbb_.AddElement<int8_t>(16, color, 2); }
- MonsterBuilder(flatbuffers::FlatBufferBuilder &_fbb) : fbb_(_fbb) { start_ = fbb_.StartTable(); }
- flatbuffers::Offset<Monster> Finish() { return flatbuffers::Offset<Monster>(fbb_.EndTable(start_, 7)); }
-};
-MonsterBuilder is the base helper struct to construct a table using a FlatBufferBuilder. You can add the fields in any order, and the Finish call will ensure the correct vtable gets generated.
inline flatbuffers::Offset<Monster> CreateMonster(flatbuffers::FlatBufferBuilder &_fbb, const Vec3 *pos, int16_t mana, int16_t hp, flatbuffers::Offset<flatbuffers::String> name, flatbuffers::Offset<flatbuffers::Vector<uint8_t>> inventory, int8_t color) {
- MonsterBuilder builder_(_fbb);
- builder_.add_inventory(inventory);
- builder_.add_name(name);
- builder_.add_pos(pos);
- builder_.add_hp(hp);
- builder_.add_mana(mana);
- builder_.add_color(color);
- return builder_.Finish();
-}
-CreateMonster is a convenience function that calls all functions in MonsterBuilder above for you. Note that if you pass values which are defaults as arguments, it will not actually construct that field, so you can probably use this function instead of the builder class in almost all cases.
inline const Monster *GetMonster(const void *buf) { return flatbuffers::GetRoot<Monster>(buf); }
-This function is only generated for the root table type, to be able to start traversing a FlatBuffer from a raw buffer pointer.
}; // namespace MyGame -}; // namespace Sample -
Encoding example.
-Below is a sample encoding for the following JSON corresponding to the above schema:
{ pos: { x: 1, y: 2, z: 3 }, name: "fred", hp: 50 }
-Resulting in this binary buffer:
// Start of the buffer: -uint32_t 20 // Offset to the root table. - -// Start of the vtable. Not shared in this example, but could be: -uint16_t 16 // Size of table, starting from here. -uint16_t 22 // Size of object inline data. -uint16_t 4, 0, 20, 16, 0, 0 // Offsets to fields from start of (root) table, 0 for not present. - -// Start of the root table: -int32_t 16 // Offset to vtable used (default negative direction) -float 1, 2, 3 // the Vec3 struct, inline. -uint32_t 8 // Offset to the name string. -int16_t 50 // hp field. -int16_t 0 // Padding for alignment. - -// Start of name string: -uint32_t 4 // Length of string. -int8_t 'f', 'r', 'e', 'd', 0, 0, 0, 0 // Text + 0 termination + padding. -
Note that this not the only possible encoding, since the writer has some flexibility in which of the children of root object to write first (though in this case there's only one string), and what order to write the fields in. Different orders may also cause different alignments to happen.
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
FlatBuffers supports reading and writing binary FlatBuffers in Java and C#. Generate code for Java with the -j option to flatc, or for C# with -n (think .Net).
Note that this document is from the perspective of Java. Code for both languages is generated in the same way, with only minor differences. These differences are explained in a section below.
-See javaTest.java for an example. Essentially, you read a FlatBuffer binary file into a byte[], which you then turn into a ByteBuffer, which you pass to the getRootAsMyRootType function:
Now you can access values much like C++:
-Note that whenever you access a new object like in the pos example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), there's a second pos() method to which you can pass a Vec3 object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.
Java does not support unsigned scalars. This means that any unsigned types you use in your schema will actually be represented as a signed value. This means all bits are still present, but may represent a negative value when used. For example, to read a byte b as an unsigned number, you can do: (short)(b & 0xFF)
The default string accessor (e.g. monster.name()) currently always create a new Java String when accessed, since FlatBuffer's UTF-8 strings can't be used in-place by String. Alternatively, use monster.nameAsByteBuffer() which returns a ByteBuffer referring to the UTF-8 data in the original ByteBuffer, which is much more efficient. The ByteBuffer's position points to the first character, and its limit to just after the last.
Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access:
Alternatively, much like strings, you can use monster.inventoryAsByteBuffer() to get a ByteBuffer referring to the whole vector. Use ByteBuffer methods like asFloatBuffer to get specific views if needed.
If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using:
-Buffer construction in Java
-You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class:
-Create strings:
-Create a table with a struct contained therein:
-For some simpler types, you can use a convenient create function call that allows you to construct tables in one function call. This example definition however contains an inline struct field, so we have to create the table manually. This is to create the buffer without using temporary object allocation.
It's important to understand that fields that are structs are inline (like Vec3 above), and MUST thus be created between the start and end calls of a table. Everything else (other tables, strings, vectors) MUST be created before the start of the table they are referenced in.
Structs do have convenient methods that even have arguments for nested structs.
-As you can see, references to other objects (e.g. the string above) are simple ints, and thus do not have the type-safety of the Offset type in C++. Extra care must thus be taken that you set the right offset on the right field.
-Vectors can be created from the corresponding Java array like so:
-This works for arrays of scalars and (int) offsets to strings/tables, but not structs. If you want to write structs, or what you want to write does not sit in an array, you can also use the start/end pattern:
-You can use the generated method startInventoryVector to conveniently call startVector with the right element size. You pass the number of elements you want to write. Note how you write the elements backwards since the buffer is being constructed back to front. You then pass inv to the corresponding Add call when you construct the table containing it afterwards.
There are add functions for all the scalar types. You use addOffset for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate create function in-line, as shown above in the Monster example.
To finish the buffer, call:
-The buffer is now ready to be transmitted. It is contained in the ByteBuffer which you can obtain from fbb.dataBuffer(). Importantly, the valid data does not start from offset 0 in this buffer, but from fbb.dataBuffer().position() (this is because the data was built backwards in memory). It ends at fbb.capacity().
Differences in C-sharp
-C# code works almost identically to Java, with only a few minor differences. You can see an example of C# code in tests/FlatBuffers.Test/FlatBuffersExampleTests.cs.
First of all, naming follows standard C# style with PascalCasing identifiers, e.g. GetRootAsMyRootType. Also, values (except vectors and unions) are available as properties instead of parameterless accessor methods as in Java. The performance-enhancing methods to which you can pass an already created object are prefixed with Get, e.g.:
Text parsing
-There currently is no support for parsing text (Schema's and JSON) directly from Java or C#, though you could use the C++ parser through native call interfaces available to each language. Please see the C++ documentation for more on text parsing.
-Mutating FlatBuffers
-As you saw above, typically once you have created a FlatBuffer, it is read-only from that moment on. There are however cases where you have just received a FlatBuffer, and you'd like to modify something about it before sending it on to another recipient. With the above functionality, you'd have to generate an entirely new FlatBuffer, while tracking what you modify in your own data structures. This is inconvenient.
-For this reason FlatBuffers can also be mutated in-place. While this is great for making small fixes to an existing buffer, you generally want to create buffers from scratch whenever possible, since it is much more efficient and the API is much more general purpose.
-To get non-const accessors, invoke flatc with --gen-mutable.
You now can:
-We use the somewhat verbose term mutate instead of set to indicate that this is a special use case, not to be confused with the default way of constructing FlatBuffer data.
After the above mutations, you can send on the FlatBuffer to a new recipient without any further work!
-Note that any mutate functions on tables return a boolean, which is false if the field we're trying to set isn't present in the buffer. Fields are not present if they weren't set, or even if they happen to be equal to the default value. For example, in the creation code above we set the mana field to 150, which is the default value, so it was never stored in the buffer. Trying to call mutateMana() on such data will return false, and the value won't actually be modified!
One way to solve this is to call forceDefaults() on a FlatBufferBuilder to force all fields you set to actually be written. This of course increases the size of the buffer somewhat, but this may be acceptable for a mutable buffer.
|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
There's experimental support for reading FlatBuffers in Python. Generate code for Python with the -p option to flatc.
See py_test.py for an example. You import the generated code, read a FlatBuffer binary file into a bytearray, which you pass to the GetRootAsMonster function:
Now you can access values like this:
-To access vectors you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access:
You can also construct these buffers in Python using the functions found in the generated code, and the FlatBufferBuilder class:
-Create strings:
-Create a table with a struct contained therein:
-Unlike C++, Python does not support table creation functions like 'createMonster()'. This is to create the buffer without using temporary object allocation (since the Vec3 is an inline component of Monster, it has to be created right where it is added, whereas the name and the inventory are not inline, and must be created outside of the table creation sequence). Structs do have convenient methods that allow you to construct them in one call. These also have arguments for nested structs, e.g. if a struct has a field a and a nested struct field b (which has fields c and d), then the arguments will be a, c and d.
Vectors also use this start/end pattern to allow vectors of both scalar types and structs:
-The generated method 'StartInventoryVector' is provided as a convenience function which calls 'StartVector' with the correct element size of the vector type which in this case is 'ubyte' or 1 byte per vector element. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front. Use the correct Prepend call for the type, or PrependUOffsetT for offsets. You then pass inv to the corresponding Add call when you construct the table containing it afterwards.
There are Prepend functions for all the scalar types. You use PrependUOffset for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate create function in-line, as shown above in the Monster example.
Once you're done constructing a buffer, you call Finish with the root object offset (mon in the example above). Your data now resides in Builder.Bytes. Important to note is that the real data starts at the index indicated by Head(), for Offset() bytes (this is because the buffer is constructed backwards). If you wanted to read the buffer right after creating it (using GetRootAsMonster above), the second argument, instead of 0 would thus also be Head().
Text Parsing
-There currently is no support for parsing text (Schema's and JSON) directly from Python, though you could use the C++ parser through SWIG or ctypes. Please see the C++ documentation for more on text parsing.
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
The syntax of the schema language (aka IDL, Interface Definition Language) should look quite familiar to users of any of the C family of languages, and also to users of other IDLs. Let's look at an example first:
// example IDL file
-
-namespace MyGame;
-
-attribute "priority";
-
-enum Color : byte { Red = 1, Green, Blue }
-
-union Any { Monster, Weapon, Pickup }
-
-struct Vec3 {
- x:float;
- y:float;
- z:float;
-}
-
-table Monster {
- pos:Vec3;
- mana:short = 150;
- hp:short = 100;
- name:string;
- friendly:bool = false (deprecated, priority: 1);
- inventory:[ubyte];
- color:Color = Blue;
- test:Any;
-}
-
-root_type Monster;
-(Weapon & Pickup not defined as part of this example).
-Tables
-Tables are the main way of defining objects in FlatBuffers, and consist of a name (here Monster) and a list of fields. Each field has a name, a type, and optionally a default value (if omitted, it defaults to 0 / NULL).
Each field is optional: It does not have to appear in the wire representation, and you can choose to omit fields for each individual object. As a result, you have the flexibility to add fields without fear of bloating your data. This design is also FlatBuffer's mechanism for forward and backwards compatibility. Note that:
--
-
- You can add new fields in the schema ONLY at the end of a table definition. Older data will still read correctly, and give you the default value when read. Older code will simply ignore the new field. If you want to have flexibility to use any order for fields in your schema, you can manually assign ids (much like Protocol Buffers), see the
idattribute below.
- - You cannot delete fields you don't use anymore from the schema, but you can simply stop writing them into your data for almost the same effect. Additionally you can mark them as
deprecatedas in the example above, which will prevent the generation of accessors in the generated C++, as a way to enforce the field not being used any more. (careful: this may break code!).
- - You may change field names and table names, if you're ok with your code breaking until you've renamed them there too. -
Structs
-Similar to a table, only now none of the fields are optional (so no defaults either), and fields may not be added or be deprecated. Structs may only contain scalars or other structs. Use this for simple objects where you are very sure no changes will ever be made (as quite clear in the example Vec3). Structs use less memory than tables and are even faster to access (they are always stored in-line in their parent object, and use no virtual table).
Types
-Built-in scalar types are:
--
-
- 8 bit:
byte ubyte bool
- - 16 bit:
short ushort
- - 32 bit:
int uint float
- - 64 bit:
long ulong double
-
Built-in non-scalar types:
--
-
- Vector of any other type (denoted with
[type]). Nesting vectors is not supported, instead you can wrap the inner vector in a table.
- string, which may only hold UTF-8 or 7-bit ASCII. For other text encodings or general binary data use vectors ([byte]or[ubyte]) instead.
-- References to other tables or structs, enums or unions (see below). -
You can't change types of fields once they're used, with the exception of same-size data where a reinterpret_cast would give you a desirable result, e.g. you could change a uint to an int if no values in current data use the high bit yet.
(Default) Values
-Values are a sequence of digits, optionally followed by a . and more digits for float constants, and optionally prefixed by a -. Floats may end with an e or E, followed by a + or - and more digits (scientific notation).
Only scalar values can have defaults, non-scalar (string/vector/table) fields default to NULL when not present.
-You generally do not want to change default values after they're initially defined. Fields that have the default value are not actually stored in the serialized data but are generated in code, so when you change the default, you'd now get a different value than from code generated from an older version of the schema. There are situations however where this may be desirable, especially if you can ensure a simultaneous rebuild of all code.
-Enums
-Define a sequence of named constants, each with a given value, or increasing by one from the previous one. The default first value is 0. As you can see in the enum declaration, you specify the underlying integral type of the enum with : (in this case byte), which then determines the type of any fields declared with this enum type.
Unions
-Unions share a lot of properties with enums, but instead of new names for constants, you use names of tables. You can then declare a union field which can hold a reference to any of those types, and additionally a hidden field with the suffix _type is generated that holds the corresponding enum value, allowing you to know which type to cast to at runtime.
Unions are a good way to be able to send multiple message types as a FlatBuffer. Note that because a union field is really two fields, it must always be part of a table, it cannot be the root of a FlatBuffer by itself.
-If you have a need to distinguish between different FlatBuffers in a more open-ended way, for example for use as files, see the file identification feature below.
-Namespaces
-These will generate the corresponding namespace in C++ for all helper code, and packages in Java. You can use . to specify nested namespaces / packages.
Includes
-You can include other schemas files in your current one, e.g.:
include "mydefinitions.fbs"; -
This makes it easier to refer to types defined elsewhere. include automatically ensures each file is parsed just once, even when referred to more than once.
When using the flatc compiler to generate code for schema definitions, only definitions in the current file will be generated, not those from the included files (those you still generate separately).
Root type
-This declares what you consider to be the root table (or struct) of the serialized data. This is particular important for parsing JSON data, which doesn't include object type information.
-File identification and extension
-Typically, a FlatBuffer binary buffer is not self-describing, i.e. it needs you to know its schema to parse it correctly. But if you want to use a FlatBuffer as a file format, it would be convenient to be able to have a "magic number" in there, like most file formats have, to be able to do a sanity check to see if you're reading the kind of file you're expecting.
-Now, you can always prefix a FlatBuffer with your own file header, but FlatBuffers has a built-in way to add an identifier to a FlatBuffer that takes up minimal space, and keeps the buffer compatible with buffers that don't have such an identifier.
-You can specify in a schema, similar to root_type, that you intend for this type of FlatBuffer to be used as a file format:
file_identifier "MYFI"; -
Identifiers must always be exactly 4 characters long. These 4 characters will end up as bytes at offsets 4-7 (inclusive) in the buffer.
-For any schema that has such an identifier, flatc will automatically add the identifier to any binaries it generates (with -b), and generated calls like FinishMonsterBuffer also add the identifier. If you have specified an identifier and wish to generate a buffer without one, you can always still do so by calling FlatBufferBuilder::Finish explicitly.
After loading a buffer, you can use a call like MonsterBufferHasIdentifier to check if the identifier is present.
Note that this is best for open-ended uses such as files. If you simply wanted to send one of a set of possible messages over a network for example, you'd be better off with a union.
-Additionally, by default flatc will output binary files as .bin. This declaration in the schema will change that to whatever you want:
file_extension "ext"; -
Comments & documentation
-May be written as in most C-based languages. Additionally, a triple comment (///) on a line by itself signals that a comment is documentation for whatever is declared on the line after it (table/struct/field/enum/union/element), and the comment is output in the corresponding C++ code. Multiple such lines per item are allowed.
Attributes
-Attributes may be attached to a declaration, behind a field, or after the name of a table/struct/enum/union. These may either have a value or not. Some attributes like deprecated are understood by the compiler, user defined ones need to be declared with the attribute declaration (like priority in the example above), and are available to query if you parse the schema at runtime. This is useful if you write your own code generators/editors etc., and you wish to add additional information specific to your tool (such as a help text).
Current understood attributes:
--
-
id: n(on a table field): manually set the field identifier ton. If you use this attribute, you must use it on ALL fields of this table, and the numbers must be a contiguous range from 0 onwards. Additionally, since a union type effectively adds two fields, its id must be that of the second field (the first field is the type field and not explicitly declared in the schema). For example, if the last field before the union field had id 6, the union field should have id 8, and the unions type field will implicitly be 7. IDs allow the fields to be placed in any order in the schema. When a new field is added to the schema is must use the next available ID.
-deprecated(on a field): do not generate accessors for this field anymore, code should stop using this data.
-required(on a non-scalar table field): this field must always be set. By default, all fields are optional, i.e. may be left out. This is desirable, as it helps with forwards/backwards compatibility, and flexibility of data structures. It is also a burden on the reading code, since for non-scalar fields it requires you to check against NULL and take appropriate action. By specifying this field, you force code that constructs FlatBuffers to ensure this field is initialized, so the reading code may access it directly, without checking for NULL. If the constructing code does not initialize this field, they will get an assert, and also the verifier will fail on buffers that have missing required fields.
-original_order(on a table): since elements in a table do not need to be stored in any particular order, they are often optimized for space by sorting them to size. This attribute stops that from happening.
-force_align: size(on a struct): force the alignment of this struct to be something higher than what it is naturally aligned to. Causes these structs to be aligned to that amount inside a buffer, IF that buffer is allocated with that alignment (which is not necessarily the case for buffers accessed directly inside aFlatBufferBuilder).
-bit_flags(on an enum): the values of this field indicate bits, meaning that any value N specified in the schema will end up representing 1<<N, or if you don't specify values at all, you'll get the sequence 1, 2, 4, 8, ...
-nested_flatbuffer: "table_name"(on a field): this indicates that the field (which must be a vector of ubyte) contains flatbuffer data, for which the root type is given bytable_name. The generated code will then produce a convenient accessor for the nested FlatBuffer.
-key(on a field): this field is meant to be used as a key when sorting a vector of the type of table it sits in. Can be used for in-place binary search.
-
JSON Parsing
-The same parser that parses the schema declarations above is also able to parse JSON objects that conform to this schema. So, unlike other JSON parsers, this parser is strongly typed, and parses directly into a FlatBuffer (see the compiler documentation on how to do this from the command line, or the C++ documentation on how to do this at runtime).
-Besides needing a schema, there are a few other changes to how it parses JSON:
--
-
- It accepts field names with and without quotes, like many JSON parsers already do. It outputs them without quotes as well, though can be made to output them using the
strict_jsonflag.
- - If a field has an enum type, the parser will recognize symbolic enum values (with or without quotes) instead of numbers, e.g.
field: EnumVal. If a field is of integral type, you can still use symbolic names, but values need to be prefixed with their type and need to be quoted, e.g.field: "Enum.EnumVal". For enums representing flags, you may place multiple inside a string separated by spaces to OR them, e.g.field: "EnumVal1 EnumVal2"orfield: "Enum.EnumVal1 Enum.EnumVal2".
- - Similarly, for unions, these need to specified with two fields much like you do when serializing from code. E.g. for a field
foo, you must add a fieldfoo_type: FooOneright before thefoofield, whereFooOnewould be the table out of the union you want to use.
-
When parsing JSON, it recognizes the following escape codes in strings:
--
-
\n- linefeed.
-\t- tab.
-\r- carriage return.
-\b- backspace.
-\f- form feed.
-\"- double quote.
-\\- backslash.
-\/- forward slash.
-\uXXXX- 16-bit unicode code point, converted to the equivalent UTF-8 representation.
-\xXX- 8-bit binary hexadecimal number XX. This is the only one that is not in the JSON spec (see http://json.org/), but is needed to be able to encode arbitrary binary in strings to text and back without losing information (e.g. the byte 0xFF can't be represented in standard JSON).
-
It also generates these escape codes back again when generating JSON from a binary representation.
-Gotchas
-Schemas and version control
-FlatBuffers relies on new field declarations being added at the end, and earlier declarations to not be removed, but be marked deprecated when needed. We think this is an improvement over the manual number assignment that happens in Protocol Buffers (and which is still an option using the id attribute mentioned above).
One place where this is possibly problematic however is source control. If user A adds a field, generates new binary data with this new schema, then tries to commit both to source control after user B already committed a new field also, and just auto-merges the schema, the binary files are now invalid compared to the new schema.
-The solution of course is that you should not be generating binary data before your schema changes have been committed, ensuring consistency with the rest of the world. If this is not practical for you, use explicit field ids, which should always generate a merge conflict if two people try to allocate the same id.
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
FlatBuffers is actively being worked on, which means that certain platform / language / feature combinations may not be available yet.
-This page tries to track those issues, to make informed decisions easier. In general:
--
-
- Languages: language support beyond the ones created by the original FlatBuffer authors typically depends on community contributions. -
- Features: C++ was the first language supported, since our original target was high performance game development. It thus has the richest feature set, and is likely most robust. Other languages are catching up however. -
- Platforms: All language implementations are typically portable to most platforms, unless where noted otherwise. -
NOTE: this table is a start, it needs to be extended.
-| Feature | C++ | Java | C# | Go | Python | JS | C | PHP | Ruby |
|---|---|---|---|---|---|---|---|---|---|
| Codegen for all basic features | Yes | Yes | Yes | Yes | Yes | Yes | WiP | WiP | WiP |
| JSON parsing | Yes | No | No | No | No | No | No | No | No |
| Simple mutation | Yes | WIP | WIP | No | No | No | No | No | No |
| Reflection | Yes | No | No | No | No | No | No | No | No |
| Buffer verifier | Yes | No | No | No | No | No | No | No | No |
| Testing: basic | Yes | Yes | Yes | Yes | Yes | Yes | ? | ? | ? |
| Testing: fuzz | Yes | No | No | Yes | Yes | No | ? | ? | ? |
| Performance: | Superb | Great | Great | Great | Ok | ? | Superb | ? | ? |
| Platform: Windows | VS2010 | Yes | Yes | ? | ? | ? | ? | ? | ? |
| Platform: Linux | GCC282 | Yes | ? | Yes | Yes | ? | ? | ? | ? |
| Platform: OS X | Xcode4 | ? | ? | ? | Yes | ? | ? | ? | ? |
| Platform: Android | NDK10d | Yes | ? | ? | ? | ? | ? | ? | ? |
| Platform: iOS | ? | ? | ? | ? | ? | ? | ? | ? | ? |
| Engine: Unity | ? | ? | Yes | ? | ? | ? | ? | ? | ? |
| Primary authors (github) | gwvo | gwvo | ev*/js* | rw | rw | evanw/ev* | mik* | ch* | rw |
-
-
- ev = evolutional -
- js = jonsimantov -
- mik = mikkelfj -
- ch = chobie -
|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
This document tries to shed some light on to the "why" of FlatBuffers, a new serialization library.
-Motivation
-Back in the good old days, performance was all about instructions and cycles. Nowadays, processing units have run so far ahead of the memory subsystem, that making an efficient application should start and finish with thinking about memory. How much you use of it. How you lay it out and access it. How you allocate it. When you copy it.
-Serialization is a pervasive activity in a lot programs, and a common source of memory inefficiency, with lots of temporary data structures needed to parse and represent data, and inefficient allocation patterns and locality.
-If it would be possible to do serialization with no temporary objects, no additional allocation, no copying, and good locality, this could be of great value. The reason serialization systems usually don't manage this is because it goes counter to forwards/backwards compatability, and platform specifics like endianness and alignment.
-FlatBuffers is what you get if you try anyway.
-In particular, FlatBuffers focus is on mobile hardware (where memory size and memory bandwidth is even more constrained than on desktop hardware), and applications that have the highest performance needs: games.
-FlatBuffers
-This is a summary of FlatBuffers functionality, with some rationale. A more detailed description can be found in the FlatBuffers documentation.
-Summary
-A FlatBuffer is a binary buffer containing nested objects (structs, tables, vectors,..) organized using offsets so that the data can be traversed in-place just like any pointer-based data structure. Unlike most in-memory data structures however, it uses strict rules of alignment and endianness (always little) to ensure these buffers are cross platform. Additionally, for objects that are tables, FlatBuffers provides forwards/backwards compatibility and general optionality of fields, to support most forms of format evolution.
-You define your object types in a schema, which can then be compiled to C++ or Java for low to zero overhead reading & writing. Optionally, JSON data can be dynamically parsed into buffers.
-Tables
-Tables are the cornerstone of FlatBuffers, since format evolution is essential for most applications of serialization. Typically, dealing with format changes is something that can be done transparently during the parsing process of most serialization solutions out there. But a FlatBuffer isn't parsed before it is accessed.
-Tables get around this by using an extra indirection to access fields, through a vtable. Each table comes with a vtable (which may be shared between multiple tables with the same layout), and contains information where fields for this particular kind of instance of vtable are stored. The vtable may also indicate that the field is not present (because this FlatBuffer was written with an older version of the software, of simply because the information was not necessary for this instance, or deemed deprecated), in which case a default value is returned.
-Tables have a low overhead in memory (since vtables are small and shared) and in access cost (an extra indirection), but provide great flexibility. Tables may even cost less memory than the equivalent struct, since fields do not need to be stored when they are equal to their default.
-FlatBuffers additionally offers "naked" structs, which do not offer forwards/backwards compatibility, but can be even smaller (useful for very small objects that are unlikely to change, like e.g. a coordinate pair or a RGBA color).
-Schemas
-While schemas reduce some generality (you can't just read any data without having its schema), they have a lot of upsides:
--
-
- Most information about the format can be factored into the generated code, reducing memory needed to store data, and time to access it. -
- The strong typing of the data definitions means less error checking/handling at runtime (less can go wrong). -
- A schema enables us to access a buffer without parsing. -
FlatBuffer schemas are fairly similar to those of the incumbent, Protocol Buffers, and generally should be readable to those familiar with the C family of languages. We chose to improve upon the features offered by .proto files in the following ways:
--
-
- Deprecation of fields instead of manual field id assignment. Extending an object in a .proto means hunting for a free slot among the numbers (preferring lower numbers since they have a more compact representation). Besides being inconvenient, it also makes removing fields problematic: you either have to keep them, not making it obvious that this field shouldn't be read/written anymore, and still generating accessors. Or you remove it, but now you risk that there's still old data around that uses that field by the time someone reuses that field id, with nasty consequences. -
- Differentiating between tables and structs (see above). Effectively all table fields are
optional, and all struct fields arerequired.
- - Having a native vector type instead of
repeated. This gives you a length without having to collect all items, and in the case of scalars provides for a more compact representation, and one that guarantees adjacency.
- - Having a native
uniontype instead of using a series ofoptionalfields, all of which must be checked individually.
- - Being able to define defaults for all scalars, instead of having to deal with their optionality at each access. -
- A parser that can deal with both schemas and data definitions (JSON compatible) uniformly. -
');
-}
-
-function showSyncOn(n,relpath)
-{
- n.html('
');
-}
-
-function toggleSyncButton(relpath)
-{
- var navSync = $('#nav-sync');
- if (navSync.hasClass('sync')) {
- navSync.removeClass('sync');
- showSyncOff(navSync,relpath);
- storeLink(stripPath2(pathName())+hashUrl());
- } else {
- navSync.addClass('sync');
- showSyncOn(navSync,relpath);
- deleteLink();
- }
-}
-
-function initNavTree(toroot,relpath)
-{
- var o = new Object();
- o.toroot = toroot;
- o.node = new Object();
- o.node.li = document.getElementById("nav-tree-contents");
- o.node.childrenData = NAVTREE;
- o.node.children = new Array();
- o.node.childrenUL = document.createElement("ul");
- o.node.getChildrenUL = function() { return o.node.childrenUL; };
- o.node.li.appendChild(o.node.childrenUL);
- o.node.depth = 0;
- o.node.relpath = relpath;
- o.node.expanded = false;
- o.node.isLast = true;
- o.node.plus_img = document.createElement("img");
- o.node.plus_img.src = relpath+"ftv2pnode.png";
- o.node.plus_img.width = 16;
- o.node.plus_img.height = 22;
-
- if (localStorageSupported()) {
- var navSync = $('#nav-sync');
- if (cachedLink()) {
- showSyncOff(navSync,relpath);
- navSync.removeClass('sync');
- } else {
- showSyncOn(navSync,relpath);
- }
- navSync.click(function(){ toggleSyncButton(relpath); });
- }
-
- $(window).load(function(){
- navTo(o,toroot,hashUrl(),relpath);
- showRoot();
- });
-
- $(window).bind('hashchange', function(){
- if (window.location.hash && window.location.hash.length>1){
- var a;
- if ($(location).attr('hash')){
- var clslink=stripPath(pathName())+':'+hashValue();
- a=$('.item a[class$="'+clslink.replace(/
-
-
-
-
-
-
-|
-
- |
-
- FlatBuffers
-
-
- An open source project by FPL.
-
- |
-
diff --git a/docs/source/Building.md b/docs/source/Building.md index f2e1818b6aac8..f57371d5f5900 100755 --- a/docs/source/Building.md +++ b/docs/source/Building.md @@ -1,8 +1,13 @@ -# Building +Building {#flatbuffers_guide_building} +======== + +## Building with Visual Studio or Xcode projects There are project files for Visual Studio and Xcode that should allow you to build the compiler `flatc`, the samples and the tests out of the box. +## Building with CMake + Alternatively, the distribution comes with a `cmake` file that should allow you to build project/make files for any platform. For details on `cmake`, see
diff --git a/docs/source/FlatBuffers.md b/docs/source/FlatBuffers.md index 79846d77689fe..bbb0dce589368 100644 --- a/docs/source/FlatBuffers.md +++ b/docs/source/FlatBuffers.md @@ -1,9 +1,12 @@ -# FlatBuffers +FlatBuffers {#flatbuffers_index} +=========== -FlatBuffers is an efficient cross platform serialization library for C++, Java, -C#, Go, Python and JavaScript (C, PHP & Ruby in progress). -It was originally created at Google for game development and other -performance-critical applications. +# Overview {#flatbuffers_overview} + +[FlatBuffers](@ref flatbuffers_overview) is an efficient cross platform +serialization library for C++, C#, Go, Java, JavaScript, PHP, and Python +(C and Ruby in progress). It was originally created at Google for game +development and other performance-critical applications. It is available as Open Source on [GitHub](http://github.com/google/flatbuffers) under the Apache license, v2 (see LICENSE.txt). @@ -26,7 +29,7 @@ under the Apache license, v2 (see LICENSE.txt). projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the - [benchmarks](md__benchmarks.html) for details. + [benchmarks](@ref flatbuffers_benchmarks) for details. - **Flexible** - Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for @@ -76,7 +79,7 @@ In this context, it is only a better choice for systems that have very little to no information ahead of time about what data needs to be stored. Read more about the "why" of FlatBuffers in the -[white paper](md__white_paper.html). +[white paper](@ref flatbuffers_white_paper). ### Who uses FlatBuffers? - [Cocos2d-x](http://www.cocos2d-x.org/), the #1 open source mobile game @@ -118,22 +121,23 @@ sections provide a more in-depth usage guide. ## In-depth documentation -- How to [build the compiler](md__building.html) and samples on various - platforms. -- How to [use the compiler](md__compiler.html). -- How to [write a schema](md__schemas.html). -- How to [use the generated C++ code](md__cpp_usage.html) in your own - programs. -- How to [use the generated Java/C# code](md__java_usage.html) in your own - programs. -- How to [use the generated Go code](md__go_usage.html) in your own - programs. -- [Support matrix](md__support.html) for platforms/languages/features. -- Some [benchmarks](md__benchmarks.html) showing the advantage of using +- How to [build the compiler](@ref flatbuffers_guide_building) and samples on + various platforms. +- How to [use the compiler](@ref flatbuffers_guide_using_schema_compiler). +- How to [write a schema](@ref flatbuffers_guide_writing_schema). +- How to [use the generated C++ code](@ref flatbuffers_guide_use_cpp) in your + own programs. +- How to [use the generated Java/C# code](@ref flatbuffers_guide_use_java_c-sharp) + in your own programs. +- How to [use the generated Go code](@ref flatbuffers_guide_use_go) in your + own programs. +- [Support matrix](@ref flatbuffers_support) for platforms/languages/features. +- Some [benchmarks](@ref flatbuffers_benchmarks) showing the advantage of + using FlatBuffers. +- A [white paper](@ref flatbuffers_white_paper) explaining the "why" of FlatBuffers. -- A [white paper](md__white_paper.html) explaining the "why" of FlatBuffers. -- A description of the [internals](md__internals.html) of FlatBuffers. -- A formal [grammar](md__grammar.html) of the schema language. +- A description of the [internals](@ref flatbuffers_internals) of FlatBuffers. +- A formal [grammar](@ref flatbuffers_grammar) of the schema language. ## Online resources diff --git a/docs/source/GoApi.md b/docs/source/GoApi.md new file mode 100644 index 0000000000000..98be2b6a9e4ec --- /dev/null +++ b/docs/source/GoApi.md @@ -0,0 +1,26 @@ +Go API +====== + +\addtogroup flatbuffers_go_api + + +\snippet GoApi_generated.txt Go API diff --git a/docs/source/GoApi_generated.txt b/docs/source/GoApi_generated.txt new file mode 100644 index 0000000000000..3d4e0fc77aea2 --- /dev/null +++ b/docs/source/GoApi_generated.txt @@ -0,0 +1,125 @@ +// This file was generated using `godoc` and customized for use with the +// API Reference documentation. To recreate this file, use the `godoc` tool +// (http://godoc.org) with the files in the `flatbuffers/go` folder. +// +// Note: You may need to ensure that copies of the files exist in the +// `src/` subfolder at the path set by the `$GOROOT` environment variable. +// You can either move the files to `$GOROOT/src/flatbuffers` manually, if +// `$GOROOT` is already set, otherwise you will need to manually set the +// `$GOROOT` variable to a path and create `src/flatbuffers` subfolders at that +// path. Then copy these files into `$GOROOT/src/flatbuffers`. (Some versions of +// `godoc` include a `-path` flag. This could be used instead, if available). +// +// Once the files exist at the `$GOROOT/src/flatbuffers` location, you can +// regenerate this doc using the following command: +// `godoc flatbuffers > GoApi_generated.txt`. +// +// After the documentation is generated, you will have to manually remove any +// non-user facing documentation from this file. + +/// [Go API] +PACKAGE DOCUMENTATION + +package flatbuffers + Package flatbuffers provides facilities to read and write flatbuffers + objects. + +TYPES + +type Builder struct { + // `Bytes` gives raw access to the buffer. Most users will want to use + // FinishedBytes() instead. + Bytes []byte +} + Builder is a state machine for creating FlatBuffer objects. Use a + Builder to construct object(s) starting from leaf nodes. + + A Builder constructs byte buffers in a last-first manner for simplicity + and performance. + +FUNCTIONS + +func NewBuilder(initialSize int) *Builder + NewBuilder initializes a Builder of size `initial_size`. The internal + buffer is grown as needed. + +func (b *Builder) CreateByteString(s []byte) UOffsetT + CreateByteString writes a byte slice as a string (null-terminated). + +func (b *Builder) CreateByteVector(v []byte) UOffsetT + CreateByteVector writes a ubyte vector + +func (b *Builder) CreateString(s string) UOffsetT + CreateString writes a null-terminated string as a vector. + +func (b *Builder) EndVector(vectorNumElems int) UOffsetT + EndVector writes data necessary to finish vector construction. + +func (b *Builder) Finish(rootTable UOffsetT) + Finish finalizes a buffer, pointing to the given `rootTable`. + +func (b *Builder) FinishedBytes() []byte + FinishedBytes returns a pointer to the written data in the byte buffer. + Panics if the builder is not in a finished state (which is caused by + calling `Finish()`). + +func (b *Builder) Head() UOffsetT + Head gives the start of useful data in the underlying byte buffer. Note: + unlike other functions, this value is interpreted as from the left. + +func (b *Builder) PrependBool(x bool) + PrependBool prepends a bool to the Builder buffer. Aligns and checks for + space. + +func (b *Builder) PrependByte(x byte) + PrependByte prepends a byte to the Builder buffer. Aligns and checks for + space. + +func (b *Builder) PrependFloat32(x float32) + PrependFloat32 prepends a float32 to the Builder buffer. Aligns and + checks for space. + +func (b *Builder) PrependFloat64(x float64) + PrependFloat64 prepends a float64 to the Builder buffer. Aligns and + checks for space. + +func (b *Builder) PrependInt16(x int16) + PrependInt16 prepends a int16 to the Builder buffer. Aligns and checks + for space. + +func (b *Builder) PrependInt32(x int32) + PrependInt32 prepends a int32 to the Builder buffer. Aligns and checks + for space. + +func (b *Builder) PrependInt64(x int64) + PrependInt64 prepends a int64 to the Builder buffer. Aligns and checks + for space. + +func (b *Builder) PrependInt8(x int8) + PrependInt8 prepends a int8 to the Builder buffer. Aligns and checks for + space. + +func (b *Builder) PrependUOffsetT(off UOffsetT) + PrependUOffsetT prepends an UOffsetT, relative to where it will be + written. + +func (b *Builder) PrependUint16(x uint16) + PrependUint16 prepends a uint16 to the Builder buffer. Aligns and checks + for space. + +func (b *Builder) PrependUint32(x uint32) + PrependUint32 prepends a uint32 to the Builder buffer. Aligns and checks + for space. + +func (b *Builder) PrependUint64(x uint64) + PrependUint64 prepends a uint64 to the Builder buffer. Aligns and checks + for space. + +func (b *Builder) PrependUint8(x uint8) + PrependUint8 prepends a uint8 to the Builder buffer. Aligns and checks + for space. + +func (b *Builder) Reset() + Reset truncates the underlying Builder buffer, facilitating alloc-free + reuse of a Builder. It also resets bookkeeping data. +/// [Go API] diff --git a/docs/source/GoUsage.md b/docs/source/GoUsage.md index c2a7495c3af0f..a3a0e928ab6e9 100644 --- a/docs/source/GoUsage.md +++ b/docs/source/GoUsage.md @@ -1,122 +1,76 @@ -# Use in Go +Use in Go {#flatbuffers_guide_use_go} +========= -There's experimental support for reading FlatBuffers in Go. Generate code -for Go with the `-g` option to `flatc`. +## Before you get started -See `go_test.go` for an example. You import the generated code, read a -FlatBuffer binary file into a `[]byte`, which you pass to the -`GetRootAsMonster` function: +Before diving into the FlatBuffers usage in Go, it should be noted that +the [Tutorial](@ref flatbuffers_guide_tutorial) page has a complete guide +to general FlatBuffers usage in all of the supported languages (including Go). +This page is designed to cover the nuances of FlatBuffers usage, specific to +Go. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} - import ( - example "MyGame/Example" - flatbuffers "github.com/google/flatbuffers/go" +You should also have read the [Building](@ref flatbuffers_guide_building) +documentation to build `flatc` and should be familiar with +[Using the schema compiler](@ref flatbuffers_guide_using_schema_compiler) and +[Writing a schema](@ref flatbuffers_guide_writing_schema). - io/ioutil - ) +## FlatBuffers Go library code location - buf, err := ioutil.ReadFile("monster.dat") - // handle err - monster := example.GetRootAsMonster(buf, 0) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The code for the FlatBuffers Go library can be found at +`flatbuffers/go`. You can browse the library code on the [FlatBuffers +GitHub page](https://github.com/google/flatbuffers/tree/master/go). -Now you can access values like this: +## Testing the FlatBuffers Go library -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} - hp := monster.Hp() - pos := monster.Pos(nil) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The code to test the Go library can be found at `flatbuffers/tests`. +The test code itself is located in [go_test.go](https://github.com/google/ +flatbuffers/blob/master/tests/go_test.go). -Note that whenever you access a new object like in the `Pos` example above, -a new temporary accessor object gets created. If your code is very performance -sensitive (you iterate through a lot of objects), you can replace nil with a -pointer to a `Vec3` object you've already created. This allows -you to reuse it across many calls and reduce the amount of object allocation -(and thus garbage collection) your program does. +To run the tests, use the [GoTest.sh](https://github.com/google/flatbuffers/ +blob/master/tests/GoTest.sh) shell script. -To access vectors you pass an extra index to the -vector field accessor. Then a second method with the same name suffixed -by `Length` let's you know the number of elements you can access: +*Note: The shell script requires [Go](https://golang.org/doc/install) to +be installed.* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} - for i := 0; i < monster.InventoryLength(); i++ { - monster.Inventory(i) // do something here - } -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +## Using the FlatBuffers Go library -You can also construct these buffers in Go using the functions found in the -generated code, and the FlatBufferBuilder class: +*Note: See [Tutorial](@ref flatbuffers_guide_tutorial) for a more in-depth +example of how to use FlatBuffers in Go.* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} - builder := flatbuffers.NewBuilder(0) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +FlatBuffers supports reading and writing binary FlatBuffers in Go. + +To use FlatBuffers in your own code, first generate Go classes from your +schema with the `--go` option to `flatc`. Then you can include both FlatBuffers +and the generated code to read or write a FlatBuffer. -Create strings: +For example, here is how you would read a FlatBuffer binary file in Go: First, +include the library and generated code. Then read a FlatBuffer binary file into +a `[]byte`, which you pass to the `GetRootAsMonster` function: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} - str := builder.CreateString("MyMonster") -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + import ( + example "MyGame/Example" + flatbuffers "github.com/google/flatbuffers/go" -Create a table with a struct contained therein: + io/ioutil + ) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} - example.MonsterStart(builder) - example.MonsterAddPos(builder, example.CreateVec3(builder, 1.0, 2.0, 3.0, 3.0, 4, 5, 6)) - example.MonsterAddHp(builder, 80) - example.MonsterAddName(builder, str) - example.MonsterAddInventory(builder, inv) - example.MonsterAddTest_Type(builder, 1) - example.MonsterAddTest(builder, mon2) - example.MonsterAddTest4(builder, test4s) - mon := example.MonsterEnd(builder) + buf, err := ioutil.ReadFile("monster.dat") + // handle err + monster := example.GetRootAsMonster(buf, 0) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Unlike C++, Go does not support table creation functions like 'createMonster()'. -This is to create the buffer without -using temporary object allocation (since the `Vec3` is an inline component of -`Monster`, it has to be created right where it is added, whereas the name and -the inventory are not inline, and **must** be created outside of the table -creation sequence). -Structs do have convenient methods that allow you to construct them in one call. -These also have arguments for nested structs, e.g. if a struct has a field `a` -and a nested struct field `b` (which has fields `c` and `d`), then the arguments -will be `a`, `c` and `d`. - -Vectors also use this start/end pattern to allow vectors of both scalar types -and structs: +Now you can access values like this: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} - example.MonsterStartInventoryVector(builder, 5) - for i := 4; i >= 0; i-- { - builder.PrependByte(byte(i)) - } - inv := builder.EndVector(5) + hp := monster.Hp() + pos := monster.Pos(nil) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The generated method 'StartInventoryVector' is provided as a convenience -function which calls 'StartVector' with the correct element size of the vector -type which in this case is 'ubyte' or 1 byte per vector element. -You pass the number of elements you want to write. -You write the elements backwards since the buffer -is being constructed back to front. Use the correct `Prepend` call for the type, -or `PrependUOffsetT` for offsets. You then pass `inv` to the corresponding -`Add` call when you construct the table containing it afterwards. - -There are `Prepend` functions for all the scalar types. You use -`PrependUOffset` for any previously constructed objects (such as other tables, -strings, vectors). For structs, you use the appropriate `create` function -in-line, as shown above in the `Monster` example. - -Once you're done constructing a buffer, you call `Finish` with the root object -offset (`mon` in the example above). Your data now resides in Builder.Bytes. -Important to note is that the real data starts at the index indicated by Head(), -for Offset() bytes (this is because the buffer is constructed backwards). -If you wanted to read the buffer right after creating it (using -`GetRootAsMonster` above), the second argument, instead of `0` would thus -also be `Head()`. - ## Text Parsing There currently is no support for parsing text (Schema's and JSON) directly from Go, though you could use the C++ parser through cgo. Please see the C++ documentation for more on text parsing. + +
diff --git a/docs/source/Grammar.md b/docs/source/Grammar.md index bcc5f7496f635..b6b48c08b0eb7 100755 --- a/docs/source/Grammar.md +++ b/docs/source/Grammar.md @@ -1,4 +1,5 @@ -# Grammar of the schema language +Grammar of the schema language {#flatbuffers_grammar} +============================== schema = include* ( namespace\_decl | type\_decl | enum\_decl | root\_decl | diff --git a/docs/source/Internals.md b/docs/source/Internals.md index 215842d36a4c8..a7dce4fb0a7e8 100755 --- a/docs/source/Internals.md +++ b/docs/source/Internals.md @@ -1,4 +1,5 @@ -# FlatBuffer Internals +FlatBuffer Internals {#flatbuffers_internals} +==================== This section is entirely optional for the use of FlatBuffers. In normal usage, you should never need the information contained herein. If you're @@ -16,7 +17,7 @@ byte-swap intrinsics. On purpose, the format leaves a lot of details about where exactly things live in memory undefined, e.g. fields in a table can have any -order, and objects to some extend can be stored in many orders. This is +order, and objects to some extent can be stored in many orders. This is because the format doesn't need this information to be efficient, and it leaves room for optimization and extension (for example, fields can be packed in a way that is most compact). Instead, the format is defined in @@ -228,7 +229,12 @@ Otherwise, it uses the entry as an offset into the table to locate the field. `FlatBufferBuilder`. You can add the fields in any order, and the `Finish` call will ensure the correct vtable gets generated. - inline flatbuffers::Offset
diff --git a/docs/source/JavaCsharpUsage.md b/docs/source/JavaCsharpUsage.md new file mode 100755 index 0000000000000..3af9cfa488147 --- /dev/null +++ b/docs/source/JavaCsharpUsage.md @@ -0,0 +1,141 @@ +Use in Java/C# {#flatbuffers_guide_use_java_c-sharp} +============== + +## Before you get started + +Before diving into the FlatBuffers usage in Java or C#, it should be noted that +the [Tutorial](@ref flatbuffers_guide_tutorial) page has a complete guide to +general FlatBuffers usage in all of the supported languages (including both Java +and C#). This page is designed to cover the nuances of FlatBuffers usage, +specific to Java and C#. + +You should also have read the [Building](@ref flatbuffers_guide_building) +documentation to build `flatc` and should be familiar with +[Using the schema compiler](@ref flatbuffers_guide_using_schema_compiler) and +[Writing a schema](@ref flatbuffers_guide_writing_schema). + +## FlatBuffers Java and C-sharp code location + +#### Java + +The code for the FlatBuffers Java library can be found at +`flatbuffers/java/com/google/flatbuffers`. You can browse the library on the +[FlatBuffers GitHub page](https://github.com/google/flatbuffers/tree/master/ +java/com/google/flatbuffers). + +#### C-sharp + +The code for the FlatBuffers C# library can be found at +`flatbuffers/net/FlatBuffers`. You can browse the library on the +[FlatBuffers GitHub page](https://github.com/google/flatbuffers/tree/master/net/ +FlatBuffers). + +## Testing the FlatBuffers Java and C-sharp libraries + +The code to test the libraries can be found at `flatbuffers/tests`. + +#### Java + +The test code for Java is located in [JavaTest.java](https://github.com/google +/flatbuffers/blob/master/tests/JavaTest.java). + +To run the tests, use either [JavaTest.sh](https://github.com/google/ +flatbuffers/blob/master/tests/JavaTest.sh) or [JavaTest.bat](https://github.com/ +google/flatbuffers/blob/master/tests/JavaTest.bat), depending on your operating +system. + +*Note: These scripts require that [Java](https://www.oracle.com/java/index.html) +is installed.* + +#### C-sharp + +The test code for C# is located in the [FlatBuffers.Test](https://github.com/ +google/flatbuffers/tree/master/tests/FlatBuffers.Test) subfolder. To run the +tests, open `FlatBuffers.Test.csproj` in [Visual Studio]( +https://www.visualstudio.com), and compile/run the project. + +Optionally, you can run this using [Mono](http://www.mono-project.com/) instead. +Once you have installed `Mono`, you can run the tests from the command line +by running the following commands from inside the `FlatBuffers.Test` folder: + +~~~{.sh} + mcs *.cs ../MyGame/Example/*.cs ../../net/FlatBuffers/*.cs + mono Assert.exe +~~~ + +## Using the FlatBuffers Java (and C#) library + +*Note: See [Tutorial](@ref flatbuffers_guide_tutorial) for a more in-depth +example of how to use FlatBuffers in Java or C#.* + +FlatBuffers supports reading and writing binary FlatBuffers in Java and C#. + +To use FlatBuffers in your own code, first generate Java classes from your +schema with the `--java` option to `flatc`. (Or for C# with `--csharp`). +Then you can include both FlatBuffers and the generated code to read +or write a FlatBuffer. + +For example, here is how you would read a FlatBuffer binary file in Java: +First, import the library and generated code. Then, you read a FlatBuffer binary +file into a `byte[]`. You then turn the `byte[]` into a `ByteBuffer`, which you +pass to the `getRootAsMyRootType` function: + +*Note: The code here is written from the perspective of Java. Code for both +languages is both generated and used in nearly the exact same way, with only +minor differences. These differences are +[explained in a section below](#differences_in_c-sharp).* + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} + import MyGame.Example.*; + import com.google.flatbuffers.FlatBufferBuilder; + + // This snippet ignores exceptions for brevity. + File file = new File("monsterdata_test.mon"); + RandomAccessFile f = new RandomAccessFile(file, "r"); + byte[] data = new byte[(int)f.length()]; + f.readFully(data); + f.close(); + + ByteBuffer bb = ByteBuffer.wrap(data); + Monster monster = Monster.getRootAsMonster(bb); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Now you can access the data from the `Monster monster`: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} + short hp = monster.hp(); + Vec3 pos = monster.pos(); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +#### Differences in C-sharp + + +C# code works almost identically to Java, with only a few minor differences. +You can see an example of C# code in +`tests/FlatBuffers.Test/FlatBuffersExampleTests.cs` or +`samples/SampleBinary.cs`. + +First of all, naming follows standard C# style with `PascalCasing` identifiers, +e.g. `GetRootAsMyRootType`. Also, values (except vectors and unions) are +available as properties instead of parameterless accessor methods as in Java. +The performance-enhancing methods to which you can pass an already created +object are prefixed with `Get`, e.g.: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cs} + // property + var pos = monster.Pos; + + // method filling a preconstructed object + var preconstructedPos = new Vec3(); + monster.GetPos(preconstructedPos); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +## Text parsing + +There currently is no support for parsing text (Schema's and JSON) directly +from Java or C#, though you could use the C++ parser through native call +interfaces available to each language. Please see the +C++ documentation for more on text parsing. + +
diff --git a/docs/source/JavaScriptUsage.md b/docs/source/JavaScriptUsage.md new file mode 100755 index 0000000000000..c321c95e3c8c2 --- /dev/null +++ b/docs/source/JavaScriptUsage.md @@ -0,0 +1,105 @@ +Use in JavaScript {#flatbuffers_guide_use_javascript} +================= + +## Before you get started + +Before diving into the FlatBuffers usage in JavaScript, it should be noted that +the [Tutorial](@ref flatbuffers_guide_tutorial) page has a complete guide to +general FlatBuffers usage in all of the supported languages +(including JavaScript). This page is specifically designed to cover the nuances +of FlatBuffers usage in JavaScript. + +You should also have read the [Building](@ref flatbuffers_guide_building) +documentation to build `flatc` and should be familiar with +[Using the schema compiler](@ref flatbuffers_guide_using_schema_compiler) and +[Writing a schema](@ref flatbuffers_guide_writing_schema). + +## FlatBuffers JavaScript library code location + +The code for the FlatBuffers JavaScript library can be found at +`flatbuffers/js`. You can browse the library code on the [FlatBuffers +GitHub page](https://github.com/google/flatbuffers/tree/master/js). + +## Testing the FlatBuffers JavaScript library + +The code to test the JavaScript library can be found at `flatbuffers/tests`. +The test code itself is located in [JavaScriptTest.js](https://github.com/ +google/flatbuffers/blob/master/tests/JavaScriptTest.js). + +To run the tests, use the [JavaScriptTest.sh](https://github.com/google/ +flatbuffers/blob/master/tests/JavaScriptTest.sh) shell script. + +*Note: The JavaScript test file requires [Node.js](https://nodejs.org/en/).* + +## Using the FlatBuffers JavaScript libary + +*Note: See [Tutorial](@ref flatbuffers_guide_tutorial) for a more in-depth +example of how to use FlatBuffers in JavaScript.* + +FlatBuffers supports both reading and writing FlatBuffers in JavaScript. + +To use FlatBuffers in your own code, first generate JavaScript classes from your +schema with the `--js` option to `flatc`. Then you can include both FlatBuffers +and the generated code to read or write a FlatBuffer. + +For example, here is how you would read a FlatBuffer binary file in Javascript: +First, include the library and generated code. Then read the file into an +`Uint8Array`. Make a `flatbuffers.ByteBuffer` out of the `Uint8Array`, and pass +the ByteBuffer to the `getRootAsMonster` function. + +*Note: Both JavaScript module loaders (e.g. Node.js) and browser-based +HTML/JavaScript code segments are shown below in the following snippet:* + +~~~{.js} + // Note: These require functions are specific to JavaScript module loaders + // (namely, Node.js). See below for a browser-based example. + var fs = require('fs'); + + var flatbuffers = require('../flatbuffers').flatbuffers; + var MyGame = require('./monster_generated').MyGame; + + var data = new Uint8Array(fs.readFileSync('monster.dat')); + var buf = new flatbuffers.ByteBuffer(data); + + var monster = MyGame.Example.Monster.getRootAsMonster(buf); + + //--------------------------------------------------------------------------// + + // Note: This code is specific to browser-based HTML/JavaScript. See above + // for the code using JavaScript module loaders (e.g. Node.js). + + + + + // Open the HTML file in a browser and select "monster.dat" from with the + // field. + +~~~ + +Now you can access values like this: + +~~~{.js} + var hp = monster.hp(); + var pos = monster.pos(); +~~~ + +## Text parsing FlatBuffers in JavaScript + +There currently is no support for parsing text (Schema's and JSON) directly +from JavaScript. diff --git a/docs/source/JavaUsage.md b/docs/source/JavaUsage.md deleted file mode 100755 index 4819e5b7927d3..0000000000000 --- a/docs/source/JavaUsage.md +++ /dev/null @@ -1,224 +0,0 @@ -# Use in Java/C-sharp - -FlatBuffers supports reading and writing binary FlatBuffers in Java and C#. -Generate code for Java with the `-j` option to `flatc`, or for C# with `-n` -(think .Net). - -Note that this document is from the perspective of Java. Code for both languages -is generated in the same way, with only minor differences. These differences -are [explained in a section below](#differences-in-c-sharp). - -See `javaTest.java` for an example. Essentially, you read a FlatBuffer binary -file into a `byte[]`, which you then turn into a `ByteBuffer`, which you pass to -the `getRootAsMyRootType` function: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - ByteBuffer bb = ByteBuffer.wrap(data); - Monster monster = Monster.getRootAsMonster(bb); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Now you can access values much like C++: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - short hp = monster.hp(); - Vec3 pos = monster.pos(); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Note that whenever you access a new object like in the `pos` example above, -a new temporary accessor object gets created. If your code is very performance -sensitive (you iterate through a lot of objects), there's a second `pos()` -method to which you can pass a `Vec3` object you've already created. This allows -you to reuse it across many calls and reduce the amount of object allocation -(and thus garbage collection) your program does. - -Java does not support unsigned scalars. This means that any unsigned types you -use in your schema will actually be represented as a signed value. This means -all bits are still present, but may represent a negative value when used. -For example, to read a `byte b` as an unsigned number, you can do: -`(short)(b & 0xFF)` - -The default string accessor (e.g. `monster.name()`) currently always create -a new Java `String` when accessed, since FlatBuffer's UTF-8 strings can't be -used in-place by `String`. Alternatively, use `monster.nameAsByteBuffer()` -which returns a `ByteBuffer` referring to the UTF-8 data in the original -`ByteBuffer`, which is much more efficient. The `ByteBuffer`'s `position` -points to the first character, and its `limit` to just after the last. - -Vector access is also a bit different from C++: you pass an extra index -to the vector field accessor. Then a second method with the same name -suffixed by `Length` let's you know the number of elements you can access: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - for (int i = 0; i < monster.inventoryLength(); i++) - monster.inventory(i); // do something here -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Alternatively, much like strings, you can use `monster.inventoryAsByteBuffer()` -to get a `ByteBuffer` referring to the whole vector. Use `ByteBuffer` methods -like `asFloatBuffer` to get specific views if needed. - -If you specified a file_indentifier in the schema, you can query if the -buffer is of the desired type before accessing it using: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - if (Monster.MonsterBufferHasIdentifier(bb)) ... -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -## Buffer construction in Java - -You can also construct these buffers in Java using the static methods found -in the generated code, and the FlatBufferBuilder class: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - FlatBufferBuilder fbb = new FlatBufferBuilder(); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Create strings: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - int str = fbb.createString("MyMonster"); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Create a table with a struct contained therein: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - Monster.startMonster(fbb); - Monster.addPos(fbb, Vec3.createVec3(fbb, 1.0f, 2.0f, 3.0f, 3.0, (byte)4, (short)5, (byte)6)); - Monster.addHp(fbb, (short)80); - Monster.addName(fbb, str); - Monster.addInventory(fbb, inv); - Monster.addTest_type(fbb, (byte)1); - Monster.addTest(fbb, mon2); - Monster.addTest4(fbb, test4s); - int mon = Monster.endMonster(fbb); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For some simpler types, you can use a convenient `create` function call that -allows you to construct tables in one function call. This example definition -however contains an inline struct field, so we have to create the table -manually. -This is to create the buffer without using temporary object allocation. - -It's important to understand that fields that are structs are inline (like -`Vec3` above), and MUST thus be created between the start and end calls of -a table. Everything else (other tables, strings, vectors) MUST be created -before the start of the table they are referenced in. - -Structs do have convenient methods that even have arguments for nested structs. - -As you can see, references to other objects (e.g. the string above) are simple -ints, and thus do not have the type-safety of the Offset type in C++. Extra -care must thus be taken that you set the right offset on the right field. - -Vectors can be created from the corresponding Java array like so: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - int inv = Monster.createInventoryVector(fbb, new byte[] { 0, 1, 2, 3, 4 }); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This works for arrays of scalars and (int) offsets to strings/tables, -but not structs. If you want to write structs, or what you want to write -does not sit in an array, you can also use the start/end pattern: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - Monster.startInventoryVector(fbb, 5); - for (byte i = 4; i >=0; i--) fbb.addByte(i); - int inv = fbb.endVector(); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can use the generated method `startInventoryVector` to conveniently call -`startVector` with the right element size. You pass the number of -elements you want to write. Note how you write the elements backwards since -the buffer is being constructed back to front. You then pass `inv` to the -corresponding `Add` call when you construct the table containing it afterwards. - -There are `add` functions for all the scalar types. You use `addOffset` for -any previously constructed objects (such as other tables, strings, vectors). -For structs, you use the appropriate `create` function in-line, as shown -above in the `Monster` example. - -To finish the buffer, call: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - Monster.finishMonsterBuffer(fbb, mon); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The buffer is now ready to be transmitted. It is contained in the `ByteBuffer` -which you can obtain from `fbb.dataBuffer()`. Importantly, the valid data does -not start from offset 0 in this buffer, but from `fbb.dataBuffer().position()` -(this is because the data was built backwards in memory). -It ends at `fbb.capacity()`. - - -## Differences in C-sharp - -C# code works almost identically to Java, with only a few minor differences. -You can see an example of C# code in `tests/FlatBuffers.Test/FlatBuffersExampleTests.cs`. - -First of all, naming follows standard C# style with `PascalCasing` identifiers, -e.g. `GetRootAsMyRootType`. Also, values (except vectors and unions) are available -as properties instead of parameterless accessor methods as in Java. The -performance-enhancing methods to which you can pass an already created object -are prefixed with `Get`, e.g.: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cs} - // property - var pos = monster.Pos; - // method filling a preconstructed object - var preconstructedPos = new Vec3(); - monster.GetPos(preconstructedPos); -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -## Text parsing - -There currently is no support for parsing text (Schema's and JSON) directly -from Java or C#, though you could use the C++ parser through native call -interfaces available to each language. Please see the -C++ documentation for more on text parsing. - -### Mutating FlatBuffers - -As you saw above, typically once you have created a FlatBuffer, it is -read-only from that moment on. There are however cases where you have just -received a FlatBuffer, and you'd like to modify something about it before -sending it on to another recipient. With the above functionality, you'd have -to generate an entirely new FlatBuffer, while tracking what you modify in your -own data structures. This is inconvenient. - -For this reason FlatBuffers can also be mutated in-place. While this is great -for making small fixes to an existing buffer, you generally want to create -buffers from scratch whenever possible, since it is much more efficient and -the API is much more general purpose. - -To get non-const accessors, invoke `flatc` with `--gen-mutable`. - -You now can: - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} - Monster monster = Monster.getRootAsMonster(bb); - monster.mutateHp(10); // Set table field. - monster.pos().mutateZ(4); // Set struct field. - monster.mutateInventory(0, 1); // Set vector element. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We use the somewhat verbose term `mutate` instead of `set` to indicate that -this is a special use case, not to be confused with the default way of -constructing FlatBuffer data. - -After the above mutations, you can send on the FlatBuffer to a new recipient -without any further work! - -Note that any `mutate` functions on tables return a boolean, which is false -if the field we're trying to set isn't present in the buffer. Fields are not -present if they weren't set, or even if they happen to be equal to the -default value. For example, in the creation code above we set the `mana` field -to `150`, which is the default value, so it was never stored in the buffer. -Trying to call mutateMana() on such data will return false, and the value won't -actually be modified! - -One way to solve this is to call `forceDefaults()` on a -`FlatBufferBuilder` to force all fields you set to actually be written. This -of course increases the size of the buffer somewhat, but this may be -acceptable for a mutable buffer. diff --git a/docs/source/PHPUsage.md b/docs/source/PHPUsage.md new file mode 100644 index 0000000000000..cdff44954dd6b --- /dev/null +++ b/docs/source/PHPUsage.md @@ -0,0 +1,89 @@ +Use in PHP {#flatbuffers_guide_use_php} +========== + +## Before you get started + +Before diving into the FlatBuffers usage in PHP, it should be noted that +the [Tutorial](@ref flatbuffers_guide_tutorial) page has a complete guide to +general FlatBuffers usage in all of the supported languages +(including PHP). This page is specifically designed to cover the nuances of +FlatBuffers usage in PHP. + +You should also have read the [Building](@ref flatbuffers_guide_building) +documentation to build `flatc` and should be familiar with +[Using the schema compiler](@ref flatbuffers_guide_using_schema_compiler) and +[Writing a schema](@ref flatbuffers_guide_writing_schema). + +## FlatBuffers PHP library code location + +The code for FlatBuffers PHP library can be found at `flatbuffers/php`. You +can browse the library code on the [FlatBuffers +GitHub page](https://github.com/google/flatbuffers/tree/master/php). + +## Testing the FlatBuffers JavaScript library + +The code to test the PHP library can be found at `flatbuffers/tests`. +The test code itself is located in [phpTest.php](https://github.com/google/ +flatbuffers/blob/master/tests/phpTest.php). + +You can run the test with `php phpTest.php` from the command line. + +*Note: The PHP test file requires +[PHP](http://php.net/manual/en/install.php) to be installed.* + +## Using theFlatBuffers PHP library + +*Note: See [Tutorial](@ref flatbuffers_guide_tutorial) for a more in-depth +example of how to use FlatBuffers in PHP.* + +FlatBuffers supports both reading and writing FlatBuffers in PHP. + +To use FlatBuffers in your own code, first generate PHP classes from your schema +with the `--php` option to `flatc`. Then you can include both FlatBuffers and +the generated code to read or write a FlatBuffer. + +For example, here is how you would read a FlatBuffer binary file in PHP: +First, include the library and generated code (using the PSR `autoload` +function). Then you can read a FlatBuffer binary file, which you +pass the contents of to the `GetRootAsMonster` function: + +~~~{.php} + // It is recommended that your use PSR autoload when using FlatBuffers in PHP. + // Here is an example: + function __autoload($class_name) { + // The last segment of the class name matches the file name. + $class = substr($class_name, strrpos($class_name, "\\") + 1); + $root_dir = join(DIRECTORY_SEPARATOR, array(dirname(dirname(__FILE__)))); // `flatbuffers` root. + + // Contains the `*.php` files for the FlatBuffers library and the `flatc` generated files. + $paths = array(join(DIRECTORY_SEPARATOR, array($root_dir, "php")), + join(DIRECTORY_SEPARATOR, array($root_dir, "tests", "MyGame", "Example"))); + foreach ($paths as $path) { + $file = join(DIRECTORY_SEPARATOR, array($path, $class . ".php")); + if (file_exists($file)) { + require($file); + break; + } + } + + // Read the contents of the FlatBuffer binary file. + $filename = "monster.dat"; + $handle = fopen($filename, "rb"); + $contents = $fread($handle, filesize($filename)); + fclose($handle); + + // Pass the contents to `GetRootAsMonster`. + $monster = \MyGame\Example\Monster::GetRootAsMonster($contents); +~~~ + +Now you can access values like this: + +~~~{.php} + $hp = $monster->GetHp(); + $pos = $monster->GetPos(); +~~~ + +## Text Parsing + +There currently is no support for parsing text (Schema's and JSON) directly +from PHP. diff --git a/docs/source/PythonUsage.md b/docs/source/PythonUsage.md index 33008b3fe1254..2a0cddeea66bb 100755 --- a/docs/source/PythonUsage.md +++ b/docs/source/PythonUsage.md @@ -1,115 +1,73 @@ -# Use in Python +Use in Python {#flatbuffers_guide_use_python} +============= -There's experimental support for reading FlatBuffers in Python. Generate -code for Python with the `-p` option to `flatc`. +## Before you get started -See `py_test.py` for an example. You import the generated code, read a -FlatBuffer binary file into a `bytearray`, which you pass to the -`GetRootAsMonster` function: +Before diving into the FlatBuffers usage in Python, it should be noted that the +[Tutorial](@ref flatbuffers_guide_tutorial) page has a complete guide to general +FlatBuffers usage in all of the supported languages (including Python). This +page is designed to cover the nuances of FlatBuffers usage, specific to +Python. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py} - import MyGame.Example as example - import flatbuffers +You should also have read the [Building](@ref flatbuffers_guide_building) +documentation to build `flatc` and should be familiar with +[Using the schema compiler](@ref flatbuffers_guide_using_schema_compiler) and +[Writing a schema](@ref flatbuffers_guide_writing_schema). - buf = open('monster.dat', 'rb').read() - buf = bytearray(buf) - monster = example.GetRootAsMonster(buf, 0) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +## FlatBuffers Python library code location -Now you can access values like this: +The code for the FlatBuffers Python library can be found at +`flatbuffers/python/flatbuffers`. You can browse the library code on the +[FlatBuffers GitHub page](https://github.com/google/flatbuffers/tree/master/ +python). -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py} - hp = monster.Hp() - pos = monster.Pos() -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +## Testing the FlatBuffers Python library -To access vectors you pass an extra index to the -vector field accessor. Then a second method with the same name suffixed -by `Length` let's you know the number of elements you can access: +The code to test the Python library can be found at `flatbuffers/tests`. +The test code itself is located in [py_test.py](https://github.com/google/ +flatbuffers/blob/master/tests/py_test.py). -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py} - for i in xrange(monster.InventoryLength()): - monster.Inventory(i) # do something here -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To run the tests, use the [PythonTest.sh](https://github.com/google/flatbuffers/ +blob/master/tests/PythonTest.sh) shell script. -You can also construct these buffers in Python using the functions found -in the generated code, and the FlatBufferBuilder class: +*Note: This script requires [python](https://www.python.org/) to be +installed.* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py} - builder = flatbuffers.Builder(0) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +## Using the FlatBuffers Python library -Create strings: +*Note: See [Tutorial](@ref flatbuffers_guide_tutorial) for a more in-depth +example of how to use FlatBuffers in Python.* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py} - s = builder.CreateString("MyMonster") -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +There is support for both reading and writing FlatBuffers in Python. -Create a table with a struct contained therein: +To use FlatBuffers in your own code, first generate Python classes from your +schema with the `--python` option to `flatc`. Then you can include both +FlatBuffers and the generated code to read or write a FlatBuffer. + +For example, here is how you would read a FlatBuffer binary file in Python: +First, import the library and the generated code. Then read a FlatBuffer binary +file into a `bytearray`, which you pass to the `GetRootAsMonster` function: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py} - example.MonsterStart(builder) - example.MonsterAddPos(builder, example.CreateVec3(builder, 1.0, 2.0, 3.0, 3.0, 4, 5, 6)) - example.MonsterAddHp(builder, 80) - example.MonsterAddName(builder, str) - example.MonsterAddInventory(builder, inv) - example.MonsterAddTest_Type(builder, 1) - example.MonsterAddTest(builder, mon2) - example.MonsterAddTest4(builder, test4s) - mon = example.MonsterEnd(builder) - - final_flatbuffer = builder.Output() -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + import MyGame.Example as example + import flatbuffers -Unlike C++, Python does not support table creation functions like 'createMonster()'. -This is to create the buffer without -using temporary object allocation (since the `Vec3` is an inline component of -`Monster`, it has to be created right where it is added, whereas the name and -the inventory are not inline, and **must** be created outside of the table -creation sequence). -Structs do have convenient methods that allow you to construct them in one call. -These also have arguments for nested structs, e.g. if a struct has a field `a` -and a nested struct field `b` (which has fields `c` and `d`), then the arguments -will be `a`, `c` and `d`. + buf = open('monster.dat', 'rb').read() + buf = bytearray(buf) + monster = example.GetRootAsMonster(buf, 0) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Vectors also use this start/end pattern to allow vectors of both scalar types -and structs: +Now you can access values like this: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py} - example.MonsterStartInventoryVector(builder, 5) - i = 4 - while i >= 0: - builder.PrependByte(byte(i)) - i -= 1 - - inv = builder.EndVector(5) + hp = monster.Hp() + pos = monster.Pos() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The generated method 'StartInventoryVector' is provided as a convenience -function which calls 'StartVector' with the correct element size of the vector -type which in this case is 'ubyte' or 1 byte per vector element. -You pass the number of elements you want to write. -You write the elements backwards since the buffer -is being constructed back to front. Use the correct `Prepend` call for the type, -or `PrependUOffsetT` for offsets. You then pass `inv` to the corresponding -`Add` call when you construct the table containing it afterwards. - -There are `Prepend` functions for all the scalar types. You use -`PrependUOffset` for any previously constructed objects (such as other tables, -strings, vectors). For structs, you use the appropriate `create` function -in-line, as shown above in the `Monster` example. - -Once you're done constructing a buffer, you call `Finish` with the root object -offset (`mon` in the example above). Your data now resides in Builder.Bytes. -Important to note is that the real data starts at the index indicated by Head(), -for Offset() bytes (this is because the buffer is constructed backwards). -If you wanted to read the buffer right after creating it (using -`GetRootAsMonster` above), the second argument, instead of `0` would thus -also be `Head()`. - ## Text Parsing There currently is no support for parsing text (Schema's and JSON) directly from Python, though you could use the C++ parser through SWIG or ctypes. Please see the C++ documentation for more on text parsing. +
diff --git a/docs/source/README_TO_GENERATE_DOCS.md b/docs/source/README_TO_GENERATE_DOCS.md new file mode 100644 index 0000000000000..5df0b6a8b9be1 --- /dev/null +++ b/docs/source/README_TO_GENERATE_DOCS.md @@ -0,0 +1,32 @@ +## Prerequisites + +To generate the docs for FlatBuffers from the source files, you +will first need to install two programs. + +1. You will need to install `doxygen`. See + [Download Doxygen](http://www.stack.nl/~dimitri/doxygen/download.html). + +2. You will need to install `doxypypy` to format python comments appropriately. + Install it from [here](https://github.com/Feneric/doxypypy). + +*Note: You will need both `doxygen` and `doxypypy` to be in your +[PATH](https://en.wikipedia.org/wiki/PATH_(variable)) environment variable.* + +After you have both of those files installed and in your path, you need to +set up the `py_filter` to invoke `doxypypy` from `doxygen`. + +Follow the steps +[here](https://github.com/Feneric/doxypypy#invoking-doxypypy-from-doxygen). + +## Generating Docs + +Run the following commands to generate the docs: + +`cd flatbuffers/docs/source` +`doxygen` + +The output is placed in `flatbuffers/docs/html`. + +*Note: The Go API Reference code must be generated ahead of time. For +instructions on how to regenerated this file, please read the comments +in `GoApi.md`.* diff --git a/docs/source/Schemas.md b/docs/source/Schemas.md index c38508d8ed063..2ed3005d99e35 100755 --- a/docs/source/Schemas.md +++ b/docs/source/Schemas.md @@ -1,7 +1,8 @@ -# Writing a schema +Writing a schema {#flatbuffers_guide_writing_schema} +================ -The syntax of the schema language (aka IDL, Interface Definition -Language) should look quite familiar to users of any of the C family of +The syntax of the schema language (aka IDL, [Interface Definition Language][]) +should look quite familiar to users of any of the C family of languages, and also to users of other IDLs. Let's look at an example first: @@ -34,14 +35,14 @@ first: root_type Monster; -(Weapon & Pickup not defined as part of this example). +(`Weapon` & `Pickup` not defined as part of this example). ### Tables Tables are the main way of defining objects in FlatBuffers, and consist of a name (here `Monster`) and a list of fields. Each field has a name, -a type, and optionally a default value (if omitted, it defaults to 0 / -NULL). +a type, and optionally a default value (if omitted, it defaults to `0` / +`NULL`). Each field is optional: It does not have to appear in the wire representation, and you can choose to omit fields for each individual @@ -68,7 +69,8 @@ and backwards compatibility. Note that: - You may change field names and table names, if you're ok with your code breaking until you've renamed them there too. - +See "Schema evolution examples" below for more on this +topic. ### Structs @@ -84,13 +86,13 @@ parent object, and use no virtual table). Built-in scalar types are: -- 8 bit: `byte ubyte bool` +- 8 bit: `byte`, `ubyte`, `bool` -- 16 bit: `short ushort` +- 16 bit: `short`, `ushort` -- 32 bit: `int uint float` +- 32 bit: `int`, `uint`, `float` -- 64 bit: `long ulong double` +- 64 bit: `long`, `ulong`, `double` Built-in non-scalar types: @@ -110,18 +112,19 @@ high bit yet. ### (Default) Values -Values are a sequence of digits, optionally followed by a `.` and more digits -for float constants, and optionally prefixed by a `-`. Floats may end with an -`e` or `E`, followed by a `+` or `-` and more digits (scientific notation). +Values are a sequence of digits. Values may be optionally followed by a decimal +point (`.`) and more digits, for float constants, or optionally prefixed by +a `-`. Floats may also be in scientific notation; optionally ending with an `e` +or `E`, followed by a `+` or `-` and more digits. Only scalar values can have defaults, non-scalar (string/vector/table) fields -default to NULL when not present. +default to `NULL` when not present. You generally do not want to change default values after they're initially defined. Fields that have the default value are not actually stored in the serialized data but are generated in code, so when you change the default, you'd now get a different value than from code generated from an older version of -the schema. There are situations however where this may be +the schema. There are situations, however, where this may be desirable, especially if you can ensure a simultaneous rebuild of all code. @@ -133,11 +136,15 @@ is `0`. As you can see in the enum declaration, you specify the underlying integral type of the enum with `:` (in this case `byte`), which then determines the type of any fields declared with this enum type. +Typically, enum values should only ever be added, never removed (there is no +deprecation for enums). This requires code to handle forwards compatibility +itself, by handling unknown enum values. + ### Unions Unions share a lot of properties with enums, but instead of new names for constants, you use names of tables. You can then declare -a union field which can hold a reference to any of those types, and +a union field, which can hold a reference to any of those types, and additionally a hidden field with the suffix `_type` is generated that holds the corresponding enum value, allowing you to know which type to cast to at runtime. @@ -173,7 +180,7 @@ included files (those you still generate separately). ### Root type This declares what you consider to be the root table (or struct) of the -serialized data. This is particular important for parsing JSON data, +serialized data. This is particularly important for parsing JSON data, which doesn't include object type information. ### File identification and extension @@ -229,7 +236,7 @@ in the corresponding C++ code. Multiple such lines per item are allowed. Attributes may be attached to a declaration, behind a field, or after the name of a table/struct/enum/union. These may either have a value or -not. Some attributes like `deprecated` are understood by the compiler, +not. Some attributes like `deprecated` are understood by the compiler; user defined ones need to be declared with the attribute declaration (like `priority` in the example above), and are available to query if you parse the schema at runtime. @@ -308,6 +315,9 @@ JSON: you do when serializing from code. E.g. for a field `foo`, you must add a field `foo_type: FooOne` right before the `foo` field, where `FooOne` would be the table out of the union you want to use. +- A field that has the value `null` (e.g. `field: null`) is intended to + have the default value for that field (thus has the same effect as if + that field wasn't specified at all). When parsing JSON, it recognizes the following escape codes in strings: @@ -351,3 +361,68 @@ the world. If this is not practical for you, use explicit field ids, which should always generate a merge conflict if two people try to allocate the same id. +### Schema evolution examples + +Some examples to clarify what happens as you change a schema: + +If we have the following original schema: + + table { a:int; b:int; } + +And we extend it: + + table { a:int; b:int; c:int; } + +This is ok. Code compiled with the old schema reading data generated with the +new one will simply ignore the presence of the new field. Code compiled with the +new schema reading old data will get the default value for `c` (which is 0 +in this case, since it is not specified). + + table { a:int (deprecated); b:int; } + +This is also ok. Code compiled with the old schema reading newer data will now +always get the default value for `a` since it is not present. Code compiled +with the new schema now cannot read nor write `a` anymore (any existing code +that tries to do so will result in compile errors), but can still read +old data (they will ignore the field). + + table { c:int a:int; b:int; } + +This is NOT ok, as this makes the schemas incompatible. Old code reading newer +data will interpret `c` as if it was `a`, and new code reading old data +accessing `a` will instead receive `b`. + + table { c:int (id: 2); a:int (id: 0); b:int (id: 1); } + +This is ok. If your intent was to order/group fields in a way that makes sense +semantically, you can do so using explicit id assignment. Now we are compatible +with the original schema, and the fields can be ordered in any way, as long as +we keep the sequence of ids. + + table { b:int; } + +NOT ok. We can only remove a field by deprecation, regardless of wether we use +explicit ids or not. + + table { a:uint; b:uint; } + +This is MAYBE ok, and only in the case where the type change is the same size, +like here. If old data never contained any negative numbers, this will be +safe to do. + + table { a:int = 1; b:int = 2; } + +Generally NOT ok. Any older data written that had 0 values were not written to +the buffer, and rely on the default value to be recreated. These will now have +those values appear to `1` and `2` instead. There may be cases in which this +is ok, but care must be taken. + + table { aa:int; bb:int; } + +Occasionally ok. You've renamed fields, which will break all code (and JSON +files!) that use this schema, but as long as the change is obvious, this is not +incompatible with the actual binary buffers, since those only ever address +fields by id/offset. +
+ + [Interface Definition Language]: https://en.wikipedia.org/wiki/Interface_description_language diff --git a/docs/source/Support.md b/docs/source/Support.md index 7df42499bd9a4..ba8c76579f9f9 100755 --- a/docs/source/Support.md +++ b/docs/source/Support.md @@ -1,4 +1,5 @@ -# Platform / Language / Feature support +Platform / Language / Feature support {#flatbuffers_support} +===================================== FlatBuffers is actively being worked on, which means that certain platform / language / feature combinations may not be available yet. @@ -39,3 +40,5 @@ Primary authors (github) | gwvo | gwvo | ev*/js*| rw | rw | ev * js = jonsimantov * mik = mikkelfj * ch = chobie + +
diff --git a/docs/source/Tutorial.md b/docs/source/Tutorial.md new file mode 100644 index 0000000000000..74941be8c0c2c --- /dev/null +++ b/docs/source/Tutorial.md @@ -0,0 +1,1720 @@ +Tutorial {#flatbuffers_guide_tutorial} +======== + +## Overview + +This tutorial provides a basic example of how to work with +[FlatBuffers](@ref flatbuffers_overview). We will step through a simple example +application, which shows you how to: + + - Write a FlatBuffer `schema` file. + - Use the `flatc` FlatBuffer compiler. + - Parse [JSON](http://json.org) files that conform to a schema into + FlatBuffer binary files. + - Use the generated files in many of the supported languages (such as C++, + Java, and more.) + +During this example, imagine that you are creating a game where the main +character, the hero of the story, needs to slay some `orc`s. We will walk +through each step necessary to create this monster type using FlatBuffers. + +Please select your desired language for our quest: + +\htmlonly + +\endhtmlonly + +\htmlonly + +\endhtmlonly + +## Where to Find the Example Code + +Samples demonstating the concepts in this example are located in the source code +package, under the `samples` directory. You can browse the samples on GitHub +[here](https://github.com/google/flatbuffers/tree/master/samples). + +For your chosen language, please cross-reference with: + +
+*Note: Since we passing `150` as the `mana` field, which happens to be the +default value, the field will not actually be written to the buffer, since the +default value will be returned on query anyway. This is a nice space savings, +especially if default values are common in your data. It also means that you do +not need to be worried of adding a lot of fields that are only used in a small +number of instances, as it will not bloat the buffer if unused.* +
+If you do not wish to set every field in a `table`, it may be more convenient to +manually set each field of your monster, instead of calling `CreateMonster()`. +The following snippet is functionally equivalent to the above code, but provides +a bit more flexibility. +
+~~~{.cpp} + // You can use this code instead of `CreateMonster()`, to create our orc + // manually. + MonsterBuilder monster_builder(builder); + monster_builder.add_pos(&pos); + monster_builder.add_hp(hp); + monster_builder.add_name(name); + monster_builder.add_inventory(inventory); + monster_builder.add_color(Color_Red); + monster_builder.add_weapons(weapons); + monster_builder.add_equipped_type(Equipment_Weapon); + monster_builder.add_equpped(axe); + auto orc = monster_builder.Finish(); +~~~ +
diff --git a/docs/source/WhitePaper.md b/docs/source/WhitePaper.md index 71b2b240415d3..e504ada41d5f1 100755 --- a/docs/source/WhitePaper.md +++ b/docs/source/WhitePaper.md @@ -1,4 +1,5 @@ -# FlatBuffers white paper +FlatBuffers white paper {#flatbuffers_white_paper} +======================= This document tries to shed some light on to the "why" of FlatBuffers, a new serialization library. @@ -124,4 +125,4 @@ offered by .proto files in the following ways: - A parser that can deal with both schemas and data definitions (JSON compatible) uniformly. - +
diff --git a/docs/source/doxyfile b/docs/source/doxyfile index 2cb373bc6b3e2..ba6fbcbd6310b 100755 --- a/docs/source/doxyfile +++ b/docs/source/doxyfile @@ -88,7 +88,7 @@ OUTPUT_LANGUAGE = English # documentation (similar to Javadoc). Set to NO to disable this. # The default value is: YES. -BRIEF_MEMBER_DESC = NO +BRIEF_MEMBER_DESC = YES # If the REPEAT_BRIEF tag is set to YES doxygen will prepend the brief # description of a member or function before the detailed description @@ -97,7 +97,7 @@ BRIEF_MEMBER_DESC = NO # brief descriptions will be completely suppressed. # The default value is: YES. -REPEAT_BRIEF = NO +REPEAT_BRIEF = YES # This tag implements a quasi-intelligent brief description abbreviator that is # used to form the text in various listings. Each string in this list, if found @@ -177,7 +177,7 @@ SHORT_NAMES = NO # description.) # The default value is: NO. -JAVADOC_AUTOBRIEF = NO +JAVADOC_AUTOBRIEF = YES # If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first # line (until the first dot) of a Qt-style comment as the brief description. If @@ -203,7 +203,7 @@ MULTILINE_CPP_IS_BRIEF = NO # documentation from any documented member that it re-implements. # The default value is: YES. -INHERIT_DOCS = NO +INHERIT_DOCS = YES # If the SEPARATE_MEMBER_PAGES tag is set to YES, then doxygen will produce a # new page for each member. If set to NO, the documentation of a member will be @@ -216,7 +216,7 @@ SEPARATE_MEMBER_PAGES = NO # uses this value to replace tabs by spaces in code fragments. # Minimum value: 1, maximum value: 16, default value: 4. -TAB_SIZE = 1 +TAB_SIZE = 2 # This tag can be used to specify a number of aliases that act as commands in # the documentation. An alias has the form: @@ -296,7 +296,9 @@ MARKDOWN_SUPPORT = YES # or globally by setting AUTOLINK_SUPPORT to NO. # The default value is: YES. -AUTOLINK_SUPPORT = YES +AUTOLINK_SUPPORT = NO # Due to the multiple languages included in the API + # reference for FlatBuffers, the Auto-links were + # wrong more often than not. # If you use STL classes (i.e. std::string, std::vector, etc.) but do not want # to include (a tag file for) the STL sources as input, then you should set this @@ -347,7 +349,7 @@ DISTRIBUTE_GROUP_DOC = NO # \nosubgrouping command. # The default value is: YES. -SUBGROUPING = NO +SUBGROUPING = YES # When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions # are shown inside the group in which they are included (e.g. using \ingroup) @@ -424,7 +426,7 @@ EXTRACT_PACKAGE = NO # included in the documentation. # The default value is: NO. -EXTRACT_STATIC = NO +EXTRACT_STATIC = YES # If the EXTRACT_LOCAL_CLASSES tag is set to YES classes (and structs) defined # locally in source files will be included in the documentation. If set to NO @@ -508,7 +510,7 @@ HIDE_SCOPE_NAMES = NO # the files that are included by a file in the documentation of that file. # The default value is: YES. -SHOW_INCLUDE_FILES = NO +SHOW_INCLUDE_FILES = YES # If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include # files with double quotes in the documentation rather than with sharp brackets. @@ -520,21 +522,21 @@ FORCE_LOCAL_INCLUDES = NO # documentation for inline members. # The default value is: YES. -INLINE_INFO = NO +INLINE_INFO = YES # If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the # (detailed) documentation of file and class members alphabetically by member # name. If set to NO the members will appear in declaration order. # The default value is: YES. -SORT_MEMBER_DOCS = NO +SORT_MEMBER_DOCS = YES # If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief # descriptions of file, namespace and class members alphabetically by member # name. If set to NO the members will appear in declaration order. # The default value is: NO. -SORT_BRIEF_DOCS = NO +SORT_BRIEF_DOCS = YES # If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the # (brief and detailed) documentation of class members so that constructors and @@ -600,7 +602,7 @@ GENERATE_BUGLIST = NO # the documentation. # The default value is: YES. -GENERATE_DEPRECATEDLIST= NO +GENERATE_DEPRECATEDLIST= YES # The ENABLED_SECTIONS tag can be used to enable conditional documentation # sections, marked by \if
* The expected sequence of calls is: *
-
*
- Start the array using this method. - *
- Call {@link #addOffset(int)} {@code num_elems} number of times to set + *
- Call {@link #addOffset(int)} `num_elems` number of times to set * the offset of each element in the array. *
- Call {@link #endVector()} to retrieve the offset of the array. *
* For example, using the "Monster" code found on the - * landing page. An - * object of type {@code Monster} can be created using the following code: + * @ref flatbuffers_guide_use_java_c-sharp "landing page". An + * object of type `Monster` can be created using the following code: * *
{@code
* int testArrayOfString = Monster.createTestarrayofstringVector(fbb, new int[] {
@@ -351,14 +453,14 @@ public void Nested(int obj) {
*
* Here:
*
-
- *
- The call to {@code Monster#startMonster(FlatBufferBuilder)} will call this + *
- The call to `Monster#startMonster(FlatBufferBuilder)` will call this * method with the right number of fields set. - *
- {@code Monster#endMonster(FlatBufferBuilder)} will ensure {@link #endObject()} is called. + *
- `Monster#endMonster(FlatBufferBuilder)` will ensure {@link #endObject()} is called. *
* It's not recommended to call this method directly. If it's called manually, you must ensure
* to audit all calls to it whenever fields are added or removed from your schema. This is
- * automatically done by the code generated by the {@code FlatBuffers} compiler.
+ * automatically done by the code generated by the `FlatBuffers` compiler.
*
* @param numfields The number of fields found in this object.
*/
@@ -371,17 +473,101 @@ public void startObject(int numfields) {
object_start = offset();
}
- // Add a scalar to a table at `o` into its vtable, with value `x` and default `d`
+ /**
+ * Add a `boolean` to a table at `o` into its vtable, with value `x` and default `d`.
+ *
+ * @param o The index into the vtable.
+ * @param x A `boolean` to put into the buffer, depending on how defaults are handled. If
+ * `force_defaults` is `false`, compare `x` against the default value `d`. If `x` contains the
+ * default value, it can be skipped.
+ * @param d A `boolean` default value to compare against when `force_defaults` is `false`.
+ */
public void addBoolean(int o, boolean x, boolean d) { if(force_defaults || x != d) { addBoolean(x); slot(o); } }
+
+ /**
+ * Add a `byte` to a table at `o` into its vtable, with value `x` and default `d`.
+ *
+ * @param o The index into the vtable.
+ * @param x A `byte` to put into the buffer, depending on how defaults are handled. If
+ * `force_defaults` is `false`, compare `x` against the default value `d`. If `x` contains the
+ * default value, it can be skipped.
+ * @param d A `byte` default value to compare against when `force_defaults` is `false`.
+ */
public void addByte (int o, byte x, int d) { if(force_defaults || x != d) { addByte (x); slot(o); } }
+
+ /**
+ * Add a `short` to a table at `o` into its vtable, with value `x` and default `d`.
+ *
+ * @param o The index into the vtable.
+ * @param x A `short` to put into the buffer, depending on how defaults are handled. If
+ * `force_defaults` is `false`, compare `x` against the default value `d`. If `x` contains the
+ * default value, it can be skipped.
+ * @param d A `short` default value to compare against when `force_defaults` is `false`.
+ */
public void addShort (int o, short x, int d) { if(force_defaults || x != d) { addShort (x); slot(o); } }
+
+ /**
+ * Add an `int` to a table at `o` into its vtable, with value `x` and default `d`.
+ *
+ * @param o The index into the vtable.
+ * @param x An `int` to put into the buffer, depending on how defaults are handled. If
+ * `force_defaults` is `false`, compare `x` against the default value `d`. If `x` contains the
+ * default value, it can be skipped.
+ * @param d An `int` default value to compare against when `force_defaults` is `false`.
+ */
public void addInt (int o, int x, int d) { if(force_defaults || x != d) { addInt (x); slot(o); } }
+
+ /**
+ * Add a `long` to a table at `o` into its vtable, with value `x` and default `d`.
+ *
+ * @param o The index into the vtable.
+ * @param x A `long` to put into the buffer, depending on how defaults are handled. If
+ * `force_defaults` is `false`, compare `x` against the default value `d`. If `x` contains the
+ * default value, it can be skipped.
+ * @param d A `long` default value to compare against when `force_defaults` is `false`.
+ */
public void addLong (int o, long x, long d) { if(force_defaults || x != d) { addLong (x); slot(o); } }
+
+ /**
+ * Add a `float` to a table at `o` into its vtable, with value `x` and default `d`.
+ *
+ * @param o The index into the vtable.
+ * @param x A `float` to put into the buffer, depending on how defaults are handled. If
+ * `force_defaults` is `false`, compare `x` against the default value `d`. If `x` contains the
+ * default value, it can be skipped.
+ * @param d A `float` default value to compare against when `force_defaults` is `false`.
+ */
public void addFloat (int o, float x, double d) { if(force_defaults || x != d) { addFloat (x); slot(o); } }
+
+ /**
+ * Add a `double` to a table at `o` into its vtable, with value `x` and default `d`.
+ *
+ * @param o The index into the vtable.
+ * @param x A `double` to put into the buffer, depending on how defaults are handled. If
+ * `force_defaults` is `false`, compare `x` against the default value `d`. If `x` contains the
+ * default value, it can be skipped.
+ * @param d A `double` default value to compare against when `force_defaults` is `false`.
+ */
public void addDouble (int o, double x, double d) { if(force_defaults || x != d) { addDouble (x); slot(o); } }
+
+ /**
+ * Add an `offset` to a table at `o` into its vtable, with value `x` and default `d`.
+ *
+ * @param o The index into the vtable.
+ * @param x An `offset` to put into the buffer, depending on how defaults are handled. If
+ * `force_defaults` is `false`, compare `x` against the default value `d`. If `x` contains the
+ * default value, it can be skipped.
+ * @param d An `offset` default value to compare against when `force_defaults` is `false`.
+ */
public void addOffset (int o, int x, int d) { if(force_defaults || x != d) { addOffset (x); slot(o); } }
- // Structs are stored inline, so nothing additional is being added. `d` is always 0.
+ /**
+ * Add a struct to the table. Structs are stored inline, so nothing additional is being added.
+ *
+ * @param voffset The index into the vtable.
+ * @param x The offset of the created struct.
+ * @param d The default value is always `0`.
+ */
public void addStruct(int voffset, int x, int d) {
if(x != d) {
Nested(x);
@@ -389,7 +575,12 @@ public void addStruct(int voffset, int x, int d) {
}
}
- // Set the current vtable at `voffset` to the current location in the buffer.
+ /**
+ * Set the current vtable at `voffset` to the current location in the buffer.
+ *
+ * @param voffset The index into the vtable to store the offset relative to the end of the
+ * buffer.
+ */
public void slot(int voffset) {
vtable[voffset] = offset();
}
@@ -397,7 +588,7 @@ public void slot(int voffset) {
/**
* Finish off writing the object that is under construction.
*
- * @return The offset to the object inside {@link #dataBuffer()}
+ * @return The offset to the object inside {@link #dataBuffer()}.
* @see #startObject(int)
*/
public int endObject() {
@@ -453,8 +644,13 @@ public int endObject() {
return vtableloc;
}
- // This checks a required field has been set in a given table that has
- // just been constructed.
+ /**
+ * Checks that a required field has been set in a given table that has
+ * just been constructed.
+ *
+ * @param table The offset to the start of the table from the `ByteBuffer` capacity.
+ * @param field The offset to the field in the vtable.
+ */
public void required(int table, int field) {
int table_start = bb.capacity() - table;
int vtable_start = table_start - bb.getInt(table_start);
@@ -463,7 +659,13 @@ public void required(int table, int field) {
if (!ok)
throw new AssertionError("FlatBuffers: field " + field + " must be set");
}
+ /// @endcond
+ /**
+ * Finalize a buffer, pointing to the given `root_table`.
+ *
+ * @param root_table An offset to be added to the buffer.
+ */
public void finish(int root_table) {
prep(minalign, SIZEOF_INT);
addOffset(root_table);
@@ -471,6 +673,13 @@ public void finish(int root_table) {
finished = true;
}
+ /**
+ * Finalize a buffer, pointing to the given `root_table`.
+ *
+ * @param root_table An offset to be added to the buffer.
+ * @param file_identifier A FlatBuffer file identifier to be added to the buffer before
+ * `root_table`.
+ */
public void finish(int root_table, String file_identifier) {
prep(minalign, SIZEOF_INT + FILE_IDENTIFIER_LENGTH);
if (file_identifier.length() != FILE_IDENTIFIER_LENGTH)
@@ -487,17 +696,19 @@ public void finish(int root_table, String file_identifier) {
* don't get serialized into the buffer. Forcing defaults provides a
* way to manually disable this optimization.
*
- * @param forceDefaults true always serializes default values
- * @return this
+ * @param forceDefaults When set to `true`, always serializes default values.
+ * @return Returns `this`.
*/
public FlatBufferBuilder forceDefaults(boolean forceDefaults){
this.force_defaults = forceDefaults;
return this;
}
- // Get the ByteBuffer representing the FlatBuffer. Only call this after you've
- // called finish(). The actual data starts at the ByteBuffer's current position,
- // not necessarily at 0.
+ /**
+ * Get the ByteBuffer representing the FlatBuffer. Only call this after you've
+ * called `finish()`. The actual data starts at the ByteBuffer's current position,
+ * not necessarily at `0`.
+ */
public ByteBuffer dataBuffer() {
finished();
return bb;
@@ -518,13 +729,13 @@ private int dataStart() {
}
/**
- * Utility function for copying a byte array from {@code start} to
- * {@code start} + {@code length}
+ * A utility function to copy and return the ByteBuffer data from `start` to
+ * `start` + `length` as a `byte[]`.
*
- * @param start Start copying at this offset
- * @param length How many bytes to copy
- * @return A range copy of the {@link #dataBuffer() data buffer}
- * @throws IndexOutOfBoundsException If the range of bytes is ouf of bound
+ * @param start Start copying at this offset.
+ * @param length How many bytes to copy.
+ * @return A range copy of the {@link #dataBuffer() data buffer}.
+ * @throws IndexOutOfBoundsException If the range of bytes is ouf of bound.
*/
public byte[] sizedByteArray(int start, int length){
finished();
@@ -535,11 +746,13 @@ public byte[] sizedByteArray(int start, int length){
}
/**
- * Utility function for copying a byte array that starts at 0.
+ * A utility function to copy and return the ByteBuffer data as a `byte[]`.
*
- * @return A full copy of the {@link #dataBuffer() data buffer}
+ * @return A full copy of the {@link #dataBuffer() data buffer}.
*/
public byte[] sizedByteArray() {
return sizedByteArray(space, bb.capacity() - space);
}
}
+
+/// @}
diff --git a/java/com/google/flatbuffers/Struct.java b/java/com/google/flatbuffers/Struct.java
index 9e6fe4a2492af..ae31553144856 100644
--- a/java/com/google/flatbuffers/Struct.java
+++ b/java/com/google/flatbuffers/Struct.java
@@ -18,8 +18,16 @@
import java.nio.ByteBuffer;
-// All structs in the generated code derive from this class, and add their own accessors.
+/// @cond FLATBUFFERS_INTERNAL
+
+/**
+ * All structs in the generated code derive from this class, and add their own accessors.
+ */
public class Struct {
+ /** Used to hold the position of the `bb` buffer. */
protected int bb_pos;
+ /** The underlying ByteBuffer to hold the data of the Struct. */
protected ByteBuffer bb;
}
+
+/// @endcond
diff --git a/java/com/google/flatbuffers/Table.java b/java/com/google/flatbuffers/Table.java
index c4a93756e7c7e..3d10012a8649f 100644
--- a/java/com/google/flatbuffers/Table.java
+++ b/java/com/google/flatbuffers/Table.java
@@ -20,34 +20,61 @@
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
-// All tables in the generated code derive from this class, and add their own accessors.
+/// @cond FLATBUFFERS_INTERNAL
+
+/**
+ * All tables in the generated code derive from this class, and add their own accessors.
+ */
public class Table {
+ /** Used to hold the position of the `bb` buffer. */
protected int bb_pos;
+ /** The underlying ByteBuffer to hold the data of the Table. */
protected ByteBuffer bb;
+ /**
+ * Get the underlying ByteBuffer.
+ *
+ * @return Returns the Table's ByteBuffer.
+ */
public ByteBuffer getByteBuffer() { return bb; }
- // Look up a field in the vtable, return an offset into the object, or 0 if the field is not
- // present.
+ /**
+ * Look up a field in the vtable.
+ *
+ * @param vtable_offset An `int` offset to the vtable in the Table's ByteBuffer.
+ * @return Returns an offset into the object, or `0` if the field is not present.
+ */
protected int __offset(int vtable_offset) {
int vtable = bb_pos - bb.getInt(bb_pos);
return vtable_offset < bb.getShort(vtable) ? bb.getShort(vtable + vtable_offset) : 0;
}
- // Retrieve the relative offset stored at "offset"
+ /**
+ * Retrieve a relative offset.
+ *
+ * @param offset An `int` index into the Table's ByteBuffer containing the relative offset.
+ * @return Returns the relative offset stored at `offset`.
+ */
protected int __indirect(int offset) {
return offset + bb.getInt(offset);
}
- // Create a java String from UTF-8 data stored inside the flatbuffer.
- // This allocates a new string and converts to wide chars upon each access,
- // which is not very efficient. Instead, each FlatBuffer string also comes with an
- // accessor based on __vector_as_bytebuffer below, which is much more efficient,
- // assuming your Java program can handle UTF-8 data directly.
+ /**
+ * Create a Java `String` from UTF-8 data stored inside the FlatBuffer.
+ *
+ * This allocates a new string and converts to wide chars upon each access,
+ * which is not very efficient. Instead, each FlatBuffer string also comes with an
+ * accessor based on __vector_as_bytebuffer below, which is much more efficient,
+ * assuming your Java program can handle UTF-8 data directly.
+ *
+ * @param offset An `int` index into the Table's ByteBuffer.
+ * @return Returns a `String` from the data stored inside the FlatBuffer at `offset`.
+ */
protected String __string(int offset) {
offset += bb.getInt(offset);
if (bb.hasArray()) {
- return new String(bb.array(), bb.arrayOffset() + offset + SIZEOF_INT, bb.getInt(offset), FlatBufferBuilder.utf8charset);
+ return new String(bb.array(), bb.arrayOffset() + offset + SIZEOF_INT, bb.getInt(offset),
+ FlatBufferBuilder.utf8charset);
} else {
// We can't access .array(), since the ByteBuffer is read-only,
// off-heap or a memory map
@@ -60,23 +87,36 @@ protected String __string(int offset) {
}
}
- // Get the length of a vector whose offset is stored at "offset" in this object.
+ /**
+ * Get the length of a vector.
+ *
+ * @param offset An `int` index into the Table's ByteBuffer.
+ * @return Returns the length of the vector whose offset is stored at `offset`.
+ */
protected int __vector_len(int offset) {
offset += bb_pos;
offset += bb.getInt(offset);
return bb.getInt(offset);
}
- // Get the start of data of a vector whose offset is stored at "offset" in this object.
+ /**
+ * Get the start data of a vector.
+ *
+ * @param offset An `int` index into the Table's ByteBuffer.
+ * @return Returns the start of the vector data whose offset is stored at `offset`.
+ */
protected int __vector(int offset) {
offset += bb_pos;
return offset + bb.getInt(offset) + SIZEOF_INT; // data starts after the length
}
- // Get a whole vector as a ByteBuffer. This is efficient, since it only allocates a new
- // bytebuffer object, but does not actually copy the data, it still refers to the same
- // bytes as the original ByteBuffer.
- // Also useful with nested FlatBuffers etc.
+ /**
+ * Get a whole vector as a ByteBuffer.
+ *
+ * This is efficient, since it only allocates a new bytebuffer object, but does not actually copy
+ * the data, it still refers to the same bytes as the original ByteBuffer. Also useful with nested
+ * FlatBuffers, etc.
+ */
protected ByteBuffer __vector_as_bytebuffer(int vector_offset, int elem_size) {
int o = __offset(vector_offset);
if (o == 0) return null;
@@ -87,7 +127,13 @@ protected ByteBuffer __vector_as_bytebuffer(int vector_offset, int elem_size) {
return bb;
}
- // Initialize any Table-derived type to point to the union at the given offset.
+ /**
+ * Initialize any Table-derived type to point to the union at the given `offset`.
+ *
+ * @param t A `Table`-derived type that should point to the union at `offset`.
+ * @param offset An `int` index into the Table's ByteBuffer.
+ * @return Returns the Table that points to the union at `offset`.
+ */
protected Table __union(Table t, int offset) {
offset += bb_pos;
t.bb_pos = offset + bb.getInt(offset);
@@ -95,6 +141,12 @@ protected Table __union(Table t, int offset) {
return t;
}
+ /**
+ * Check if a ByteBuffer contains a file identifier.
+ *
+ * @param bb A `ByteBuffer` to check if it contains the identifier `ident`.
+ * @param ident A `String` identifier of the flatbuffer file.
+ */
protected static boolean __has_identifier(ByteBuffer bb, String ident) {
if (ident.length() != FILE_IDENTIFIER_LENGTH)
throw new AssertionError("FlatBuffers: file identifier must be length " +
@@ -105,3 +157,5 @@ protected static boolean __has_identifier(ByteBuffer bb, String ident) {
return true;
}
}
+
+/// @endcond
diff --git a/js/flatbuffers.js b/js/flatbuffers.js
index efa76d94649c5..f3c483425d078 100644
--- a/js/flatbuffers.js
+++ b/js/flatbuffers.js
@@ -1,3 +1,7 @@
+/// @file
+/// @addtogroup flatbuffers_javascript_api
+/// @{
+/// @cond FLATBUFFERS_INTERNAL
var flatbuffers = {};
/**
@@ -105,9 +109,11 @@ flatbuffers.Long.prototype.equals = function(other) {
*/
flatbuffers.Long.ZERO = new flatbuffers.Long(0, 0);
+/// @endcond
////////////////////////////////////////////////////////////////////////////////
-
/**
+ * Create a FlatBufferBuilder.
+ *
* @constructor
* @param {number=} initial_size
*/
@@ -228,6 +234,7 @@ flatbuffers.Builder.prototype.asUint8Array = function() {
return this.bb.bytes().subarray(this.bb.position(), this.bb.position() + this.offset());
};
+/// @cond FLATBUFFERS_INTERNAL
/**
* Prepare to write an element of `size` after `additional_bytes` have been
* written, e.g. if you write a string, you need to align such the int length
@@ -307,9 +314,11 @@ flatbuffers.Builder.prototype.writeFloat32 = function(value) {
flatbuffers.Builder.prototype.writeFloat64 = function(value) {
this.bb.writeFloat64(this.space -= 8, value);
};
+/// @endcond
/**
- * @param {number} value
+ * Add an `int8` to the buffer, properly aligned, and grows the buffer (if necessary).
+ * @param {number} value The `int8` to add the the buffer.
*/
flatbuffers.Builder.prototype.addInt8 = function(value) {
this.prep(1, 0);
@@ -317,7 +326,8 @@ flatbuffers.Builder.prototype.addInt8 = function(value) {
};
/**
- * @param {number} value
+ * Add an `int16` to the buffer, properly aligned, and grows the buffer (if necessary).
+ * @param {number} value The `int16` to add the the buffer.
*/
flatbuffers.Builder.prototype.addInt16 = function(value) {
this.prep(2, 0);
@@ -325,7 +335,8 @@ flatbuffers.Builder.prototype.addInt16 = function(value) {
};
/**
- * @param {number} value
+ * Add an `int32` to the buffer, properly aligned, and grows the buffer (if necessary).
+ * @param {number} value The `int32` to add the the buffer.
*/
flatbuffers.Builder.prototype.addInt32 = function(value) {
this.prep(4, 0);
@@ -333,7 +344,8 @@ flatbuffers.Builder.prototype.addInt32 = function(value) {
};
/**
- * @param {flatbuffers.Long} value
+ * Add an `int64` to the buffer, properly aligned, and grows the buffer (if necessary).
+ * @param {flatbuffers.Long} value The `int64` to add the the buffer.
*/
flatbuffers.Builder.prototype.addInt64 = function(value) {
this.prep(8, 0);
@@ -341,7 +353,8 @@ flatbuffers.Builder.prototype.addInt64 = function(value) {
};
/**
- * @param {number} value
+ * Add a `float32` to the buffer, properly aligned, and grows the buffer (if necessary).
+ * @param {number} value The `float32` to add the the buffer.
*/
flatbuffers.Builder.prototype.addFloat32 = function(value) {
this.prep(4, 0);
@@ -349,13 +362,15 @@ flatbuffers.Builder.prototype.addFloat32 = function(value) {
};
/**
- * @param {number} value
+ * Add a `float64` to the buffer, properly aligned, and grows the buffer (if necessary).
+ * @param {number} value The `float64` to add the the buffer.
*/
flatbuffers.Builder.prototype.addFloat64 = function(value) {
this.prep(8, 0);
this.writeFloat64(value);
};
+/// @cond FLATBUFFERS_INTERNAL
/**
* @param {number} voffset
* @param {number} value
@@ -515,17 +530,19 @@ flatbuffers.Builder.growByteBuffer = function(bb) {
nbb.bytes().set(bb.bytes(), new_buf_size - old_buf_size);
return nbb;
};
+/// @endcond
/**
* Adds on offset, relative to where it will be written.
*
- * @param {flatbuffers.Offset} offset The offset to add
+ * @param {flatbuffers.Offset} offset The offset to add.
*/
flatbuffers.Builder.prototype.addOffset = function(offset) {
this.prep(flatbuffers.SIZEOF_INT, 0); // Ensure alignment is already done.
this.writeInt32(this.offset() - offset + flatbuffers.SIZEOF_INT);
};
+/// @cond FLATBUFFERS_INTERNAL
/**
* Start encoding a new object in the buffer. Users will not usually need to
* call this directly. The FlatBuffers compiler will generate helper methods
@@ -606,8 +623,11 @@ outer_loop:
this.isNested = false;
return vtableloc;
};
+/// @endcond
/**
+ * Finalize a buffer, poiting to the given `root_table`.
+ *
* @param {flatbuffers.Offset} root_table
* @param {string=} file_identifier
*/
@@ -628,6 +648,7 @@ flatbuffers.Builder.prototype.finish = function(root_table, file_identifier) {
this.bb.setPosition(this.space);
};
+/// @cond FLATBUFFERS_INTERNAL
/**
* This checks a required field has been set in a given table that has
* just been constructed.
@@ -673,6 +694,7 @@ flatbuffers.Builder.prototype.endVector = function() {
this.writeInt32(this.vector_num_elems);
return this.offset();
};
+/// @endcond
/**
* Encode the string `s` in the buffer using UTF-8. If a Uint8Array is passed
@@ -729,10 +751,11 @@ flatbuffers.Builder.prototype.createString = function(s) {
}
return this.endVector();
};
-
////////////////////////////////////////////////////////////////////////////////
-
+/// @cond FLATBUFFERS_INTERNAL
/**
+ * Create a new ByteBuffer with a given array of bytes (`Uint8Array`).
+ *
* @constructor
* @param {Uint8Array} bytes
*/
@@ -751,6 +774,8 @@ flatbuffers.ByteBuffer = function(bytes) {
};
/**
+ * Create and allocate a new ByteBuffer with a given size.
+ *
* @param {number} byte_size
* @returns {flatbuffers.ByteBuffer}
*/
@@ -759,6 +784,8 @@ flatbuffers.ByteBuffer.allocate = function(byte_size) {
};
/**
+ * Get the underlying `Uint8Array`.
+ *
* @returns {Uint8Array}
*/
flatbuffers.ByteBuffer.prototype.bytes = function() {
@@ -766,6 +793,8 @@ flatbuffers.ByteBuffer.prototype.bytes = function() {
};
/**
+ * Get the buffer's position.
+ *
* @returns {number}
*/
flatbuffers.ByteBuffer.prototype.position = function() {
@@ -773,6 +802,8 @@ flatbuffers.ByteBuffer.prototype.position = function() {
};
/**
+ * Set the buffer's position.
+ *
* @param {number} position
*/
flatbuffers.ByteBuffer.prototype.setPosition = function(position) {
@@ -780,6 +811,8 @@ flatbuffers.ByteBuffer.prototype.setPosition = function(position) {
};
/**
+ * Get the buffer's capacity.
+ *
* @returns {number}
*/
flatbuffers.ByteBuffer.prototype.capacity = function() {
@@ -1070,3 +1103,6 @@ flatbuffers.ByteBuffer.prototype.__has_identifier = function(ident) {
// Exports for Node.js and RequireJS
this.flatbuffers = flatbuffers;
+
+/// @endcond
+/// @}
diff --git a/net/FlatBuffers/FlatBufferBuilder.cs b/net/FlatBuffers/FlatBufferBuilder.cs
index 7d21158851e57..c320ea8ccdf7a 100644
--- a/net/FlatBuffers/FlatBufferBuilder.cs
+++ b/net/FlatBuffers/FlatBufferBuilder.cs
@@ -18,12 +18,15 @@
using System;
using System.Text;
+/// @file
+/// @addtogroup flatbuffers_csharp_api
+/// @{
namespace FlatBuffers
{
///