TL-B serialization in Rust

[mitinarseny]

Summary

Implement a Rust library for effective and convenient TL-B (Typed Language Binary) serialization and deserialization.

Context

Why it's Important?

Implementing TL-B serialization in Rust is imperative to offer wider language support allowing more programmers to contribute to TON.

Problem showcase:

From TL-B Language docs:

TL-B (Type Language - Binary) serves to describe the type system, constructors and existing functions. For example, we can use TL-B schemes to build binary structures associated with TON Blockchain. Special TL-B parsers can read schemes to deserialize binary data into different objects.

At the moment, the only existing Rust library that has very basic support for TL-B serialization and deserialization is tonlib-rs.
While it implements basic functionality for storing uints/strings/addresses into Cell and reading these exact types back from it, but it becomes hard to serialize complex nested structures (e.g. DeDust nested Swaps). The library does not define unified traits for serialization/deserialization, so you can't compose types with each other, while using powerful features of Rust type system. This might discourage Rust developers from contributing to the project.

Potential Solution:

Here is my initial implementation of a Rust library that supports TL-B serialization and deserialization: https://github.com/mitinarseny/tlb.

It already supports effective TL-B (de)serialization for primitive types and composing them into larger ones, following best practices and leveraging features of Rust type system.

The library has similar design to serde and serde_with traits. One might ask: why to define your own (de)serialization traits and not to use existing serde ecosystem and implement a new data format for TL-B? That's a reasonable question to ask and I tried this way in the beginning. But, unfortunately, per-bit serialization and references to Cells cannot be expressed with serde's data model.

Things needed to be done:

• Documentation
• Cell serialization into bytes needs to be optimized and to reuse existing BitSerialize trait from this crate.
• More testing
• Publish crate to crates.io registry
• Open a PR for integration with tonlib-rs

Future improvements:

• syn-powered TL-B schema parsing
• Code-gen for (de)serialization by using #[derive(TLBSerialize)]

References

• tlb – this library
• TL-B Language
• tonlib-rs

Estimate suggested reward

$5K

[ProgramCrafter]

I'd like to add my initial implementation: https://github.com/ProgramCrafter/tlb-rust-serialization/blob/master/src/main.rs.
Right now it only collects a list of stores to be done into a cell, but that can be modified easily.

#[tlb_enum_serializable]
#[tlb_assert_unsafe(items_prefixes_nonoverlap)]
enum CommonMsgInfo {
    #[tlb_item_serializable(u 0 1bit, ihr_disabled, bounce, bounced, src, dest,
                            value, ihr_fee, fwd_fee, created_lt, created_at)]
    int_msg_info {
        ihr_disabled: bool,
        bounce: bool,
        bounced: bool,
        src: ton::Address,
        dest: ton::Address,
        value: ton::CurrencyCollection,
        ihr_fee: ton::Coins,
        fwd_fee: ton::Coins,
        created_lt: u64,
        created_at: u32
    }
}

...
println!("{:?}", CommonMsgInfo::default().serialize());
// ["u 0 1bit", "u 1 1bit", "u 1 1bit", "u 0 1bit",
//  "u 4 3bit", "u 0 8bit", "u 0 128bit", "u 0 128bit", "u 4 3bit", "u 0 8bit", "u 0 128bit", "u 0 128bit",
//  "u 0 4bit", "u 0 0bit", "u 0 1bit", "u 0 4bit", "u 0 0bit", "u 0 4bit", "u 0 0bit",
//  "u 10001 64bit", "u 0 32bit"]

[hacker-volodya]

At the moment, the only existing Rust library that has very basic support for TL-B serialization and deserialization is tonlib-rs

There is also broxus/ton-labs-block which already has full implementation of block.tlb.

[mitinarseny]

There is also broxus/ton-labs-block which already has full implementation of block.tlb.

Good catch, thanks! Will take a closer look at it

[delovoyhomie]

Thank you for your initiative, but it seems that there are already tools of this kind available now.

[mitinarseny]

I'm happy to announce that tlb toner finally found its usage in ton-grpc!
toner is going to be used to implement balancing over shards and to safely implement deserialization of complex nested structs, such as:

_ (HashmapE 32 ^(BinTree ShardDescr)) = ShardHashes;

@akostylev0 Please, correct me if I'm wrong here.

I'm going to continue working on toner and will add a proper documentation soon.
If anyone also wants to use it, check out the docs. Please, feel free to reach out in case you have any questions or suggestions.

Hope it helps! ❤️

[delovoyhomie]

LGTM!

[delovoyhomie]

@mitinarseny thank you for the contribution!

To accurately recognize your valuable contributions in our repository, we kindly request you to submit a Pull Request to the Hall of Fame file, providing the wallet address and a link to the bounty with the number.

Please follow these steps:

1. Fork the repository (if you haven't already).

2. Edit the Hall of Fame file, commit, and push your changes.

3. Create a Pull Request from your fork to the main repository, providing the wallet address and a link to the bounty with the number (for example, Pull Request Article: Generation of block random seed #136).
   For reference on what your entry should look like, please see the examples of past merged pull requests.

4. And please follow the questbook proposal stage in accordance with the bounty guideline