tlv: add library for new message/payload serialization format #3061

cfromknecht · 2019-05-09T00:50:53Z

An implementation of lightning/bolts#607

tlv/stream.go

watchtower/wtwire/wtwire.go

tlv/stream_test.go

tlv/stream.go

tlv/stream_test.go

joostjager · 2019-07-03T12:13:01Z

Nice work in this PR. It shows careful engineering.

tlv/record.go

tlv/stream.go

Roasbeef · 2019-07-10T01:39:17Z

tlv/stream.go

+			panic(ErrNoDecoder.Error())
+		}
+		if record.typ == math.MaxUint64 {
+			overflow = true


If the final type is what overflows, then it appears we won't panic at all?

if the final type is math.MaxUint64 that's okay, so long as there isn't a number after it. overflow in this case signifies that a record hit math.MaxUint64, and any other records should be forbidden

Roasbeef · 2019-07-10T01:39:44Z

tlv/stream.go

+			panic(ErrNoEncoder.Error())
+		}
+		if record.decoder == nil {
+			panic(ErrNoDecoder.Error())


Depending on the context in which this is used within lnd itself, we may actually want to return an error instead.

agreed, i'll change this to use Must paradigm employed by the go std library

hmm, got a better name than tlv.MustStream?

MustNewStream? lol

Either one works IMO. Must can can just wrap the regular and panic instead of panicing.

What about the case where we only decode a stream and don't have an encoder available. Should record.encoder be set to a dummy encoder? (same question for encode-only)

Also I actually rely on the decoder being nil for my TLV-EOB PR within the router. When we receive a fully serialized TLV record stream over the RPC from the user (custom route for SendToRoute) I attach a shim encoder that just writes the raw bytes, and I have no need for a decoder since we'll never decode into the record. As a work around, I guess I can provide one that just does nothing instead since it's a special case.

perhaps instead we could make a nop encoder/decoder and automatically set them if no encoder or decoder is provided

Roasbeef · 2019-07-10T01:43:39Z

tlv/stream.go

+// inspect each record that is parsed and check to see if it has a corresponding
+// Record to facilitate deserialization of that field. If the record is unknown,
+// the Stream will discard the record's bytes and proceed to the subsequent
+// record.


If the record is unknown,
// the Stream will discard the record's bytes and proceed to the subsequent
// record.

Is this subject to the even/odd rule? Also perhaps we should allow a "strict" stream that'll always error our rather than silently go to the next record to be decoded.

yes a strict mode would definitely be useful. the original version of this pr included that, but i removed it for now to simplify the diff

Roasbeef · 2019-07-10T01:45:49Z

tlv/stream.go

+// We permit an io.EOF error only when reading the type byte which signals that
+// the last record was read cleanly and we should stop parsing. All other io.EOF
+// or io.ErrUnexpectedEOF errors are returned.
+func (s *Stream) Decode(r io.Reader) error {


Not a blocker for this PR, but this method would make for a good fuzz testing target. Even before the, we can assert compliance of our encoder by using testing/quick here to ensure we can always ingest what we produce, and don't inadvertently create any invalid streams.

ooo good idea, maybe we can add it to the fuzzing harnesses that @Crypt-iQ has been working on

tlv/record.go

tlv/primitive.go

Roasbeef · 2019-07-31T00:23:18Z

Reviewee bump!

halseth

Awesome work, pretty stoked about finally starting to circle in on a sane serialization format! 🤓

tlv/record.go

tlv/varint.go

tlv/stream.go

halseth · 2019-08-05T13:02:46Z

tlv/primitive.go

+// val is not a **btcec.PublicKey.
+func EPubKey(w io.Writer, val interface{}, _ *[8]byte) error {
+	if pk, ok := val.(**btcec.PublicKey); ok {
+		_, err := w.Write((*pk).SerializeCompressed())


Do we really need methods for PublicKey, or could we just require the caller to use [33]byte?

the test vectors in the spec require us to fail when reading an invalid point, so only parsing [33]byte isn't strict enough. arguably that check could be deferred to a higher level, but it this is the safest approach

tlv/primitive.go

Roasbeef · 2019-08-07T03:10:27Z

tlv/stream.go

+			panic(ErrNoEncoder.Error())
+		}
+		if record.decoder == nil {
+			panic(ErrNoDecoder.Error())


MustNewStream? lol

Either one works IMO. Must can can just wrap the regular and panic instead of panicing.

tlv/tlv_test.go

Roasbeef · 2019-08-07T03:26:03Z

tlv/truncated.go

+// value was not minimally encoded.
+var ErrTUintNotMinimal = errors.New("truncated uint not minimally encoded")
+
+// numLeadingZeroBytes16 computes the number of leading zeros for a uint16.


Are there test vectors in the spec for these truncated variants?

the spec tests the 32 and 64 bit variants, tho not the uint16

joostjager

Only a few non-blocking questions. LGTM

tlv/varint_test.go

joostjager · 2019-08-07T07:24:37Z

tlv/stream.go

+			panic(ErrNoEncoder.Error())
+		}
+		if record.decoder == nil {
+			panic(ErrNoDecoder.Error())


What about the case where we only decode a stream and don't have an encoder available. Should record.encoder be set to a dummy encoder? (same question for encode-only)

joostjager · 2019-08-07T07:41:29Z

tlv/bench_test.go

+
+// BenchmarkEncodeCreateSession benchmarks encoding of the non-TLV
+// CreateSession.
+func BenchmarkEncodeCreateSession(t *testing.B) {


I ran the benchmark and observed the difference between tlv and non-tlv. Pretty impressive. I was wondering, would the gap be much wider if the non-tlv case was optimized more? (mainly thinking about also reusing a buffer)

yes there's a good bit of optimization that could be done to non-tlv messages. amongst other things, this was one of the optimizations included in btcsuite/btcd#1426. i don't think we'd see as significant of gains for lnwire messages since the primary speed up was due to reducing RTTs with the buffer pool, but this certainly helps remove unnecessary allocations and gc pressure

Ok, interesting. Yes, my main thinking was that for a fair comparison, both non-tlv and tlv should be as optimized as possible. But then there is also the "custom tlv encoder/decoder" method, where (unrolled) code is generated for a specific stream. From a theoretical pov it would be nice to see how all three of them compare. What the absolute minimum overhead of tlv is. Definitely not the most important question atm.

agreed, i too would be interested to see that. my goal here was to squeeze some low hanging fruit out of tlv, though as you said the actual overhead might be a little higher if we optimize the traditional encoding. there's also the question of even if those optimizations were possible, would they ever be deployed. if we think tlv will be the base encoding for our messages going forward, we may have to just accept whatever it is :P

tlv/truncated.go

tlv/tlv_test.go

tlv/varint_test.go

Roasbeef

LGTM 💿

This varint has the same serialization as the varint in btcd and bitcoind, but has different behavior wrt returned errors. In order to ensure the inner loop properly detects cleanly written records, ReadVarInt will not only return EOF if it can't read the first byte, as that means the reader has zero bytes left. It also modifies the API to allow the caller to provided a static byte array, which can be reused across all encoding and decoding and increases performance.

This commit adds concrete encoding methods for primitive integral types. When external libs need to create custom encoders, this allows them to do so without incurring an extra allocation on the heap. Previously, the need to pass a pointer to the integer using an interface{} would cause the argument to escape, which we avoid by having them copied directly.

This commit adds the truncated integer encodings used in the variable-size onion payloads. The amount and cltv delta both use the truncated encoding to shave bytes in the overall size, and will likely be used in the future for additional extensions where size is a constraint.

Chinwendu20 · 2023-09-12T17:16:17Z

Do you think it would be a good idea to include a boolean to the primitives?

cfromknecht mentioned this pull request May 9, 2019

BOLT01: TLV proposal lightning/bolts#607

Merged

Roasbeef reviewed May 21, 2019

View reviewed changes

tlv/stream.go Outdated Show resolved Hide resolved

cfromknecht force-pushed the wire-tlv branch from 5e89a58 to f3f19aa Compare May 21, 2019 01:24

cfromknecht force-pushed the wire-tlv branch 5 times, most recently from 6ef4874 to 5fcad7f Compare June 1, 2019 02:05

Roasbeef added P2 should be fixed if one has time spec wire protocol Encoding of messages for the communication between nodes labels Jun 18, 2019

Roasbeef requested a review from joostjager July 2, 2019 02:26

joostjager reviewed Jul 3, 2019

View reviewed changes

cfromknecht force-pushed the wire-tlv branch from 5fcad7f to ebffb22 Compare July 4, 2019 04:37

Roasbeef requested changes Jul 10, 2019

View reviewed changes

cfromknecht force-pushed the wire-tlv branch from ebffb22 to 1a96bdd Compare July 15, 2019 19:28

Roasbeef added the v0.8.0 label Jul 31, 2019

wpaulino added this to the 0.8.0 milestone Jul 31, 2019

Roasbeef mentioned this pull request Jul 31, 2019

multi: update the switch+router to be aware of the new TLV EOB format #3362

Merged

3 tasks

halseth reviewed Aug 5, 2019

View reviewed changes

cfromknecht force-pushed the wire-tlv branch 2 times, most recently from 8df5fa4 to 2058189 Compare August 7, 2019 02:04

cfromknecht requested review from Roasbeef, halseth and joostjager August 7, 2019 02:15

cfromknecht changed the title ~~[tlv]: wire tlv proposal~~ tlv: add library for new message/payload serialization format Aug 7, 2019

cfromknecht force-pushed the wire-tlv branch from 2058189 to 2fedd1e Compare August 7, 2019 02:22

Roasbeef reviewed Aug 7, 2019

View reviewed changes

joostjager reviewed Aug 7, 2019

View reviewed changes

cfromknecht force-pushed the wire-tlv branch from 2fedd1e to fa4e1a6 Compare August 7, 2019 21:43

Roasbeef approved these changes Aug 7, 2019

View reviewed changes

cfromknecht added 9 commits August 7, 2019 15:03

tlv/varint_test: add tests vectors for custom Read/WriteVarInt

75fcf1c

tlv/primitive: add primitive encodings

6773d47

tlv/record: adds various tlv record constructors

96e0bb1

tlv/stream: adds tlv stream encoding/decoding

bc1f23d

tlv/bench_test: add basic benchmark

abdcd47

tlv/tlv_test: add BOLT1 test vectors

3690999

cfromknecht force-pushed the wire-tlv branch from aa2f1d9 to 3690999 Compare August 7, 2019 22:05

Roasbeef merged commit ea77ff9 into lightningnetwork:master Aug 7, 2019

cfromknecht deleted the wire-tlv branch August 7, 2019 22:51

cfromknecht added tlv P2 should be fixed if one has time and removed P2 should be fixed if one has time labels Aug 8, 2019

Chinwendu20 mentioned this pull request Oct 4, 2023

tlv: Added bool to primitive #8057

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tlv: add library for new message/payload serialization format #3061

tlv: add library for new message/payload serialization format #3061

cfromknecht commented May 9, 2019

joostjager commented Jul 3, 2019

Roasbeef Jul 10, 2019

cfromknecht Aug 7, 2019

Roasbeef Jul 10, 2019

cfromknecht Aug 7, 2019

cfromknecht Aug 7, 2019

Roasbeef Aug 7, 2019

joostjager Aug 7, 2019

Roasbeef Aug 7, 2019

cfromknecht Aug 7, 2019

Roasbeef Jul 10, 2019

cfromknecht Aug 7, 2019

Roasbeef Jul 10, 2019

cfromknecht Aug 7, 2019

Roasbeef commented Jul 31, 2019

halseth left a comment

halseth Aug 5, 2019

cfromknecht Aug 7, 2019

Roasbeef Aug 7, 2019

Roasbeef Aug 7, 2019

cfromknecht Aug 7, 2019

joostjager left a comment

joostjager Aug 7, 2019

joostjager Aug 7, 2019

cfromknecht Aug 7, 2019

joostjager Aug 8, 2019

cfromknecht Aug 8, 2019

Roasbeef left a comment

Chinwendu20 commented Sep 12, 2023

tlv: add library for new message/payload serialization format #3061

tlv: add library for new message/payload serialization format #3061

Conversation

cfromknecht commented May 9, 2019

joostjager commented Jul 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef commented Jul 31, 2019

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef left a comment

Choose a reason for hiding this comment

Chinwendu20 commented Sep 12, 2023