Update MaxEncodedLen derive macro #512

pgherveou · 2023-09-06T21:12:12Z

#508 fixed the generated code for MaxEncoded derive trait

#[codec(compact)]
foo: u64

// compile to
// old:  u64::max_encoded_len()
// new: <::codec::Compact<u64> as ::codec::MaxEncodedLen>::max_encoded_len()

This is all fine but cause some compilation errors when this field is generic, since Rust can not tell that Compact<T> implements MaxEncodedLen when T implements it, without extra constraints

#[codec(compact)]
deposit: BalanceOf<T>,

// 53 |         pub deposit: BalanceOf<T, OldCurrency>,
//    |                      ^^^^^^^^^ the trait `parity_scale_codec::MaxEncodedLen` is not implemented for `parity_scale_codec::Compact<<OldCurrency as Currency<<T as frame_system::Config>::AccountId>>::Balance>`

src/max_encoded_len.rs

koute

Could you add a test which would not compile without this change?

koute · 2023-09-07T08:48:46Z

src/max_encoded_len.rs

+	fn max_compact_encoded_len() -> usize {
+		Self::max_encoded_len()
+	}


Hmm... personally I'd probably prefer to keep this as a #[no_doc] and make it an internal implementation detail, instead of making it public.

yes good point

by internal you mean simply adding the #[doc(hidden)] attribute or is there other things you can do to truly keep it internal?

by internal you mean simply adding the #[doc(hidden)] attribute or is there other things you can do to truly keep it internal?

Just hide it. I guess you could also prefix its name, e.g. prepend scale_internal_ to the name (which I did when I was adding scale_internal_decode_bytes on Input)

I don't get why we need this at all.

The first thing should also be that there is a test added that fails with the old derive macro.

koute · 2023-09-07T09:01:22Z

Also, a side note only tangentially related to this PR - when reviewing this code I've noticed that a compact u16's maximum encoded length is apparently 4 bytes!? That is strange, because for all of the other types it only takes at most one extra byte over the size of the type itself, and IIRC the bare minimum to represent a u16 in a space-efficient varint encoding should be 3 bytes (which can actually encode a number up to 21 bits, [edit]although, okay, this assumes that the encodings for different types are compatible and that the maximum size is 64-bit; with different constraints the maximum number of bits is going to be different[/edit]).

But apparently we kinda screwed this up, and that's how we encode u16s:

assert_eq!(Compact(u16::MAX).encode().len(), 4); // Doesn't panic!

Kinda unfortunate that we waste space here.

bkchr · 2023-09-07T09:29:56Z

src/max_encoded_len.rs

 impl_primitives!(
+	i8, i16, i32, i64, i128, bool,


You know that you skipped all unsigned types?

For the unsigned types the impl_compact macro is used now, so they're not skipped. (:

bkchr · 2023-09-07T09:30:21Z

src/max_encoded_len.rs

+	fn max_compact_encoded_len() -> usize {
+		Self::max_encoded_len()
+	}


I don't get why we need this at all.

The first thing should also be that there is a test added that fails with the old derive macro.

koute · 2023-09-07T09:53:16Z

tests/max_encoded_len.rs

-	);
+	let max_len = CompactField::<u64>::max_encoded_len();
+	assert_eq!(max_len, Compact::<u64>::max_encoded_len() + u64::max_encoded_len());
+	assert_eq!(max_len, 17);
 }


I think I'd prefer to have a separate test for this, instead of modifying an existing one.

done, this test would fail with

Compiling parity-scale-codec v3.6.6 (/Users/pg/github/parity-scale-codec) error[E0277]: the trait bound `Compact<T>: MaxEncodedLen` is not satisfied --> tests/max_encoded_len.rs:87:5 | 87 | t: T, | ^ the trait `MaxEncodedLen` is not implemented for `Compact<T>` | help: consider extending the `where` clause, but there might be an alternative better way to express this requirement | 84 | #[derive(Encode, MaxEncodedLen, Compact<T>: MaxEncodedLen)] | +++++++++++++++++++++++++++

```rust impl<T> ::parity_scale_codec::MaxEncodedLen for Generic<T> where T: ::parity_scale_codec::MaxEncodedLen, ``` instead of ``` impl<T> ::parity_scale_codec::MaxEncodedLen for Generic<T> where T: ::parity_scale_codec::MaxEncodedLen, T: ::parity_scale_codec::MaxEncodedLen, T: ::parity_scale_codec::MaxEncodedLen, ```

derive/src/trait_bounds.rs

ggwpez

Wait, why is this adding a new function max_compact_encoded_len instead of fixing the impl of the max_encoded_len for the Compact wrapper?

pgherveou · 2023-09-07T11:00:30Z

Wait, why is this adding a new function max_compact_encoded_len instead of fixing the impl of the max_encoded_len for the Compact wrapper?

the impl that we fixed previously #508 is not wrong, but it forces us to add extra constraint on the type when used with generics.

See #512 (comment)

If you have a better solution, I'll take it.

bkchr · 2023-09-07T15:55:24Z

Pushed the proper solution. We add a new required trait for HasCompact::Type. But I doubt that someone implements this trait downstream.

We need to seal this trait.

src/compact.rs

pgherveou · 2023-09-08T07:38:05Z

lgtm too, checking if everything compile properly on polkadot-sdk and will merge this next and create a new release

This reverts commit 1e3f80c.

* Revert "Update MaxEncodedLen derive macro (#512)" This reverts commit 1e3f80c. * Revert "Fix max_encoded_len for Compact fields (#508)" This reverts commit ddf9439 * Bump version to 3.6.9 * Cargo.lock * ui test

pgherveou requested a review from bkchr September 6, 2023 21:32

pgherveou commented Sep 6, 2023

View reviewed changes

src/max_encoded_len.rs Outdated Show resolved Hide resolved

pgherveou force-pushed the pg/max-encoded-len-2 branch from 9891cf3 to 58ac1de Compare September 7, 2023 08:03

pgherveou marked this pull request as ready for review September 7, 2023 08:07

koute reviewed Sep 7, 2023

View reviewed changes

bkchr requested changes Sep 7, 2023

View reviewed changes

pgherveou added 3 commits September 7, 2023 11:45

Add failing test

e3336d1

add MaxCompactEncodedLen trait

0af66c6

Add max_compact_encoded_len to MaxEncodedLen trait

49d2bab

pgherveou force-pushed the pg/max-encoded-len-2 branch from 58ac1de to 49d2bab Compare September 7, 2023 09:46

koute reviewed Sep 7, 2023

View reviewed changes

pgherveou added 2 commits September 7, 2023 12:08

undo

8818af7

pgherveou commented Sep 7, 2023

View reviewed changes

derive/src/trait_bounds.rs Outdated Show resolved Hide resolved

pgherveou and others added 3 commits September 7, 2023 12:45

Update derive/src/trait_bounds.rs

561960c

fix tests

4b72be5

add nodoc

f14dcda

ggwpez reviewed Sep 7, 2023

View reviewed changes

pgherveou requested review from bkchr and koute September 7, 2023 10:58

pgherveou requested a review from ggwpez September 7, 2023 11:01

Rework

d652607

bkchr and others added 3 commits September 7, 2023 18:25

Fix

04f935a

fix fix

f96b221

Revert

e8b44f9

bkchr approved these changes Sep 7, 2023

View reviewed changes

pgherveou commented Sep 7, 2023

View reviewed changes

src/compact.rs Show resolved Hide resolved

pgherveou changed the title ~~Add MaxCompactEncodedLen~~ Update MaxEncodedLen derive macro Sep 8, 2023

pgherveou merged commit 1e3f80c into master Sep 8, 2023

prestwich mentioned this pull request Sep 11, 2023

Backwards incompatible change in patch version #520

Open

pgherveou mentioned this pull request Sep 12, 2023

fix scale support for upcoming versions recmo/uint#313

Open

3 tasks

ggwpez deleted the pg/max-encoded-len-2 branch October 27, 2023 19:29

ascjones added a commit that referenced this pull request Nov 29, 2023

Revert "Update MaxEncodedLen derive macro (#512)"

5bbda86

This reverts commit 1e3f80c.

ascjones mentioned this pull request Nov 29, 2023

Revert breaking changes, bump version to 3.6.9 #545

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update MaxEncodedLen derive macro #512

Update MaxEncodedLen derive macro #512

pgherveou commented Sep 6, 2023 •

edited

Loading

koute left a comment

koute Sep 7, 2023

pgherveou Sep 7, 2023

pgherveou Sep 7, 2023

koute Sep 7, 2023

bkchr Sep 7, 2023

koute commented Sep 7, 2023 •

edited

Loading

bkchr Sep 7, 2023

koute Sep 7, 2023

bkchr Sep 7, 2023

koute Sep 7, 2023

bkchr Sep 7, 2023

pgherveou Sep 7, 2023

ggwpez left a comment

pgherveou commented Sep 7, 2023

bkchr commented Sep 7, 2023

pgherveou commented Sep 8, 2023

Update MaxEncodedLen derive macro #512

Update MaxEncodedLen derive macro #512

Conversation

pgherveou commented Sep 6, 2023 • edited Loading

koute left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koute commented Sep 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggwpez left a comment

Choose a reason for hiding this comment

pgherveou commented Sep 7, 2023

bkchr commented Sep 7, 2023

pgherveou commented Sep 8, 2023

pgherveou commented Sep 6, 2023 •

edited

Loading

koute commented Sep 7, 2023 •

edited

Loading