msgpack: initial support for friendly binary serialization format (#374)

* Regenerate pedersen lookup tables if they're empty * re-init generator tables if they're empty. * feat(nullifier_tree): make empty nullifier tree leaves hash be 0 (#360) * feat(nullifier_tree): make empty nullifier tree leaves be 0 * fix: add append zero behaviour and test * fix: explicit type name * clean: update class semantics --------- Co-authored-by: cheethas <[email protected]> Co-authored-by: cheethas <[email protected]> * More generators for aztec3. * update js vk (because we now use UP for merkle hashing) * Helpers for ECDSA in A3 (#364) * Add `stdlib_keccak` in cmake. Correct an assertion in `to_byte_array` in bigfield. * Add `random_element` to affine element. * negate y conditionally. * feat(nullifier_tree): make empty nullifier tree leaves hash be 0 (#360) * feat(nullifier_tree): make empty nullifier tree leaves be 0 * fix: add append zero behaviour and test * fix: explicit type name * clean: update class semantics --------- Co-authored-by: cheethas <[email protected]> Co-authored-by: cheethas <[email protected]> * Change pedersen hash c_bind to use `pedersen_hash::lookup`. * feat: add msgpack-c submodule * Give up on msgpack c_master * Working hacky msgpack test * Interim work * Interim work * Getting rid of memory hacks * fix: memory leaks * Start of demoing cbinds * Align with other methods * chore: Remove need to return from msgpack method * Iterate example * fix: Hack around generator issues * feat: iterate on msgpack in bb * fix: fork msgpack for greater checks * Refactor * cleanup * Update turbo_circuit_constructor.cpp * chore: continued cleanup * chore: continued cleanup * chore: continued cleanup * Refactor * Refactor * fix: ci * feat(wasm): hacks to make work in a fno-exceptions wasm environment * feat(wasm): bump msgpack-c * feat(msgpack): first 'complex' object bound * More wasm fixes. Was breaking throw() declaration * Fix field serialization * refactoring * Update CMakeLists.txt * Remove // TODO redundant with msgpack * Refactor to use macro * Refactor to use macro * fix printing bug * fix: fieldd msgpack endianness fix * fix: remove shared ptr reference * doc * Add static checking for MSGPACK usage * Revert log.hpp change * Update struct_map_impl.hpp * Revert * remote_build fix * Keep trying to init submodules * Keep trying to init submodules * Bump * Add missing init_submodules * Msgpack test fix * Msgpack test fix * Msgpack test fix * Msgpack test fix * Update polynomial_store.test.cpp * Merge master * Update msgpack error * Better abort distinguishing * fix: join split VK hash * Serialization updates * Fix circuits build * Try to make circuits test work again * Try to make circuits test work again * Try to make circuits test work again * fix: initialization warning * fix: prefer default constructor for field, related cleanup * Grand rename * chore: remove unused funcs * Revert fields constructor change for now * chore: Revert .circleci changes * chore: Revert foundation removal * Revert .gitmodules * Update affine_element.hpp * Update element.hpp * Revert header optimizations * Revert init line * Update polynomial_store.test.cpp * Revert header optimization * Update raw_pointer.hpp * Update raw_pointer.hpp * Update func_traits.hpp documentation * Document msgpack methods in field_impl.hpp * Update msgpack.hpp * Update cbind.hpp * Update msgpack.hpp * Update msgpack.hpp * Update schema_impl.hpp * Update g1.hpp --------- Co-authored-by: Suyash Bagad <[email protected]> Co-authored-by: Maddiaa <[email protected]> Co-authored-by: cheethas <[email protected]> Co-authored-by: cheethas <[email protected]> Co-authored-by: Suyash Bagad <[email protected]>
AztecProtocol · May 17, 2023 · 9ebfd3a · 9ebfd3a
1 parent 586bdfc
commit 9ebfd3a
Show file tree

Hide file tree

Showing 49 changed files with 1,108 additions and 43 deletions.
diff --git a/barretenberg/.gitmodules b/barretenberg/.gitmodules
@@ -1,3 +1,6 @@
+[submodule "cpp/src/msgpack-c"]
+	path = cpp/src/msgpack-c
+	url = https://github.com/AztecProtocol/msgpack-c
 [submodule "foundation"]
 	path = foundation
 	url = [email protected]:AztecProtocol/foundation.git

diff --git a/barretenberg/cpp/src/CMakeLists.txt b/barretenberg/cpp/src/CMakeLists.txt
@@ -27,7 +27,7 @@ if(CMAKE_CXX_COMPILER_ID MATCHES "GNU")
     add_compile_options(-fconstexpr-ops-limit=100000000)
 endif()
 
-include_directories(${CMAKE_CURRENT_SOURCE_DIR})
+include_directories(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/msgpack-c/include)
 
 # I feel this should be limited to ecc, however it's currently used in headers that go across libraries,
 # and there currently isn't an easy way to inherit the DDISABLE_SHENANIGANS parameter.
@@ -52,6 +52,7 @@ add_subdirectory(barretenberg/plonk)
 add_subdirectory(barretenberg/stdlib)
 add_subdirectory(barretenberg/join_split_example)
 add_subdirectory(barretenberg/dsl)
+add_subdirectory(barretenberg/serialize)
 add_subdirectory(barretenberg/solidity_helpers)
 
 if(BENCHMARKS)

diff --git a/barretenberg/cpp/src/barretenberg/common/log.hpp b/barretenberg/cpp/src/barretenberg/common/log.hpp
@@ -18,23 +18,10 @@
 
 namespace {
 
-inline void format_chain(std::ostream&) {}
-
-template <typename T> void format_chain(std::ostream& os, T const& first)
-{
-    os << first;
-}
-
-template <typename T, typename... Args> void format_chain(std::ostream& os, T const& first, Args const&... args)
-{
-    os << first;
-    format_chain(os, args...);
-}
-
 template <typename... Args> std::string format(Args... args)
 {
     std::ostringstream os;
-    format_chain(os, args...);
+    ((os << args), ...);
     return os.str();
 }
 

diff --git a/barretenberg/cpp/src/barretenberg/common/serialize.hpp b/barretenberg/cpp/src/barretenberg/common/serialize.hpp
@@ -1,5 +1,6 @@
 /**
- * This is the core serialization library.
+ * This is a non-msgpack flat buffer serialization library.
+ * It is currently used alongside msgpack, with hope to eventually move to msgpack.
  * It enables the reading and writing of big-endian formatted integers and various standard library types
  * to and from the following supported types:
  *  - uint8_t*
@@ -40,6 +41,7 @@
 #ifndef __i386__
 __extension__ using uint128_t = unsigned __int128;
 #endif
+
 namespace serialize {
 // Basic integer read / write, to / from raw buffers.
 // Pointers to buffers are advanced by length of type.

diff --git a/barretenberg/cpp/src/barretenberg/common/throw_or_abort.hpp b/barretenberg/cpp/src/barretenberg/common/throw_or_abort.hpp
@@ -7,7 +7,7 @@ inline void throw_or_abort [[noreturn]] (std::string const& err)
 #ifndef __wasm__
     throw std::runtime_error(err);
 #else
-    info(err);
+    info("abort: ", err);
     std::abort();
 #endif
-}
+}
diff --git a/barretenberg/cpp/src/barretenberg/crypto/ecdsa/ecdsa.hpp b/barretenberg/cpp/src/barretenberg/crypto/ecdsa/ecdsa.hpp
@@ -3,6 +3,7 @@
 #include <array>
 #include <string>
 #include "barretenberg/ecc/curves/secp256k1/secp256k1.hpp"
+#include "barretenberg/serialize/msgpack.hpp"
 
 namespace crypto {
 namespace ecdsa {
@@ -15,6 +16,8 @@ struct signature {
     std::array<uint8_t, 32> r;
     std::array<uint8_t, 32> s;
     uint8_t v;
+    // for serialization, update with any new fields
+    MSGPACK_FIELDS(r, s, v);
 };
 
 template <typename Hash, typename Fq, typename Fr, typename G1>
@@ -70,4 +73,4 @@ template <typename B> inline void write(B& buf, key_pair<secp256k1::fr, secp256k
 } // namespace ecdsa
 } // namespace crypto
 
-#include "./ecdsa_impl.hpp"
+#include "./ecdsa_impl.hpp"
diff --git a/barretenberg/cpp/src/barretenberg/crypto/ecdsa/ecdsa.test.cpp b/barretenberg/cpp/src/barretenberg/crypto/ecdsa/ecdsa.test.cpp
@@ -2,10 +2,17 @@
 #include "barretenberg/ecc/curves/grumpkin/grumpkin.hpp"
 #include "barretenberg/ecc/curves/secp256r1/secp256r1.hpp"
 #include "barretenberg/common/serialize.hpp"
+#include "barretenberg/serialize/test_helper.hpp"
 #include <gtest/gtest.h>
 
 using namespace barretenberg;
 
+TEST(ecdsa, msgpack)
+{
+    auto [actual, expected] = msgpack_roundtrip(crypto::ecdsa::signature{});
+    EXPECT_EQ(actual, expected);
+}
+
 TEST(ecdsa, verify_signature_grumpkin_sha256)
 {
     std::string message = "The quick brown dog jumped over the lazy fox.";

diff --git a/barretenberg/cpp/src/barretenberg/crypto/pedersen_commitment/c_bind.cpp b/barretenberg/cpp/src/barretenberg/crypto/pedersen_commitment/c_bind.cpp
@@ -30,7 +30,6 @@ WASM_EXPORT void pedersen_plookup_compress_fields(uint8_t const* left, uint8_t c
     barretenberg::fr::serialize_to_buffer(r, result);
 }
 
-
 WASM_EXPORT void pedersen__compress(uint8_t const* inputs_buffer, uint8_t* output)
 {
     std::vector<grumpkin::fq> to_compress;

diff --git a/barretenberg/cpp/src/barretenberg/dsl/acir_format/acir_format.test.cpp b/barretenberg/cpp/src/barretenberg/dsl/acir_format/acir_format.test.cpp
@@ -3,8 +3,13 @@
 #include <gtest/gtest.h>
 #include <vector>
 #include "barretenberg/common/streams.hpp"
+#include "barretenberg/serialize/test_helper.hpp"
 #include "ecdsa_secp256k1.hpp"
-
+TEST(acir_format, msgpack_logic_constraint)
+{
+    auto [actual, expected] = msgpack_roundtrip(acir_format::LogicConstraint {});
+    EXPECT_EQ(actual, expected);
+}
 TEST(acir_format, test_logic_gate_from_noir_circuit)
 {
     /**

diff --git a/barretenberg/cpp/src/barretenberg/dsl/acir_format/logic_constraint.hpp b/barretenberg/cpp/src/barretenberg/dsl/acir_format/logic_constraint.hpp
@@ -12,6 +12,9 @@ struct LogicConstraint {
     uint32_t is_xor_gate;
 
     friend bool operator==(LogicConstraint const& lhs, LogicConstraint const& rhs) = default;
+
+    // for serialization, update with any new fields
+    MSGPACK_FIELDS(a, b, result, num_bits, is_xor_gate);
 };
 
 void create_logic_gate(Composer& composer, uint32_t a, uint32_t b, uint32_t result, size_t num_bits, bool is_xor_gate);

diff --git a/barretenberg/cpp/src/barretenberg/dsl/acir_format/sha256_constraint.hpp b/barretenberg/cpp/src/barretenberg/dsl/acir_format/sha256_constraint.hpp
@@ -10,13 +10,17 @@ struct Sha256Input {
     uint32_t num_bits;
 
     friend bool operator==(Sha256Input const& lhs, Sha256Input const& rhs) = default;
+    // for serialization, update with any new fields
+    MSGPACK_FIELDS(witness, num_bits);
 };
 
 struct Sha256Constraint {
     std::vector<Sha256Input> inputs;
     std::vector<uint32_t> result;
 
     friend bool operator==(Sha256Constraint const& lhs, Sha256Constraint const& rhs) = default;
+    // for serialization, update with any new fields
+    MSGPACK_FIELDS(inputs, result);
 };
 
 // This function does not work (properly) because the stdlib:sha256 function is not working correctly for 512 bits

diff --git a/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/fq.hpp b/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/fq.hpp
@@ -60,4 +60,10 @@ class Bn254FqParams {
 
 typedef field<Bn254FqParams> fq;
 
-} // namespace barretenberg
+} // namespace barretenberg
+
+// define this as a named alias in msgpack schema generation
+inline void msgpack_schema_pack(auto& packer, barretenberg::fq const&)
+{
+    packer.pack_alias("Fq", "bin32");
+}
diff --git a/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/fq.test.cpp b/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/fq.test.cpp
@@ -1,9 +1,16 @@
 #include "fq.hpp"
 #include "pseudorandom.hpp"
+#include "barretenberg/serialize/test_helper.hpp"
 #include <gtest/gtest.h>
 
 using namespace barretenberg;
 
+TEST(fq, msgpack)
+{
+    auto [actual, expected] = msgpack_roundtrip(barretenberg::fq{1ull, 2ull, 3ull, 4ull});
+    EXPECT_EQ(actual, expected);
+}
+
 TEST(fq, eq)
 {
     constexpr fq a{ 0x01, 0x02, 0x03, 0x04 };

diff --git a/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/fr.hpp b/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/fr.hpp
@@ -65,4 +65,10 @@ class Bn254FrParams {
 
 typedef field<Bn254FrParams> fr;
 
-} // namespace barretenberg
+} // namespace barretenberg
+
+// define this as a named alias in msgpack schema generation
+inline void msgpack_schema_pack(auto& packer, barretenberg::fr const&)
+{
+    packer.pack_alias("Fr", "bin32");
+}
diff --git a/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/fr.test.cpp b/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/fr.test.cpp
@@ -1,8 +1,15 @@
 #include "fr.hpp"
+#include "barretenberg/serialize/test_helper.hpp"
 #include <gtest/gtest.h>
 
 using namespace barretenberg;
 
+TEST(fr, msgpack)
+{
+    auto [actual, expected] = msgpack_roundtrip(barretenberg::fr{1ull, 2ull, 3ull, 4ull});
+    EXPECT_EQ(actual, expected);
+}
+
 TEST(fr, eq)
 {
     fr a{ 0x01, 0x02, 0x03, 0x04 };

diff --git a/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/g1.hpp b/barretenberg/cpp/src/barretenberg/ecc/curves/bn254/g1.hpp
@@ -17,4 +17,12 @@ struct Bn254G1Params {
 };
 
 typedef group<fq, fr, Bn254G1Params> g1;
-} // namespace barretenberg
+
+} // namespace barretenberg
+
+// specialize the name in msgpack schema generation
+// consumed by the typescript schema compiler, helps disambiguate templates
+inline std::string msgpack_schema_name(barretenberg::g1::affine_element const&)
+{
+    return "G1AffineElement";
+}
diff --git a/barretenberg/cpp/src/barretenberg/ecc/fields/field.hpp b/barretenberg/cpp/src/barretenberg/ecc/fields/field.hpp
@@ -406,6 +406,10 @@ template <class Params> struct alignas(32) field {
 
     // BBERG_INLINE sstatic constexpr void butterfly(field& left, field& right) noexcept;
 
+    // For serialization
+    void msgpack_pack(auto& packer) const;
+    void msgpack_unpack(auto o);
+
   private:
     static constexpr uint256_t twice_modulus = modulus + modulus;
     static constexpr uint256_t not_modulus = -modulus;
@@ -543,7 +547,6 @@ template <typename B, typename Params> void read(B& it, field<Params>& value)
     read(it, result.data[0]);
     value = result.to_montgomery_form();
 }
-
 template <typename B, typename Params> void write(B& buf, field<Params> const& value)
 {
     using serialize::write;

diff --git a/barretenberg/cpp/src/barretenberg/ecc/fields/field_impl.hpp b/barretenberg/cpp/src/barretenberg/ecc/fields/field_impl.hpp
@@ -12,7 +12,6 @@
 #endif
 
 #include "field_impl_generic.hpp"
-
 namespace barretenberg {
 
 // template <class T> constexpr void field<T>::butterfly(field& left, field& right) noexcept
@@ -607,4 +606,45 @@ template <class T> constexpr field<T> field<T>::multiplicative_generator() noexc
     return target;
 }
 
+// This function is used to serialize a field. It matches the old serialization format by first 
+// converting the field from Montgomery form, which is a special representation used for efficient 
+// modular arithmetic.
+template <class Params> void field<Params>::msgpack_pack(auto& packer) const
+{
+    // The field is first converted from Montgomery form, similar to how the old format did it.
+    auto adjusted = from_montgomery_form();
+
+    // The data is then converted to big endian format using htonll, which stands for "host to network long long". 
+    // This is necessary because the data will be written to a raw msgpack buffer, which requires big endian format. 
+    uint64_t bin_data[4] = {
+        htonll(adjusted.data[3]), htonll(adjusted.data[2]), htonll(adjusted.data[1]), htonll(adjusted.data[0])
+    };
+
+    // The packer is then used to write the binary data to the buffer, just like in the old format.
+    packer.pack_bin(sizeof(bin_data));
+    packer.pack_bin_body((const char*)bin_data, sizeof(bin_data));
+}
+
+// This function is used to deserialize a field. It also matches the old deserialization format by 
+// reading the binary data as big endian uint64_t's, correcting them to the host endianness, and 
+// then converting the field back to Montgomery form. 
+template <class Params> void field<Params>::msgpack_unpack(auto o)
+{
+    // The binary data is first extracted from the msgpack object.
+    std::array<uint8_t, sizeof(data)> raw_data = o;
+
+    // The binary data is then read as big endian uint64_t's. This is done by casting the raw data to uint64_t* and then 
+    // using ntohll ("network to host long long") to correct the endianness to the host's endianness.
+    uint64_t* cast_data = (uint64_t*)&raw_data[0];
+    uint64_t reversed[] = {ntohll(cast_data[3]), ntohll(cast_data[2]), ntohll(cast_data[1]), ntohll(cast_data[0])};
+
+    // The corrected data is then copied back into the field's data array.
+    for (int i = 0; i < 4; i++) {
+        data[i] = reversed[i];  
+    }
+
+    // Finally, the field is converted back to Montgomery form, just like in the old format.
+    *this = to_montgomery_form();
+}
+
 } // namespace barretenberg
diff --git a/barretenberg/cpp/src/barretenberg/ecc/groups/affine_element.hpp b/barretenberg/cpp/src/barretenberg/ecc/groups/affine_element.hpp
@@ -3,6 +3,7 @@
 #include <vector>
 #include <type_traits>
 #include "barretenberg/ecc/curves/bn254/fq2.hpp"
+#include "barretenberg/serialize/msgpack.hpp"
 
 namespace barretenberg {
 namespace group_elements {
@@ -176,6 +177,8 @@ template <typename Fq, typename Fr, typename Params> class alignas(64) affine_el
     }
     Fq x;
     Fq y;
+    // for serialization: update with new fields
+    MSGPACK_FIELDS(x, y);
 };
 
 template <typename B, typename Fq, typename Fr, typename Params> void read(B& it, affine_element<Fq, Fr, Params>& value)

diff --git a/barretenberg/cpp/src/barretenberg/ecc/groups/affine_element.test.cpp b/barretenberg/cpp/src/barretenberg/ecc/groups/affine_element.test.cpp
@@ -5,6 +5,7 @@
 #include "barretenberg/common/test.hpp"
 #include <fstream>
 #include "barretenberg/common/serialize.hpp"
+#include "barretenberg/serialize/test_helper.hpp"
 
 namespace test_affine_element {
 template <typename G1> class test_affine_element : public testing::Test {
@@ -122,4 +123,10 @@ TEST(affine_element, infinity_ordering_regression)
     P.self_set_infinity();
     EXPECT_NE(P < Q, Q < P);
 }
+
+TEST(affine_element, msgpack)
+{
+    auto [actual, expected] = msgpack_roundtrip(secp256k1::g1::affine_element{ 1, 1 });
+    EXPECT_EQ(actual, expected);
+}
 } // namespace test_affine_element
diff --git a/barretenberg/cpp/src/barretenberg/ecc/groups/element.hpp b/barretenberg/cpp/src/barretenberg/ecc/groups/element.hpp
@@ -123,6 +123,8 @@ template <class Fq, class Fr, class Params> class alignas(32) element {
     //     }
     //     return { x, y, Fq::one() };
     // }
+    // for serialization: update with new fields
+    MSGPACK_FIELDS(x, y, z);
 
     static void conditional_negate_affine(const affine_element<Fq, Fr, Params>& in,
                                           affine_element<Fq, Fr, Params>& out,

diff --git a/barretenberg/cpp/src/barretenberg/ecc/groups/element_impl.hpp b/barretenberg/cpp/src/barretenberg/ecc/groups/element_impl.hpp
@@ -925,7 +925,6 @@ element<Fq, Fr, T> element<Fq, Fr, T>::random_coordinates_on_curve(numeric::rand
     Fq yy;
     Fq x;
     Fq y;
-    Fq t0;
     while (!found_one) {
         x = Fq::random_element(engine);
         yy = x.sqr() * x + T::b;
@@ -935,8 +934,6 @@ element<Fq, Fr, T> element<Fq, Fr, T>::random_coordinates_on_curve(numeric::rand
         auto [found_root, y1] = yy.sqrt();
         y = y1;
         found_one = found_root;
-        // t0 = y.sqr();
-        // found_one = (yy == t0);
     }
     return { x, y, Fq::one() };
 }

diff --git a/barretenberg/cpp/src/barretenberg/ecc/serialize.test.cpp b/barretenberg/cpp/src/barretenberg/ecc/serialize.test.cpp
@@ -0,0 +1,9 @@
+#include <gtest/gtest.h>
+#include "barretenberg/serialize/test_helper.hpp"
+#include "barretenberg/ecc/fields/field.hpp"
+
+TEST(msgpack_tests, msgpack_field)
+{
+    auto [actual, expected] = msgpack_roundtrip(barretenberg::fr{1ull, 2ull, 3ull, 4ull});
+    EXPECT_EQ(actual, expected);
+}
diff --git a/barretenberg/cpp/src/barretenberg/honk/sumcheck/sumcheck.test.cpp b/barretenberg/cpp/src/barretenberg/honk/sumcheck/sumcheck.test.cpp
@@ -96,7 +96,6 @@ TEST(Sumcheck, PolynomialNormalization)
     std::array<FF, multivariate_n> id_3;
     std::array<FF, multivariate_n> lagrange_first;
     std::array<FF, multivariate_n> lagrange_last;
-    std::array<FF, multivariate_n> pow_zeta;
     for (size_t i = 0; i < multivariate_n; i++) {
         w_l[i] = FF::random_element();
         w_r[i] = FF::random_element();
-Original file line number
+Diff line change
@@ Expand Up @@
         barretenberg::fr::serialize_to_buffer(r, result);
     }
     WASM_EXPORT void pedersen__compress(uint8_t const* inputs_buffer, uint8_t* output)
     {
         std::vector<grumpkin::fq> to_compress;
@@ Expand Down @@