feat: implement Zeromorph (microsoft#145)

* Schoolbook adapted KZG PoC Remove toxic waste from setup parameters Refactorings ZeroMorph skeleton * feat: provide a non-hiding variant of KZG Drafts a non-hiding variant of the KZG PCS on univariate Dense polynomial representations Details: - Changed access levels and ordering for various modules and functions like `sumcheck`, `cpu_best_multiexp`, and `mod kzg;`. - Upgraded `halo2curves` version and added new dependencies like `pairing`, `anyhow`, `rand`, and `group`. - Leveraged the UniPoly in sumcheck.rs - Created `non_hiding_kzg.rs` file, introducing new structures and functionalities like `UVUniversalKZGParam`, `UVKZGProverKey`, `UVKZGPoly`, and `UVKZGPCS` along with their implementation and tests. - Modified the import of `Engine` in `kzg.rs` from `halo2curves::pairing::Engine` to `pairing::Engine` (reflecting the new version of halo2curves). feat: add batch commit / open / prove * Small subtasks: params, phi, un function, EE Minor adjustments * chore: setup zeromorph refactor: Remove kzg and zeromorph provider modules * feat: add openings for Zeromorph - Added new data structures and their related functions including the `ZMProverKey`, `ZMVerifierKey`, `ZMCommitment`, `ZMEvaluation`, `ZMProof` and `ZMPCS` in `non_hiding_zeromorph.rs` - Implemented new functionalities in `non_hiding_zeromorph.rs` and `spartan/polynomial.rs` to provide better handling of polynomials, including fetching commitments for a polynomial, computing the quotient polynomials of an existing polynomial. - Enhanced the `UniPoly` struct in `spartan/sumcheck.rs` with `Index` and `IndexMut` for better coefficient access, and `AddAssign` and `MulAssign` for scalar and self types. - Removed and replaced certain elements in `UVUniversalKZGParam` struct in `non_hiding_kzg.rs` for improved flexibility, and included import and implementation of `TranscriptReprTrait`. draft ZM verify set up test structure defer test_quo * Include commits into ZMProof and adjust RO * feat: Refactor polynomial evaluation methods - Temporarily disabled some assertions in both `non_hiding_zeromorph.rs` and `polynomial.rs` for debugging purposes. - Introduced `evaluate_opt`, `fix_variables`, and `fix_one_variable_helper` functions in `polynomial.rs` to support multilinear polynomial evaluation and partial evaluation. fix: add domain separators - Integrated additional functionality into the ZMPCS implementation in `non_hiding_zeromorph.rs` by adding a protocol name function. - Improved security in `non_hiding_zeromorph.rs` by integrating domain separators into transcript's `open` and `verify` functions. refactor: Refactor code and test performance for NHZM - Implement `AsRef` trait for `UniPoly` structure in `src/spartan/sumcheck.rs` to facilitate direct access to its coefficients. - Refactor code in `src/provider/non_hiding_zeromorph.rs` to directly use `into_iter` in map function when creating `quotients_polys`, avoiding a large clone - Enhance test performance in `src/provider/non_hiding_zeromorph.rs` by pre-generating a `universal_setup` for tests, introducing a `max_vars` variable and RNG in the `commit_open_verify_with` test function, and adjusting the range of num_vars used for testing. fix: adjust APIs & Serialize impls for the Nova traits * feat: set up serde & abomonation bounds * add generic PCS errors * pp wip * fix: adjust type parameters for nontrivial_with_zm test fix: ignore panicking test fix: add doctest for evaluate_opt fix: remove obsolete comments chore: move UniPoly methods where they should be test: make clear current zeromorph operates in monomial basis - Added an additional test `test_dense_evaliations()` to provide more comprehensive testing for the `evaluate()` and `evaluate_opt()` functions in `MultilinearPolynomial`. refactor TranscriptReprTrait impl for compat with Commitments * feat: Implement KZG commitment trait and serialization features - Added serialization, deserialization, and Abomonation traits to UVUniversalKZGParam struct in `non_hiding_kzg.rs` file along with implementing comparison and length evaluation traits. - Created a new file `kzg_commitment.rs` which implements KZG Commitment Engine and setup, commit functions. - Integrated `kzg_commitment` in module `provider` and set up conversions between Commitment and UVKZGCommitment. - Enhanced assertion in `minroot_serde.rs` file from clone comparison to dereferenced comparison in MinRoot delay case. * Use the ZMPCS Evaluation Engine and the KZG Commitment Engine in tests. feat: Improve `prove` and `verify` methods in `ZMPCS` struct - Updated `prove` and `verify` methods in `ZMPCS<E>` struct within `non_hiding_zeromorph.rs` with actual logic for commitment, evaluation, and verification. - Adjusted the construction of `ZMCommitment` and `ZMEvaluation` within `prove` and `verify` methods. - Commented on portions of the code in `non_hiding_zeromorph.rs` that need further modification and refinement. - Modified test value for `test_pp_digest_with` in the `test_supernova_pp_digest` test case to match the new expected output. * fix evaluation reversal bug fix: remove superfluous eval functions fix: remove endianness shenanigans test: add evaluation unit test * fix: parallellize pp generation * feat: Enhance KZG commitment SRS generation efficiency using parallel computation - Introduced a new module `util` within the `provider` module, implementing fixed-base MSM, - New trait constraint has been imposed for `E::Fr: PrimeFieldBits` within the `non_hiding_zeromorph.rs` file, and usages have been adjusted in the `test` module. - Adding the `PrimeFieldBits` import from the `ff` crate and importing `provider::util` from the local crate. refactor: Refactor utility functions for elliptic curve operations - Renamed and moved `util.rs` to `fb_msm.rs` under the `provider` directory. - documented the FB MSM * Apply comments from code review Co-authored-by: Adrian Hamelink <[email protected]> * Improve documentation and testing in non_hiding_zeromorph.rs - Enhanced the clarity and accuracy of code comments and documentation in `non_hiding_zeromorph.rs`, specifically within the `ZMPCS` struct's methods. - Enriched the `quotients` function's documentation, detailing its mathematical underpinnings - Implemented a new rigorous test, `test_quotients`, to ensure the `quotients` function's output satisfies the polynomial identity. * test: refactor random ML poly generation * Finish documenting / testing Zeromorph (microsoft#142) * fix: remove a TODO using RefCast * fix: remove unsightly clone * doc: comment intermediate ZM steps * refactor: Refactor `open` function in zeromorph provider Extracted the computation of `q_hat` in the `open` function into a new function `batched_lifted_degree_quotient` for more modular code. * test: test batched_lifted_degree_quotient Addition of a new test `test_batched_lifted_degree_quotient` to validate batched degree quotient lifting for polynomials. * refactor: Refine computations in non_hiding_zeromorph.rs - Overhauled the 'eval_and_quotient_scalars' function with a revised return type with pair of vectors rather than scalar and a vector, - correspondingly split scalars `degree_check_q_scalars` and `zmpoly_q_scalars` within the `open` method of `non_hiding_zeromorph.rs`. - Adjusted the 'verify' method to accommodate the modified 'eval_and_quotient_scalars' function output changes. * test: test partially evaluated quotient \zeta_x - Introduced a new test case `test_partially_evaluated_quotient_zeta` to validate `zeta_x` construction. - correct an off-by-one error in the y_k terms * test: test partially evaluated quotient - Created a new function `phi` for evaluating using an inefficient formula in `non_hiding_zeromorph.rs` - Implemented a new test function `test_partially_evaluated_quotient_z` for validation of `Z_x` computation method in `non_hiding_zeromorph.rs` * fix: refine q_hat comment * fix: organize code more sparsely after rebase * refactor: Improve code readability and error handling in zeromorph - Updated and enhanced clarity and consistency of mathematical notation in comments across `non_hiding_kzg.rs` and `non_hiding_zeromorph.rs` files. - Implemented error handling in the `ZMPCS::verify` function within the `non_hiding_zeromorph.rs` file. * fix: compute quotient commitments in parallel * refactor: clean up --------- Co-authored-by: emmorais <[email protected]> Co-authored-by: Matej Penciak <[email protected]> Co-authored-by: Adrian Hamelink <[email protected]>
lurk-lang · May 2, 2024 · 75cfaa7 · 75cfaa7
1 parent 9d0edf2
commit 75cfaa7
Show file tree

Hide file tree

Showing 12 changed files with 1,562 additions and 26 deletions.
diff --git a/Cargo.toml b/Cargo.toml
@@ -36,6 +36,8 @@ thiserror = "1.0"
 group = "0.13.0"
 once_cell = "1.18.0"
 itertools = "0.12.0"
+rand = "0.8.5"
+ref-cast = "1.0.20"
 
 [target.'cfg(any(target_arch = "x86_64", target_arch = "aarch64"))'.dependencies]
 pasta-msm = { version = "0.1.4" }

diff --git a/src/errors.rs b/src/errors.rs
@@ -34,9 +34,9 @@ pub enum NovaError {
   /// returned if the provided number of steps is zero
   #[error("InvalidNumSteps")]
   InvalidNumSteps,
-  /// returned when an invalid PCS evaluation argument is provided
-  #[error("InvalidPCS")]
-  InvalidPCS,
+  /// returned if there is an error in the proof/verification of a PCS
+  #[error("PCSError")]
+  PCSError(#[from] PCSError),
   /// returned when an invalid sum-check proof is provided
   #[error("InvalidSumcheckProof")]
   InvalidSumcheckProof,
@@ -79,3 +79,17 @@ impl From<bellpepper_core::SynthesisError> for NovaError {
     }
   }
 }
+
+/// Errors specific to the Polynomial commitment scheme
+#[derive(Clone, Debug, Eq, PartialEq, Error)]
+pub enum PCSError {
+  /// returned when an invalid inner product argument is provided
+  #[error("InvalidIPA")]
+  InvalidIPA,
+  /// returned when there is a Zeromorph error
+  #[error("ZMError")]
+  ZMError,
+  /// returned when a length check fails in a PCS
+  #[error("LengthError")]
+  LengthError,
+}
diff --git a/src/lib.rs b/src/lib.rs
@@ -850,15 +850,16 @@ mod tests {
   use super::*;
   use crate::{
     provider::{
-      pedersen::CommitmentKeyExtTrait, traits::DlogGroup, Bn256EngineIPA, Bn256EngineKZG,
-      GrumpkinEngine, PallasEngine, Secp256k1Engine, Secq256k1Engine, VestaEngine,
+      traits::DlogGroup, Bn256EngineIPA, Bn256EngineKZG, Bn256EngineZM,
+      GrumpkinEngine, PallasEngine, Secp256k1Engine, Secq256k1Engine, VestaEngine, non_hiding_zeromorph::ZMPCS,
     },
     traits::{circuit::TrivialCircuit, evaluation::EvaluationEngineTrait, snark::default_ck_hint},
   };
   use ::bellpepper_core::{num::AllocatedNum, ConstraintSystem, SynthesisError};
   use core::{fmt::Write, marker::PhantomData};
   use expect_test::{expect, Expect};
   use ff::PrimeField;
+  use halo2curves::bn256::Bn256;
 
   type EE<E> = provider::ipa_pc::EvaluationEngine<E>;
   type EEPrime<E> = provider::hyperkzg::EvaluationEngine<E>;
@@ -913,21 +914,20 @@ mod tests {
     }
   }
 
-  fn test_pp_digest_with<E1, E2, T1, T2>(circuit1: &T1, circuit2: &T2, expected: &Expect)
+  fn test_pp_digest_with<E1, E2, T1, T2, EE1, EE2>(circuit1: &T1, circuit2: &T2, expected: &Expect)
   where
     E1: Engine<Base = <E2 as Engine>::Scalar>,
     E2: Engine<Base = <E1 as Engine>::Scalar>,
     E1::GE: DlogGroup,
     E2::GE: DlogGroup,
     T1: StepCircuit<E1::Scalar>,
     T2: StepCircuit<E2::Scalar>,
-    // required to use the IPA in the initialization of the commitment key hints below
-    <E1::CE as CommitmentEngineTrait<E1>>::CommitmentKey: CommitmentKeyExtTrait<E1>,
-    <E2::CE as CommitmentEngineTrait<E2>>::CommitmentKey: CommitmentKeyExtTrait<E2>,
+    EE1: EvaluationEngineTrait<E1>,
+    EE2: EvaluationEngineTrait<E2>,
   {
     // this tests public parameters with a size specifically intended for a spark-compressed SNARK
-    let ck_hint1 = &*SPrime::<E1, EE<E1>>::ck_floor();
-    let ck_hint2 = &*SPrime::<E2, EE<E2>>::ck_floor();
+    let ck_hint1 = &*SPrime::<E1, EE1>::ck_floor();
+    let ck_hint2 = &*SPrime::<E2, EE2>::ck_floor();
     let pp = PublicParams::<E1, E2, T1, T2>::setup(circuit1, circuit2, ck_hint1, ck_hint2).unwrap();
 
     let digest_str = pp
@@ -939,24 +939,25 @@ mod tests {
         let _ = write!(output, "{b:02x}");
         output
       });
+
     expected.assert_eq(&digest_str);
   }
 
   #[test]
   fn test_pp_digest() {
-    test_pp_digest_with::<PallasEngine, VestaEngine, _, _>(
+    test_pp_digest_with::<PallasEngine, VestaEngine, _, _, EE<_>, EE<_>>(
       &TrivialCircuit::<_>::default(),
       &TrivialCircuit::<_>::default(),
       &expect!["a69d6cf6d014c3a5cc99b77afc86691f7460faa737207dd21b30e8241fae8002"],
     );
 
-    test_pp_digest_with::<Bn256EngineIPA, GrumpkinEngine, _, _>(
+    test_pp_digest_with::<Bn256EngineIPA, GrumpkinEngine, _, _, EE<_>, EE<_>>(
       &TrivialCircuit::<_>::default(),
       &TrivialCircuit::<_>::default(),
       &expect!["b22ab3456df4bd391804a39fae582b37ed4a8d90ace377337940ac956d87f701"],
     );
 
-    test_pp_digest_with::<Secp256k1Engine, Secq256k1Engine, _, _>(
+    test_pp_digest_with::<Secp256k1Engine, Secq256k1Engine, _, _, EE<_>, EE<_>>(
       &TrivialCircuit::<_>::default(),
       &TrivialCircuit::<_>::default(),
       &expect!["c8aec89a3ea90317a0ecdc9150f4fc3648ca33f6660924a192cafd82e2939b02"],
@@ -1197,6 +1198,12 @@ mod tests {
       provider::hyperkzg::EvaluationEngine<_>,
       EE<_>,
     >();
+    test_ivc_nontrivial_with_compression_with::<
+      Bn256EngineZM,
+      GrumpkinEngine,
+      ZMPCS<Bn256, _>,
+      EE<_>,
+    >();
   }
 
   fn test_ivc_nontrivial_with_spark_compression_with<E1, E2, EE1, EE2>()
@@ -1300,6 +1307,12 @@ mod tests {
     >();
     test_ivc_nontrivial_with_spark_compression_with::<Secp256k1Engine, Secq256k1Engine, EE<_>, EE<_>>(
     );
+    test_ivc_nontrivial_with_spark_compression_with::<
+      Bn256EngineZM,
+      GrumpkinEngine,
+      ZMPCS<Bn256, _>,
+      EE<_>,
+    >();
   }
 
   fn test_ivc_nondet_with_compression_with<E1, E2, EE1, EE2>()
@@ -1440,6 +1453,8 @@ mod tests {
     test_ivc_nondet_with_compression_with::<PallasEngine, VestaEngine, EE<_>, EE<_>>();
     test_ivc_nondet_with_compression_with::<Bn256EngineKZG, GrumpkinEngine, EEPrime<_>, EE<_>>();
     test_ivc_nondet_with_compression_with::<Secp256k1Engine, Secq256k1Engine, EE<_>, EE<_>>();
+    test_ivc_nondet_with_compression_with::<Bn256EngineZM, GrumpkinEngine, ZMPCS<Bn256, _>, EE<_>>(
+    );
   }
 
   fn test_ivc_base_with<E1, E2>()

diff --git a/src/provider/ipa_pc.rs b/src/provider/ipa_pc.rs
@@ -1,6 +1,6 @@
 //! This module implements `EvaluationEngine` using an IPA-based polynomial commitment scheme
 use crate::{
-  errors::NovaError,
+  errors::{NovaError, PCSError},
   provider::{pedersen::CommitmentKeyExtTrait, traits::DlogGroup},
   spartan::polys::eq::EqPolynomial,
   traits::{
@@ -403,7 +403,7 @@ where
     if P_hat == CE::<E>::commit(&ck_hat.combine(&ck_c), &[self.a_hat, self.a_hat * b_hat]) {
       Ok(())
     } else {
-      Err(NovaError::InvalidPCS)
+      Err(NovaError::PCSError(PCSError::InvalidIPA))
     }
   }
 }
diff --git a/src/provider/kzg_commitment.rs b/src/provider/kzg_commitment.rs
@@ -0,0 +1,78 @@
+//! Commitment engine for KZG commitments
+//!
+
+use std::marker::PhantomData;
+
+use ff::PrimeFieldBits;
+use group::{prime::PrimeCurveAffine, Curve};
+use halo2curves::pairing::Engine;
+use rand::rngs::StdRng;
+use rand_core::SeedableRng;
+use serde::{Deserialize, Serialize};
+
+use crate::traits::{
+  commitment::{CommitmentEngineTrait, Len},
+  Engine as NovaEngine, Group,
+};
+
+use crate::provider::{
+  non_hiding_kzg::{UVKZGCommitment, UVUniversalKZGParam},
+  pedersen::Commitment,
+  traits::DlogGroup,
+};
+
+/// Provides a commitment engine
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct KZGCommitmentEngine<E: Engine> {
+  _p: PhantomData<E>,
+}
+
+impl<E: Engine, NE: NovaEngine<GE = E::G1, Scalar = E::Fr>> CommitmentEngineTrait<NE>
+  for KZGCommitmentEngine<E>
+where
+  E::G1: DlogGroup<AffineGroupElement = E::G1Affine, Scalar = E::Fr>,
+  E::G1Affine: Serialize + for<'de> Deserialize<'de>,
+  E::G2Affine: Serialize + for<'de> Deserialize<'de>,
+  E::Fr: PrimeFieldBits, // TODO due to use of gen_srs_for_testing, make optional
+{
+  type CommitmentKey = UVUniversalKZGParam<E>;
+  type Commitment = Commitment<NE>;
+
+  fn setup(label: &'static [u8], n: usize) -> Self::CommitmentKey {
+    // TODO: this is just for testing, replace by grabbing from a real setup for production
+    let mut bytes = [0u8; 32];
+    let len = label.len().min(32);
+    bytes[..len].copy_from_slice(&label[..len]);
+    let rng = &mut StdRng::from_seed(bytes);
+    UVUniversalKZGParam::gen_srs_for_testing(rng, n.next_power_of_two())
+  }
+
+  fn commit(ck: &Self::CommitmentKey, v: &[<E::G1 as Group>::Scalar]) -> Self::Commitment {
+    assert!(ck.length() >= v.len());
+    Commitment {
+      comm: E::G1::vartime_multiscalar_mul(v, &ck.powers_of_g[..v.len()]),
+    }
+  }
+}
+
+impl<E: Engine, NE: NovaEngine<GE = E::G1, Scalar = E::Fr>> From<Commitment<NE>>
+  for UVKZGCommitment<E>
+where
+  E::G1: Group,
+{
+  fn from(c: Commitment<NE>) -> Self {
+    UVKZGCommitment(c.comm.to_affine())
+  }
+}
+
+impl<E: Engine, NE: NovaEngine<GE = E::G1, Scalar = E::Fr>> From<UVKZGCommitment<E>>
+  for Commitment<NE>
+where
+  E::G1: Group,
+{
+  fn from(c: UVKZGCommitment<E>) -> Self {
+    Commitment {
+      comm: c.0.to_curve(),
+    }
+  }
+}
diff --git a/src/provider/mod.rs b/src/provider/mod.rs
@@ -3,6 +3,7 @@
 // public modules to be used as an evaluation engine with Spartan
 pub mod hyperkzg;
 pub mod ipa_pc;
+pub mod non_hiding_zeromorph;
 
 // crate-public modules, made crate-public mostly for tests
 pub(crate) mod bn256_grumpkin;
@@ -11,6 +12,10 @@ pub(crate) mod pedersen;
 pub(crate) mod poseidon;
 pub(crate) mod secp_secq;
 pub(crate) mod traits;
+// a non-hiding variant of {kzg, zeromorph}
+pub(crate) mod kzg_commitment;
+pub(crate) mod non_hiding_kzg;
+mod util;
 
 // crate-private modules
 mod keccak;
@@ -26,9 +31,12 @@ use crate::{
   },
   traits::Engine,
 };
+use halo2curves::bn256::Bn256;
 use pasta_curves::{pallas, vesta};
 use serde::{Deserialize, Serialize};
 
+use self::kzg_commitment::KZGCommitmentEngine;
+
 /// An implementation of Nova traits with HyperKZG over the BN256 curve
 #[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)]
 pub struct Bn256EngineKZG;
@@ -71,6 +79,20 @@ impl Engine for GrumpkinEngine {
   type CE = PedersenCommitmentEngine<Self>;
 }
 
+/// An implementation of the Nova `Engine` trait with BN254 curve and Zeromorph commitment scheme
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub struct Bn256EngineZM;
+
+impl Engine for Bn256EngineZM {
+  type Base = bn256::Base;
+  type Scalar = bn256::Scalar;
+  type GE = bn256::Point;
+  type RO = PoseidonRO<Self::Base, Self::Scalar>;
+  type ROCircuit = PoseidonROCircuit<Self::Base>;
+  type TE = Keccak256Transcript<Self>;
+  type CE = KZGCommitmentEngine<Bn256>;
+}
+
 /// An implementation of the Nova `Engine` trait with Secp256k1 curve and Pedersen commitment scheme
 #[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)]
 pub struct Secp256k1Engine;