Integrate PeerDAS cryptography library

[kevaundray]

PR Description

On top of c-kzg, we plan to have a Rust library that compiles to Java and all of the other targets. A preliminary release has been made, this PR is being made to show the end to end integration of the library and possibly surface any issues.

Fixed Issue(s)

Documentation

• I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

• I thought about adding a changelog entry, and added one if I deemed necessary.

[kevaundray]

Note: Things like namespaces will change in the future

[kevaundray]

One thing to note:

• This repo currently uses an old API for c-kzg where you are able to recover the Blob. The new consensus-specs does not allow this, it has become encapsulated in a method called recoverCellsAndproofs which returns all of the cells and the proofs for them.

[kevaundray]

Noting the use of BeforeEach for the setup method in tests. This will implicitly test that loading the same trusted setup does not throw or break. This is in addition to the explicit testKzgLoadSameTrustedSetupTwice_shouldNotThrowException function.

See here:

teku/infrastructure/kzg/src/test/java/tech/pegasys/teku/kzg/CKZG4844Test.java

Line 53 in 3768eca

@BeforeEach

*BeforeEach as opposed to BeforeAll

[kevaundray]

Currently to run tests with the rust lib, you can do ./gradlew :infrastructure:kzg:testWithPeerDas , the end solution may look something like ./gradlew test -D use.peer.das not really sure what the teku convention is here, so open to whatever is best

[kevaundray]

Something I noticed, some methods have try-catch logic while others (mainly the new peerdas methods) do not -- Not blocking for this PR, though it looks undesirable since you are not wrapping the potentially arbitrary exceptions from the underlying crypto library into a KZGException

[kevaundray]

We could also modify BlobToKZGCommitment to:

@Override
  public KZGCommitment blobToKzgCommitment(final Bytes blob) throws KZGException {

    try {
      byte[] commitmentBytes;

      if (USE_PEER_DAS) {
        commitmentBytes = peerDASinstance.blobToKZGCommitment(blob.toArrayUnsafe());
      } else {
        commitmentBytes = CKZG4844JNI.blobToKzgCommitment(blob.toArrayUnsafe());
      }

      return KZGCommitment.fromArray(commitmentBytes);
    } catch (final Exception ex) {
      throw new KZGException("Failed to produce KZG commitment from blob", ex);
    }
  }

But there are CKZG specific errors in the tests so that would require more changes than I am hoping for

[kevaundray]

Changed the flag name, tests can now be ran with ./gradlew :infrastructure:kzg:testWithRustKZG

[zilm13]

I've finished with all changes I wanted, happy to merge.
So at the end we have --Xrust-kzg-enabled command line option to enable library and separate tests for both implementation and rust with fallback variant, which will be used with flag.
I've tested with flag enabled in Kurtosis, no issues detected.

[zilm13]

LGTM

[kevaundray]

I've finished with all changes I wanted, happy to merge. So at the end we have --Xrust-kzg-enabled command line option to enable library and separate tests for both implementation and rust with fallback variant, which will be used with flag. I've tested with flag enabled in Kurtosis, no issues detected.

Thanks for doing the heavy lifting to separate the two libraries into two different files and adding a flag to the command line, and for adding 4844 fallback methods for the Rust lib!