feat: added poseidon2 hash function to barretenberg/crypto

[zac-williamson]

Preliminary work to add Poseidon2 hash function as a standard library primitive (https://eprint.iacr.org/2023/323.pdf)

Adds Poseidon2 to crypto module, following paper + specification at https://github.com/C2SP/C2SP/blob/792c1254124f625d459bfe34417e8f6bdd02eb28/poseidon-sponge.md

Checklist:

Remove the checklist to signal you've completed it. Enable auto-merge if the PR is ready to merge.

• If the pull request requires a cryptography review (e.g. cryptographic algorithm implementations) I have added the 'crypto' tag.
• I have reviewed my diff in github, line by line and removed unexpected formatting changes, testing logs, or commented-out code.
• Every change is related to the PR description.
• I have linked this pull request to relevant issues (if any exist).

[AztecBot]

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 4735bdeb and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,924	868,184	3,449,504
l1_rollup_execution_gas	842,011	3,595,292	22,204,873
l2_block_processing_time_in_ms	2,076 (+1%)	7,810 (-1%)	31,680 (-1%)
note_successful_decrypting_time_in_ms	343 (-2%)	991 (-4%)	3,600 (-4%)
note_trial_decrypting_time_in_ms	53.1 (-14%)	118 (-6%)	198 (-1%)
l2_block_building_time_in_ms	25,482	101,548	403,436 (-1%)
l2_block_rollup_simulation_time_in_ms	17,898	71,299 (-1%)	281,890 (-1%)
l2_block_public_tx_process_time_in_ms	7,534	30,092	120,980

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	23,279 (+3%)	43,807 (-1%)
note_history_successful_decrypting_time_in_ms	2,364 (+2%)	4,517 (+1%)
note_history_trial_decrypting_time_in_ms	162 (+1%)	230
node_database_size_in_bytes	3,316,086	3,662,515
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	399	43,077	20,441
private-kernel-ordering	127 (+1%)	25,833	9,689
base-rollup	3,183	659,500	873
root-rollup	93.6 (+4%)	4,072	1,097
private-kernel-inner	826	64,484	20,441
public-kernel-private-input	418	25,203	20,441
public-kernel-non-first-iteration	412	25,245	20,441
merge-rollup	10.9	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	10,323	29,083

[kevaundray]

Since the poseidon code is not comformant with the reference implementation, it probably makes sense to thoroughly review the changes outside of the poseidon2 module and the API of poseidon2.

Once we make poseidon2 conformant with a chosen reference implementation using test vectors, then it will become glaringly obvious if the implementation is "correct".

@zac-williamson what are your thoughts?

[kevaundray]

Would also say that while its not conformant we likely don't want to integrate it into other parts of the system as it may cause breaking changes once we make it conformant

[zac-williamson]

Since the poseidon code is not comformant with the reference implementation, it probably makes sense to thoroughly review the changes outside of the poseidon2 module and the API of poseidon2.

Once we make poseidon2 conformant with a chosen reference implementation using test vectors, then it will become glaringly obvious if the implementation is "correct".

@zac-williamson what are your thoughts?

I think this works and should be plan A, but it works only as long as we can get independent test vectors. The best I can find use t = 3 and we have t = 4. We might have to collaborate with another team to get an independent implementation that we can compare against

[zac-williamson]

Would also say that while its not conformant we likely don't want to integrate it into other parts of the system as it may cause breaking changes once we make it conformant

Yes I agree. I think we should get all of the hash structures implemented and working (i.e. have stuff in the stdlib, ideally also have ACIR opcodes working), but to not actually use Poseidon downstream until we've made it conformant

[lucasxia01]

Generally looks good to me. Had several questions and some nits.

[lucasxia01]

seems like a call to squeeze() and then absorb() will hit this condition

[lucasxia01]

but it does seem like we would not like to ever have an absorb AFTER a squeeze. Maybe we could throw an error in this case?

[lucasxia01]

why do we populate the cache with the same elements in state?

[lucasxia01]

I guess this is because we return cache elements as our output, but this seems like a weird way to do things.

[zac-williamson]

cache contains data that has yet to be included in the sponge, and state is the active data in the sponge.

state acts as a mutable data structure and cache doesn't. I think this is a safer way to structure the code as perform_duplex mutates state

[lucasxia01]

why not just return cache[cache_size-1]? and avoid this shifting process?

[zac-williamson]

cache is not a local variable but is a member of the Sponge class. It's state is important. We must remove the element that we are returning from the cache. Just returning cache[cache_size - 1] will not have the same behavior.

with regards to the manual action of shifting cache, we could have used a std::vector to define cache and used the pop method, but I wanted to use a fixed-size array to represent cache so that indexing the array out of bounds would more easily be caught by the compiler.

[lucasxia01]

I meant returning cache[cache_size-1] and also decrementing cache_size.

[zac-williamson]

Just thoroughness when developing the code. I wanted to rule out stale data in cache being a source of bugs.

[lucasxia01]

Since the poseidon code is not comformant with the reference implementation, it probably makes sense to thoroughly review the changes outside of the poseidon2 module and the API of poseidon2.
Once we make poseidon2 conformant with a chosen reference implementation using test vectors, then it will become glaringly obvious if the implementation is "correct".
@zac-williamson what are your thoughts?

I think this works and should be plan A, but it works only as long as we can get independent test vectors. The best I can find use t = 3 and we have t = 4. We might have to collaborate with another team to get an independent implementation that we can compare against

Can we try using an existing implementation with t=3, and modify it so that it uses t=4?

[lucasxia01]

Don't see how there's a differentiation between fixed length and variable length here. It seems like we only have fixed length hashing since we only take in field elements of 254 bits.

[zac-williamson]

the length refers to the number of field elements being hashed, not the number of bits in a field element.

Variable-length means that we are computing the hash of a dynamically-sized nubmer of field elements (i.e. the length varies from one hash to the next)

Fixed-length hash means we are computing the hash of a fixed-size nubmer of inputs (i.e. the length does not vary from one hash to the next)

A fixed-length hash is more efficient if we can get away with it, as we do not need to include the size of the hash in the preimage data being hashed.

[lucasxia01]

got it, seems like we are only doing variable length hashing then right?

[zac-williamson]

for now. Merkle membership hashes will likely use fixed length

[zac-williamson]

Since the poseidon code is not comformant with the reference implementation, it probably makes sense to thoroughly review the changes outside of the poseidon2 module and the API of poseidon2.
Once we make poseidon2 conformant with a chosen reference implementation using test vectors, then it will become glaringly obvious if the implementation is "correct".
@zac-williamson what are your thoughts?

I think this works and should be plan A, but it works only as long as we can get independent test vectors. The best I can find use t = 3 and we have t = 4. We might have to collaborate with another team to get an independent implementation that we can compare against

Can we try using an existing implementation with t=3, and modify it so that it uses t=4?

Yes I think we can do this.