Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduction of P2P networking #666

Merged
merged 101 commits into from
May 25, 2023
Merged
Show file tree
Hide file tree
Changes from 90 commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
bc8ffbb
WIP
PhilWindle Apr 30, 2023
ae8ea14
WIP
PhilWindle May 2, 2023
0d8d282
WIP
PhilWindle May 2, 2023
1d4c691
WIP
PhilWindle May 2, 2023
f397798
WIP
PhilWindle May 2, 2023
c81db00
WIP
PhilWindle May 2, 2023
fc7408e
WIP
PhilWindle May 2, 2023
9eae85c
Merge branch 'master' into pw/big-branch
PhilWindle May 3, 2023
cb6efb9
WIP
PhilWindle May 7, 2023
d91e373
Merge branch 'master' into pw/big-branch
PhilWindle May 7, 2023
71b3573
Merge fixes
PhilWindle May 7, 2023
bd809b4
Working multi provider setup
PhilWindle May 8, 2023
36ae237
Increase height of roots trees
PhilWindle May 8, 2023
1267dbb
WIP
PhilWindle May 10, 2023
5c287a7
Formatting
PhilWindle May 10, 2023
1d295fd
WIP
PhilWindle May 15, 2023
b3caf10
Reverted change
PhilWindle May 15, 2023
fdd8c6b
WIP
PhilWindle May 15, 2023
23c2d22
Docker fix
PhilWindle May 15, 2023
3974ab2
WIP
PhilWindle May 15, 2023
c1c2e6b
WIP
PhilWindle May 16, 2023
e323258
Setup relay nodes
PhilWindle May 16, 2023
fc5715b
Add dotenv to rollup provider
spalladino May 16, 2023
eaa4ebb
Don't propaget txs to node that we know have already seen them
PhilWindle May 17, 2023
1edb73e
Merge branch 'pw/big-branch' of github.com:AztecProtocol/aztec3-packa…
PhilWindle May 17, 2023
2b0cc43
Allow specification of client or server as P2P node
PhilWindle May 17, 2023
3b9ca32
Add dotenv for bootstrap node
spalladino May 17, 2023
290362f
Add env vars for announce ip:port and enabling nat
spalladino May 17, 2023
348df7b
Sync transaction pools on node start
PhilWindle May 18, 2023
db9c9b2
Handle node not started
PhilWindle May 18, 2023
1fb894d
WIP
PhilWindle May 18, 2023
18ddfeb
Message parser error
PhilWindle May 18, 2023
8abc58a
Fix tx propagation
PhilWindle May 18, 2023
73024fb
Fix tx propagation
PhilWindle May 18, 2023
99f6bd5
Review changes
PhilWindle May 19, 2023
26a93db
Review changes
PhilWindle May 19, 2023
fb38a75
Review changes
PhilWindle May 19, 2023
c642b40
Cleanup
PhilWindle May 19, 2023
431ea5b
Cleanup
PhilWindle May 19, 2023
a5eb396
Some cleanup and tests
PhilWindle May 22, 2023
287d444
More cleanup
PhilWindle May 22, 2023
13dd282
Merge branch 'master' into pw/big-branch
PhilWindle May 22, 2023
deda192
Merge fixes
PhilWindle May 22, 2023
b3cdda9
More cleanup
PhilWindle May 23, 2023
1561d87
Updated snapshots
PhilWindle May 23, 2023
208d483
More cleanup
PhilWindle May 23, 2023
8f1622e
Added job to circle ci
PhilWindle May 23, 2023
217f8f5
Fixes
PhilWindle May 23, 2023
d85089e
Cleanup
PhilWindle May 23, 2023
d8c92b1
Cleanup
PhilWindle May 23, 2023
65f9dd4
Rollup test fix
PhilWindle May 23, 2023
4ca7ecd
Test fixes
PhilWindle May 23, 2023
2e8bbe6
Test fixes
PhilWindle May 23, 2023
2df0a3e
Test fixes
PhilWindle May 23, 2023
f3e48fa
Fixes
PhilWindle May 23, 2023
a7450be
Cleanup
PhilWindle May 23, 2023
4a81c93
Test fixes
PhilWindle May 23, 2023
4b107e3
Temporarily skip tests
PhilWindle May 23, 2023
82ad983
Cleanup
PhilWindle May 23, 2023
1bb3d6f
Fixes
PhilWindle May 24, 2023
d11a154
Reverted tree size change
PhilWindle May 24, 2023
20ea874
Updated snapshots
PhilWindle May 24, 2023
ae6d27f
Updated snapshots (again)
PhilWindle May 24, 2023
e9e81f7
Attempt at e2e p2p test
PhilWindle May 24, 2023
a003e11
Build fix
PhilWindle May 24, 2023
96ee2a0
Fix
PhilWindle May 24, 2023
5a6dc8c
Test fixes
PhilWindle May 24, 2023
56f7427
Fix
PhilWindle May 24, 2023
b562885
Fix attempt
PhilWindle May 24, 2023
19c52fd
Debugging
PhilWindle May 24, 2023
6659e09
Debugging
PhilWindle May 24, 2023
116e770
Fix attempt
PhilWindle May 24, 2023
4825cab
Fix
PhilWindle May 24, 2023
dbdd99f
Test fix
PhilWindle May 24, 2023
1f0775c
Fix
PhilWindle May 24, 2023
ba48f18
Test fixes
PhilWindle May 24, 2023
b92221b
Test comments
PhilWindle May 24, 2023
6fdddaf
Working e2e test
PhilWindle May 24, 2023
2b95880
Build fix
PhilWindle May 24, 2023
6a992d5
Merge branch 'master' into pw/big-branch
PhilWindle May 24, 2023
0663d78
Merge fixes
PhilWindle May 24, 2023
e5cf247
Merge fixes
PhilWindle May 24, 2023
57852af
Merge fix
PhilWindle May 24, 2023
e2651e1
More merge fixes
PhilWindle May 24, 2023
05f4ae6
Merge fixes
PhilWindle May 24, 2023
8d2d3eb
More fixes
PhilWindle May 24, 2023
f2174a6
More fixes
PhilWindle May 24, 2023
37c8090
yarn prepare
PhilWindle May 24, 2023
1debaca
Build fix
PhilWindle May 24, 2023
a753c2b
Test fixes
PhilWindle May 24, 2023
5a4c96e
Review fixes
PhilWindle May 25, 2023
99d1284
Allow configuration of min/max peer counts
PhilWindle May 25, 2023
6b09faf
Review fixes
PhilWindle May 25, 2023
bd62b9f
Review changes
PhilWindle May 25, 2023
9eed093
Build fixes
PhilWindle May 25, 2023
a728e83
Formatting
PhilWindle May 25, 2023
d7eafce
Review changes
PhilWindle May 25, 2023
43cbfea
Build fix
PhilWindle May 25, 2023
34cbee4
Reverted old commit
PhilWindle May 25, 2023
d2fdbd4
Merge branch 'master' into pw/big-branch
PhilWindle May 25, 2023
b21d6cc
Updated snapshot
PhilWindle May 25, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,17 @@ jobs:
name: "Build and test"
command: build p2p

p2p-bootstrap:
machine:
image: ubuntu-2004:202010-01
resource_class: large
steps:
- *checkout
- *setup_env
- run:
name: "Build and test"
command: build p2p-bootstrap

acir-simulator:
machine:
image: ubuntu-2004:202010-01
Expand Down Expand Up @@ -352,6 +363,28 @@ jobs:
name: "Build"
command: build end-to-end

environment:
machine:
image: ubuntu-2004:202010-01
resource_class: large
steps:
- *checkout
- *setup_env
- run:
name: "Build"
command: build environment

rollup-provider:
machine:
image: ubuntu-2004:202010-01
resource_class: large
steps:
- *checkout
- *setup_env
- run:
name: "Build"
command: build rollup-provider

e2e-deploy-contract:
docker:
- image: aztecprotocol/alpine-build-image
Expand Down Expand Up @@ -429,6 +462,17 @@ jobs:
name: "Test"
command: cond_spot_run_tests end-to-end e2e_public_token_contract.test.ts

e2e-p2p:
docker:
- image: aztecprotocol/alpine-build-image
resource_class: small
steps:
- *checkout
- *setup_env
- run:
name: "Test"
command: cond_spot_run_tests end-to-end e2e_p2p_network.test.ts

e2e-join:
docker:
- image: cimg/base:current
Expand Down Expand Up @@ -512,6 +556,7 @@ workflows:

- aztec-js: *yarn_project
- end-to-end: *yarn_project
- environment: *yarn_project
- foundation: *yarn_project
- world-state: *yarn_project
- acir-simulator: *yarn_project
Expand All @@ -520,11 +565,13 @@ workflows:
- barretenberg-js: *yarn_project
- merkle-tree: *yarn_project
- p2p: *yarn_project
- p2p-bootstrap: *yarn_project
- noir-contracts: *yarn_project
- noir-compiler: *yarn_project
- sequencer-client: *yarn_project
- types: *yarn_project
- circuits-js: *yarn_project
- rollup-provider: *yarn_project

- e2e-join:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add the new jobs to e2e-join requirements, so all unit tests finish before we start the e2e ones

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed that, thanks

requires:
Expand Down Expand Up @@ -552,6 +599,7 @@ workflows:
- e2e-public-token-contract: *e2e_test
- e2e-l2-to-l1-messaging: *e2e_test
- integration-l1-publisher: *e2e_test
- e2e-p2p: *e2e_test

- e2e-end:
requires:
Expand All @@ -562,4 +610,5 @@ workflows:
- e2e-public-token-contract
- e2e-l2-to-l1-messaging
- integration-l1-publisher
- e2e-p2p
<<: *defaults
41 changes: 40 additions & 1 deletion build_manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@
"^yarn-project/archiver/"
],
"dependencies": [
"environment",
"foundation",
"l1-artifacts",
"types"
Expand All @@ -117,7 +118,12 @@
"^yarn-project/aztec-cli/"
],
"dependencies": [
"foundation"
"aztec-node",
"aztec.js",
"environment",
"foundation",
"l1-artifacts",
"noir-contracts"
]
},
"aztec-rpc": {
Expand Down Expand Up @@ -192,10 +198,20 @@
"foundation",
"l1-artifacts",
"noir-contracts",
"p2p",
"sequencer-client",
"world-state"
]
},
"environment": {
"buildDir": "yarn-project",
"projectDir": "yarn-project/environment",
"dockerfile": "environment/Dockerfile",
"rebuildPatterns": [
"^yarn-project/environment/"
],
"dependencies": []
},
"foundation": {
"buildDir": "yarn-project",
"projectDir": "yarn-project/foundation",
Expand Down Expand Up @@ -265,6 +281,17 @@
"types"
]
},
"p2p-bootstrap": {
"buildDir": "yarn-project",
"projectDir": "yarn-project/p2p-bootstrap",
"dockerfile": "p2p/Dockerfile",
"rebuildPatterns": [
"^yarn-project/p2p-bootstrap/"
],
"dependencies": [
"p2p"
]
},
"prover-client": {
"buildDir": "yarn-project",
"projectDir": "yarn-project/prover-client",
Expand All @@ -276,6 +303,17 @@
"foundation"
]
},
"rollup-provider": {
"buildDir": "yarn-project",
"projectDir": "yarn-project/rollup-provider",
"dockerfile": "rollup-provider/Dockerfile",
"rebuildPatterns": [
"^yarn-project/rollup-provider/"
],
"dependencies": [
"aztec-node"
]
},
"aztec-node": {
"buildDir": "yarn-project",
"projectDir": "yarn-project/aztec-node",
Expand Down Expand Up @@ -305,6 +343,7 @@
"dependencies": [
"acir-simulator",
"circuits.js",
"environment",
"foundation",
"l1-artifacts",
"merkle-tree",
Expand Down
1 change: 0 additions & 1 deletion circuits/cpp/src/aztec3/circuits/abis/c_bind.test.cpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#include "c_bind.h"

#include "function_leaf_preimage.hpp"
#include "tx_request.hpp"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ template <typename NCT> std::ostream& operator<<(std::ostream& os, PrivateCallDa
// specialize the name in msgpack schema generation
// consumed by the typescript schema compiler, helps disambiguate templates
template <typename NCT> inline std::string msgpack_schema_name(
aztec3::circuits::abis::CallStackItem<NCT, aztec3::circuits::abis::PrivateTypes> const&) // NOLINT
aztec3::circuits::abis::CallStackItem<NCT, aztec3::circuits::abis::PrivateTypes> const&) // NOLINT
{
return "PrivateCallStackItem";
}
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ template <typename NCT> std::ostream& operator<<(std::ostream& os, PublicCallDat
// specialize the name in msgpack schema generation
// consumed by the typescript schema compiler, helps disambiguate templates
template <typename NCT> inline std::string msgpack_schema_name(
aztec3::circuits::abis::CallStackItem<NCT, aztec3::circuits::abis::PublicTypes> const&) // NOLINT
aztec3::circuits::abis::CallStackItem<NCT, aztec3::circuits::abis::PublicTypes> const&) // NOLINT
{
return "PublicCallStackItem";
}
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ void perform_historical_contract_data_tree_membership_checks(DummyComposer& comp
NT::fr const leaf =
baseRollupInputs.kernel_data[i]
.public_inputs.constants.historic_tree_roots.private_historic_tree_roots.contract_tree_root;
abis::MembershipWitness<NT, PRIVATE_DATA_TREE_ROOTS_TREE_HEIGHT> const historic_root_witness =
abis::MembershipWitness<NT, CONTRACT_TREE_ROOTS_TREE_HEIGHT> const historic_root_witness =
baseRollupInputs.historic_contract_tree_root_membership_witnesses[i];

check_membership<NT>(composer,
Expand All @@ -182,7 +182,7 @@ void perform_historical_l1_to_l2_message_tree_membership_checks(DummyComposer& c
NT::fr const leaf =
baseRollupInputs.kernel_data[i]
.public_inputs.constants.historic_tree_roots.private_historic_tree_roots.l1_to_l2_messages_tree_root;
abis::MembershipWitness<NT, PRIVATE_DATA_TREE_ROOTS_TREE_HEIGHT> const historic_root_witness =
abis::MembershipWitness<NT, L1_TO_L2_MSG_TREE_ROOTS_TREE_HEIGHT> const historic_root_witness =
baseRollupInputs.historic_l1_to_l2_msg_tree_root_membership_witnesses[i];

check_membership<NT>(composer,
Expand Down
6 changes: 4 additions & 2 deletions circuits/cpp/src/aztec3/circuits/rollup/test_utils/utils.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -327,14 +327,16 @@ RootRollupInputs get_root_rollup_inputs(utils::DummyComposer& composer,
historic_contract_tree.update_element(0, contract_tree.root());
historic_l1_to_l2_msg_tree.update_element(0, l1_to_l2_msg_tree.root());

// Historic trees sibling paths
auto historic_data_sibling_path =
get_sibling_path<PRIVATE_DATA_TREE_ROOTS_TREE_HEIGHT>(historic_private_data_tree, 1, 0);
auto historic_contract_sibling_path =
get_sibling_path<CONTRACT_TREE_ROOTS_TREE_HEIGHT>(historic_contract_tree, 1, 0);
auto historic_l1_to_l2_msg_sibling_path =
get_sibling_path<L1_TO_L2_MSG_TREE_ROOTS_TREE_HEIGHT>(historic_l1_to_l2_msg_tree, 1, 0);
auto l1_to_l2_tree_sibling_path = get_sibling_path<L1_TO_L2_MSG_SUBTREE_INCLUSION_CHECK_DEPTH>(
l1_to_l2_msg_tree, 0, L1_TO_L2_MSG_SUBTREE_INCLUSION_CHECK_DEPTH);
// l1 to l2 tree
auto l1_to_l2_tree_sibling_path =
get_sibling_path<L1_TO_L2_MSG_SUBTREE_INCLUSION_CHECK_DEPTH>(l1_to_l2_msg_tree, 0, L1_TO_L2_MSG_SUBTREE_DEPTH);

// l1_to_l2_message tree snapshots
AppendOnlyTreeSnapshot const start_l1_to_l2_msg_tree_snapshot = {
Expand Down
2 changes: 1 addition & 1 deletion circuits/cpp/src/aztec3/constants.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ constexpr size_t KERNEL_PUBLIC_DATA_READS_LENGTH = 4;

constexpr size_t VK_TREE_HEIGHT = 3;
constexpr size_t FUNCTION_TREE_HEIGHT = 4;
constexpr size_t CONTRACT_TREE_HEIGHT = 4;
constexpr size_t CONTRACT_TREE_HEIGHT = 8;
constexpr size_t PRIVATE_DATA_TREE_HEIGHT = 8;
constexpr size_t NULLIFIER_TREE_HEIGHT = 8;
constexpr size_t PUBLIC_DATA_TREE_HEIGHT = 254;
Expand Down
6 changes: 3 additions & 3 deletions circuits/cpp/src/aztec3/utils/msgpack_derived_equals.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
namespace aztec3::utils {
// Auxiliary function for comparison of elements from tuples
template <typename Tuple1, typename Tuple2, std::size_t... I>
auto _compare_msgpack_tuples(const Tuple1& t1, const Tuple2& t2, std::index_sequence<I...>) // NOLINT
auto _compare_msgpack_tuples(const Tuple1& t1, const Tuple2& t2, std::index_sequence<I...>) // NOLINT
{
// Compare every 2nd value from our key1, value1, key2, value2 list
return ((std::get<I * 2 + 1>(t1) == std::get<I * 2 + 1>(t2)) && ...);
Expand All @@ -17,8 +17,8 @@ template <typename BoolLike, typename T> BoolLike msgpack_derived_equals(const T
{
BoolLike are_equal;
// De-serialize objects to alternating key-value tuples and compare the values
const_cast<T&>(obj1).msgpack([&](auto&... args1) { // NOLINT
const_cast<T&>(obj2).msgpack([&](auto&... args2) { // NOLINT
const_cast<T&>(obj1).msgpack([&](auto&... args1) { // NOLINT
const_cast<T&>(obj2).msgpack([&](auto&... args2) { // NOLINT
auto tuple1 = std::tie(args1...);
auto tuple2 = std::tie(args2...);
are_equal = _compare_msgpack_tuples(tuple1, tuple2, std::make_index_sequence<sizeof...(args1) / 2>{});
Expand Down
4 changes: 2 additions & 2 deletions circuits/cpp/src/aztec3/utils/msgpack_derived_output.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ inline void _msgpack_derived_output_helper(std::ostream& os,
os << key << ": ";
msgpack_derived_output(os, value);
os << '\n';
_msgpack_derived_output_helper(os, rest...); // NOLINT
_msgpack_derived_output_helper(os, rest...); // NOLINT
}
// Specialization if we have msgpack
template <msgpack_concepts::HasMsgPack T> void msgpack_derived_output(std::ostream& os, const T& value)
{
const_cast<T&>(value).msgpack([&](auto&... args) { _msgpack_derived_output_helper(os, args...); }); // NOLINT
const_cast<T&>(value).msgpack([&](auto&... args) { _msgpack_derived_output_helper(os, args...); }); // NOLINT
}

// Otherwise
Expand Down
9 changes: 6 additions & 3 deletions l1-contracts/src/periphery/UnverifiedDataEmitter.sol
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@ contract UnverifiedDataEmitter is IUnverifiedDataEmitter {
* @notice Logs data on chain
* @dev Emits an `UnverifiedData` event
* @param _l2BlockNum - The l2 block number that the unverified data is related to
* @param _l2BlockHash - The hash of the L2 block that this is related to
* @param _data - Raw data to share
*/
function emitUnverifiedData(uint256 _l2BlockNum, bytes calldata _data)
function emitUnverifiedData(uint256 _l2BlockNum, bytes32 _l2BlockHash, bytes calldata _data)
external
override(IUnverifiedDataEmitter)
{
emit UnverifiedData(_l2BlockNum, msg.sender, _data);
emit UnverifiedData(_l2BlockNum, msg.sender, _l2BlockHash, _data);
}

/**
Expand All @@ -31,14 +32,16 @@ contract UnverifiedDataEmitter is IUnverifiedDataEmitter {
* @param _l2BlockNum - The L2 block number that the contract deployment is related to
* @param _aztecAddress - The address of the L2 counterparty
* @param _portalAddress - The address of the L1 counterparty
* @param _l2BlockHash - The hash of the L2 block that this is related to
* @param _acir - The acir bytecode of the L2 contract
*/
function emitContractDeployment(
uint256 _l2BlockNum,
bytes32 _aztecAddress,
address _portalAddress,
bytes32 _l2BlockHash,
bytes calldata _acir
) external override(IUnverifiedDataEmitter) {
emit ContractDeployment(_l2BlockNum, _aztecAddress, _portalAddress, _acir);
emit ContractDeployment(_l2BlockNum, _aztecAddress, _portalAddress, _l2BlockHash, _acir);
}
}
11 changes: 9 additions & 2 deletions l1-contracts/src/periphery/interfaces/IUnverifiedDataEmitter.sol
Original file line number Diff line number Diff line change
Expand Up @@ -15,30 +15,37 @@ interface IUnverifiedDataEmitter {
* @param l2BlockNum - The L2 block number that the information is related to
* @param aztecAddress - The address of the L2 counterparty
* @param portalAddress - The address of the L1 counterparty
* @param l2BlockHash - The hash of the L2 block that this is related to
* @param acir - The acir bytecode of the L2 contract
*/
event ContractDeployment(
uint256 indexed l2BlockNum,
bytes32 indexed aztecAddress,
address indexed portalAddress,
bytes32 l2BlockHash,
bytes acir
);

/**
* @notice Used to share data which are not required to advance the state but are needed for other purposes
* @param l2BlockNum - The L2 block number that the information is related to
* @param sender - The address of the account sharing the information
* @param l2BlockHash - The hash of the L2 block that this is related to.
* @param data - The information represented as raw bytes
* @dev Typically contains `TxAuxData` (preimage, contract address and contract slot)
*/
event UnverifiedData(uint256 indexed l2BlockNum, address indexed sender, bytes data);
event UnverifiedData(
uint256 indexed l2BlockNum, address indexed sender, bytes32 indexed l2BlockHash, bytes data
);

function emitUnverifiedData(uint256 _l2BlockNum, bytes calldata _data) external;
function emitUnverifiedData(uint256 _l2BlockNum, bytes32 _l2BlockHash, bytes calldata _data)
external;

function emitContractDeployment(
uint256 _l2BlockNum,
bytes32 _aztecAddress,
address _portalAddress,
bytes32 _l2BlockHash,
bytes calldata _acir
) external;
}
1 change: 1 addition & 0 deletions yarn-project/archiver/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
"rootDir": "./src"
},
"dependencies": {
"@aztec/environment": "workspace:^",
"@aztec/foundation": "workspace:^",
"@aztec/l1-artifacts": "workspace:^",
"@aztec/types": "workspace:^",
Expand Down
Loading