Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: big synching case + stability #9022

Merged
merged 1 commit into from
Oct 4, 2024
Merged

chore: big synching case + stability #9022

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions yarn-project/aztec.js/src/utils/cheat_codes.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error mining: ${res.error.message}`);
}
this.logger.info(`Mined ${numberOfBlocks} blocks`);
this.logger.verbose(`Mined ${numberOfBlocks} blocks`);
}

/**
Expand All @@ -138,7 +138,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error setting balance for ${account}: ${res.error.message}`);
}
this.logger.info(`Set balance for ${account} to ${balance}`);
this.logger.verbose(`Set balance for ${account} to ${balance}`);
}

/**
Expand All @@ -150,7 +150,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error setting block interval: ${res.error.message}`);
}
this.logger.info(`Set block interval to ${interval}`);
this.logger.verbose(`Set block interval to ${interval}`);
}

/**
Expand All @@ -162,7 +162,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error setting next block timestamp: ${res.error.message}`);
}
this.logger.info(`Set next block timestamp to ${timestamp}`);
this.logger.verbose(`Set next block timestamp to ${timestamp}`);
}

/**
Expand All @@ -175,7 +175,7 @@ export class EthCheatCodes {
throw new Error(`Error warping: ${res.error.message}`);
}
await this.mine();
this.logger.info(`Warped to ${timestamp}`);
this.logger.verbose(`Warped to ${timestamp}`);
}

/**
Expand All @@ -189,7 +189,7 @@ export class EthCheatCodes {
}
const jsonContent = JSON.stringify(res.result);
fs.writeFileSync(`${fileName}.json`, jsonContent, 'utf8');
this.logger.info(`Dumped state to ${fileName}`);
this.logger.verbose(`Dumped state to ${fileName}`);
}

/**
Expand All @@ -202,7 +202,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error loading state: ${res.error.message}`);
}
this.logger.info(`Loaded state from ${fileName}`);
this.logger.verbose(`Loaded state from ${fileName}`);
}

/**
Expand All @@ -228,7 +228,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error setting storage for contract ${contract} at ${slot}: ${res.error.message}`);
}
this.logger.info(`Set storage for contract ${contract} at ${slot} to ${value}`);
this.logger.verbose(`Set storage for contract ${contract} at ${slot} to ${value}`);
}

/**
Expand All @@ -252,7 +252,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error impersonating ${who}: ${res.error.message}`);
}
this.logger.info(`Impersonating ${who}`);
this.logger.verbose(`Impersonating ${who}`);
}

/**
Expand All @@ -264,7 +264,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error when stopping the impersonation of ${who}: ${res.error.message}`);
}
this.logger.info(`Stopped impersonating ${who}`);
this.logger.verbose(`Stopped impersonating ${who}`);
}

/**
Expand All @@ -277,7 +277,7 @@ export class EthCheatCodes {
if (res.error) {
throw new Error(`Error setting bytecode for ${contract}: ${res.error.message}`);
}
this.logger.info(`Set bytecode for ${contract} to ${bytecode}`);
this.logger.verbose(`Set bytecode for ${contract} to ${bytecode}`);
}

/**
Expand Down
82 changes: 59 additions & 23 deletions yarn-project/end-to-end/src/e2e_synching.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ import { type EndToEndContext, getPrivateKeyFromIndex, setup, setupPXEService }
const SALT = 420;
const AZTEC_GENERATE_TEST_DATA = !!process.env.AZTEC_GENERATE_TEST_DATA;
const START_TIME = 1893456000; // 2030 01 01 00 00
const RUN_THE_BIG_ONE = !!process.env.RUN_THE_BIG_ONE;

const MINT_AMOUNT = 1000n;

Expand Down Expand Up @@ -338,6 +339,7 @@ const variants: VariantDefinition[] = [
{ blockCount: 10, txCount: 36, txComplexity: TxComplexity.PrivateTransfer },
{ blockCount: 10, txCount: 36, txComplexity: TxComplexity.PublicTransfer },
{ blockCount: 10, txCount: 9, txComplexity: TxComplexity.Spam },
{ blockCount: 1000, txCount: 4, txComplexity: TxComplexity.PrivateTransfer },
];

describe('e2e_synching', () => {
Expand All @@ -349,10 +351,23 @@ describe('e2e_synching', () => {
if (!AZTEC_GENERATE_TEST_DATA) {
return;
}

// @note If the `RUN_THE_BIG_ONE` flag is not set, we DO NOT run it.
if (!RUN_THE_BIG_ONE && variantDef.blockCount === 1000) {
return;
}

const variant = new TestVariant(variantDef);

// The setup is in here and not at the `before` since we are doing different setups depending on what mode we are running in.
const { teardown, pxe, sequencer, aztecNode, wallet } = await setup(1, { salt: SALT, l1StartTime: START_TIME });
// We require that at least 200 eth blocks have passed from the START_TIME before we see the first L2 block
// This is to keep the setup more stable, so as long as the setup is less than 100 L1 txs, changing the setup should not break the setup
const { teardown, pxe, sequencer, aztecNode, wallet } = await setup(1, {
salt: SALT,
l1StartTime: START_TIME,
l2StartTime: START_TIME + 200 * ETHEREUM_SLOT_DURATION,
assumeProvenThrough: 10 + variant.blockCount,
});
variant.setPXE(pxe as PXEService);

// Deploy a token, such that we could use it
Expand Down Expand Up @@ -381,12 +396,13 @@ describe('e2e_synching', () => {
await variant.writeBlocks(blocks);
await teardown();
},
2_400_000,
240_400_000,
);

const testTheVariant = async (
variant: TestVariant,
alternativeSync: (opts: Partial<EndToEndContext>, variant: TestVariant) => Promise<void>,
assumeProvenThrough: number = Number.MAX_SAFE_INTEGER,
) => {
if (AZTEC_GENERATE_TEST_DATA) {
return;
Expand All @@ -397,6 +413,7 @@ describe('e2e_synching', () => {
salt: SALT,
l1StartTime: START_TIME,
skipProtocolContracts: true,
assumeProvenThrough,
});

await (aztecNode as any).stop();
Expand Down Expand Up @@ -427,6 +444,7 @@ describe('e2e_synching', () => {
while ((await cheatCodes.eth.timestamp()) < targetTime) {
await cheatCodes.eth.mine();
}
// If it breaks here, first place you should look is the pruning.
await publisher.proposeL2Block(block);
}

Expand All @@ -436,32 +454,45 @@ describe('e2e_synching', () => {
};

describe('replay history and then do a fresh sync', () => {
it.each(variants)('vanilla - %s', async (variantDef: VariantDefinition) => {
await testTheVariant(
new TestVariant(variantDef),
async (opts: Partial<EndToEndContext>, variant: TestVariant) => {
// All the blocks have been "re-played" and we are now to simply get a new node up to speed
const timer = new Timer();
const freshNode = await AztecNodeService.createAndSync(
{ ...opts.config!, disableSequencer: true, disableValidator: true },
new NoopTelemetryClient(),
);
const syncTime = timer.s();
it.each(variants)(
'vanilla - %s',
async (variantDef: VariantDefinition) => {
// @note If the `RUN_THE_BIG_ONE` flag is not set, we DO NOT run it.
if (!RUN_THE_BIG_ONE && variantDef.blockCount === 1000) {
return;
}

const blockNumber = await freshNode.getBlockNumber();
await testTheVariant(
new TestVariant(variantDef),
async (opts: Partial<EndToEndContext>, variant: TestVariant) => {
// All the blocks have been "re-played" and we are now to simply get a new node up to speed
const timer = new Timer();
const freshNode = await AztecNodeService.createAndSync(
{ ...opts.config!, disableSequencer: true, disableValidator: true },
new NoopTelemetryClient(),
);
const syncTime = timer.s();

opts.logger!.info(
`Stats: ${variant.description()}: ${JSON.stringify({
numberOfBlocks: blockNumber,
syncTime,
})}`,
);
},
);
});
const blockNumber = await freshNode.getBlockNumber();

opts.logger!.info(
`Stats: ${variant.description()}: ${JSON.stringify({
numberOfBlocks: blockNumber,
syncTime,
})}`,
);

await freshNode.stop();
},
);
},
RUN_THE_BIG_ONE ? 600_000 : 300_000,
);
});

describe('a wild prune appears', () => {
const ASSUME_PROVEN_THROUGH = 0;

it('archiver following catches reorg as it occur and deletes blocks', async () => {
if (AZTEC_GENERATE_TEST_DATA) {
return;
Expand Down Expand Up @@ -557,7 +588,10 @@ describe('e2e_synching', () => {
[LogType.NOTEENCRYPTED, LogType.ENCRYPTED, LogType.UNENCRYPTED].forEach(async t => {
expect(await archiver.getLogs(blockTip.number, 1, t)).toEqual([]);
});

await archiver.stop();
},
ASSUME_PROVEN_THROUGH,
);
});

Expand Down Expand Up @@ -621,6 +655,7 @@ describe('e2e_synching', () => {
await aztecNode.stop();
await watcher.stop();
},
ASSUME_PROVEN_THROUGH,
);
});

Expand Down Expand Up @@ -679,6 +714,7 @@ describe('e2e_synching', () => {
await aztecNode.stop();
await watcher.stop();
},
ASSUME_PROVEN_THROUGH,
);
});
});
Expand Down
1,012 changes: 1,012 additions & 0 deletions yarn-project/end-to-end/src/fixtures/synching_blocks/1000_4_1/blocks.json
View file
Open in desktop

Large diffs are not rendered by default.

30 changes: 15 additions & 15 deletions yarn-project/end-to-end/src/fixtures/synching_blocks/10_36_0/blocks.json
View file
Open in desktop

Large diffs are not rendered by default.

36 changes: 18 additions & 18 deletions yarn-project/end-to-end/src/fixtures/synching_blocks/10_36_1/blocks.json
View file
Open in desktop

Large diffs are not rendered by default.

34 changes: 17 additions & 17 deletions yarn-project/end-to-end/src/fixtures/synching_blocks/10_36_2/blocks.json
View file
Open in desktop

Large diffs are not rendered by default.

32 changes: 16 additions & 16 deletions yarn-project/end-to-end/src/fixtures/synching_blocks/10_9_3/blocks.json
View file
Open in desktop

Large diffs are not rendered by default.

12 changes: 11 additions & 1 deletion yarn-project/end-to-end/src/fixtures/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,10 @@ export type SetupOptions = {
l1BlockTime?: number;
/** Anvil Start time */
l1StartTime?: number;
/** The anvil time where we should at the earliest be seeing L2 blocks */
l2StartTime?: number;
/** How far we should assume proven */
assumeProvenThrough?: number;
Comment on lines +319 to +320
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hate seeing this guy making its way across more parts of the codebase, but let's endure it for now

} & Partial<AztecNodeConfig>;

/** Context for an end-to-end test as returned by the `setup` function */
Expand Down Expand Up @@ -420,12 +424,18 @@ export async function setup(
config.l1RpcUrl,
publisherHdAccount!,
logger,
{ salt: opts.salt, initialValidators: opts.initialValidators },
{ salt: opts.salt, initialValidators: opts.initialValidators, assumeProvenThrough: opts.assumeProvenThrough },
chain,
));

config.l1Contracts = deployL1ContractsValues.l1ContractAddresses;

if (opts.l2StartTime) {
// This should only be used in synching test or when you need to have a stable
// timestamp for the first l2 block.
await ethCheatCodes.warp(opts.l2StartTime);
}

const watcher = new AnvilTestWatcher(
new EthCheatCodes(config.l1RpcUrl),
deployL1ContractsValues.l1ContractAddresses.rollupAddress,
Expand Down
Loading