[TS SDK] Add account transaction management #8854

0xmaayan · 2023-06-27T03:39:50Z

Description

based on #7987

This PR provides a layer for managing and submitting as many transactions from a single account at once.

AccountSequenceNumber class - a class that handles and manages an account sequence number.
TransactionWorker class - a class that provides workers to manage transactions, from generate through sign and submit to chain. It validates the transaction submission and monitors the submission success/failure.

In addition, provide an example demonstrates the transaction management layer and how one can implement it.

Test Plan

tests are passing
output from running the example against devnet using a wifi connection with download speed of 23.38 mbps and upload speed of 8.40 mbps

starting...
5 sender accounts and 5 recipient accounts created in 0.03299999237060547 seconds
5 sender accounts funded in 2.1549999713897705 seconds
5 sender account balances checked in 0.10400009155273438 seconds
sends 500 transactions to chain....
transactions sent in 59.35899996757507 seconds
transactions sent in 59.45599985122681 seconds
transactions executed in 59.46299982070923 seconds
transactions executed in 59.561999797821045 seconds
transactions sent in 59.56699991226196 seconds
transactions verified in  59.569000005722046  seconds
sender account 0xeb9f3a017bf41af5fc66e38debe5a395dd361f898ea8f37aa2ce202e3c51b874 final sequence number is 500
sender account 0x7d5606f584525c8033dd3726e5c1853fb3d6afd2efe7c348af71b00fba100dcb final sequence number is 500
sender account 0xc3a1ce16add2ddd50917dec2fa711d20d378ba5beb5a0e971c0bac2df12527a7 final sequence number is 500
sender account 0xa32ca88eeba3fa7a1513a642237b4acbe3380aff197142b8ec577dd851f6fc07 final sequence number is 500
sender account 0xae65fa420711b4e225df1caf9475bb8dc481ad3d863e5272d655f0757d737191 final sequence number is 500

ecosystem/typescript/sdk/examples/typescript/transactions_management.ts

ecosystem/typescript/sdk/src/transactions/account_sequence_number.ts

ecosystem/typescript/sdk/src/transactions/tests/account_sequence_number.test.ts

ecosystem/typescript/sdk/src/transactions/transaction_worker.ts

0xmigo

Looks good overall, some nits but good to land :)

davidiw

good start!

I wonder if it might be useful for me to host a session that goes over what I did in Python?

Some areas of focus in this PR:

Needs error handling, if there's any error, the service cannot recover.
Limited concurrency -- we should probably make more workers that just pull on the next available transaction and another worker that just processes transactions.
Logging would benefit from a cleanup
Timestamps to indicate decent throughput would be good too
Descriptions of each major components

I'd also be curious to understand how you landed on EventEmitter3 versus say worker threads and making it a js only concept. It might be useful to come up with a story around this and evaluate if it even makes sense to have a high throughput construct for browsers or just node.

ecosystem/typescript/sdk/examples/typescript/transactions_management.ts

ecosystem/typescript/sdk/src/transactions/transaction_worker.ts

davidiw

looking good

ecosystem/typescript/sdk/examples/typescript/transactions_management.ts

davidiw · 2023-07-18T13:12:49Z

ecosystem/typescript/sdk/examples/typescript/transactions_management.ts

+    const res = await Promise.all(waitFor);
+    console.log(`transactions verified`);
+    console.log(new Date().toTimeString());
+    for (const account in res) {


line numbers don't make sense, could you please update or hit me up with the lines that do this?

the reason I made this comment was below.. leaving comment there

ecosystem/typescript/sdk/examples/typescript/transactions_management.ts

ecosystem/typescript/sdk/src/transactions/account_sequence_number.ts

davidiw · 2023-07-18T13:18:13Z

ecosystem/typescript/sdk/src/transactions/account_sequence_number.ts

+  async synchronize(): Promise<void> {
+    if (this.lastUncommintedNumber === this.currentNumber) return;
+
+    while (this.lock) {


I'm not seeing how this connects. Considering that this is a public function, it is a bit dangerous to not get the lock here. Especially with calls into initialize..

ecosystem/typescript/sdk/src/transactions/transaction_worker.ts

davidiw

looking good

ecosystem/typescript/sdk/examples/typescript/transactions_management.ts

davidiw · 2023-07-21T13:18:30Z

ecosystem/typescript/sdk/examples/typescript/transactions_management.ts

+      console.log("sentFailed", data);
+    });
+
+    transactionWorker.on(TransactionWorkerEvents.TransactionExecuted, async (data) => {


so this is where you're confirming success and syncing?

technically we confirm it here https://github.com/aptos-labs/aptos-core/pull/8854/files#diff-52d3c5103be2ebfaf76511751a50a8fc3ce5f720a77ddd6fed4e74891074edb0R172 , if execution succeed/failed we emit a TransactionExecuted/TransactionExecutionFailed event that the client can listen to and do whatever they want.

davidiw · 2023-07-21T13:19:15Z

ecosystem/typescript/sdk/src/transactions/account_sequence_number.ts

+  async synchronize(): Promise<void> {
+    if (this.lastUncommintedNumber === this.currentNumber) return;
+
+    while (this.lock) {


ecosystem/typescript/sdk/src/transactions/async_queue.ts

davidiw

unblocking, but please wait for the lock in synchronize

davidiw · 2023-07-26T04:55:03Z

ecosystem/typescript/sdk/src/transactions/account_sequence_number.ts

+  async synchronize(): Promise<void> {
+    if (this.lastUncommintedNumber === this.currentNumber) return;
+
+    this.lock = true;


you can't get the lock without waiting for it, you could theoretically have multiple calls to synchronize... I don't think anything would really bad happen because of the excessive serial nature here, but....

transaction management layer add sequnce number tests transaction worker class add transaction worker and queue class add start and stop to transaction worker add txn worker tests write full flow test add account transactions management

github-actions · 2023-07-27T22:06:15Z

✅ Forge suite `compat` success on `aptos-node-v1.5.1` ==> `ebf3233569b4b6f7e5c0ffd36656f86321fe1a14`

Compatibility test results for aptos-node-v1.5.1 ==> ebf3233569b4b6f7e5c0ffd36656f86321fe1a14 (PR)
1. Check liveness of validators at old version: aptos-node-v1.5.1
compatibility::simple-validator-upgrade::liveness-check : committed: 4390 txn/s, latency: 7205 ms, (p50: 7800 ms, p90: 9800 ms, p99: 11800 ms), latency samples: 166820
2. Upgrading first Validator to new version: ebf3233569b4b6f7e5c0ffd36656f86321fe1a14
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1836 txn/s, latency: 15700 ms, (p50: 18600 ms, p90: 22100 ms, p99: 22600 ms), latency samples: 91840
3. Upgrading rest of first batch to new version: ebf3233569b4b6f7e5c0ffd36656f86321fe1a14
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1847 txn/s, latency: 15667 ms, (p50: 19000 ms, p90: 22000 ms, p99: 22300 ms), latency samples: 92360
4. upgrading second batch to new version: ebf3233569b4b6f7e5c0ffd36656f86321fe1a14
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2359 txn/s, latency: 9364 ms, (p50: 10200 ms, p90: 13300 ms, p99: 13800 ms), latency samples: 139200
5. check swarm health
Compatibility test for aptos-node-v1.5.1 ==> ebf3233569b4b6f7e5c0ffd36656f86321fe1a14 passed
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions · 2023-07-27T22:08:46Z

✅ Forge suite `realistic_env_max_load` success on `ebf3233569b4b6f7e5c0ffd36656f86321fe1a14`

two traffics test: inner traffic : committed: 6356 txn/s, latency: 6153 ms, (p50: 5700 ms, p90: 7800 ms, p99: 12300 ms), latency samples: 2758860
two traffics test : committed: 100 txn/s, latency: 3092 ms, (p50: 2800 ms, p90: 3700 ms, p99: 7100 ms), latency samples: 1760
Max round gap was 1 [limit 4] at version 1301731. Max no progress secs was 4.011074 [limit 10] at version 1301731.
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions · 2023-07-27T22:18:40Z

✅ Forge suite `framework_upgrade` success on `aptos-node-v1.5.1` ==> `ebf3233569b4b6f7e5c0ffd36656f86321fe1a14`

Compatibility test results for aptos-node-v1.5.1 ==> ebf3233569b4b6f7e5c0ffd36656f86321fe1a14 (PR)
Upgrade the nodes to version: ebf3233569b4b6f7e5c0ffd36656f86321fe1a14
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 4057 txn/s, latency: 7654 ms, (p50: 6900 ms, p90: 11700 ms, p99: 22900 ms), latency samples: 166360
5. check swarm health
Compatibility test for aptos-node-v1.5.1 ==> ebf3233569b4b6f7e5c0ffd36656f86321fe1a14 passed
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

* add account transaction management transaction management layer add sequnce number tests transaction worker class add transaction worker and queue class add start and stop to transaction worker add txn worker tests write full flow test add account transactions management * add method comments * address comments * implement async queue * events as enum * address feedback

0xmaayan requested a review from davidiw June 27, 2023 03:39

0xmaayan requested review from gregnazario, banool and 0xmigo as code owners June 27, 2023 03:39

0xmaayan force-pushed the transaction_management branch 3 times, most recently from a5f14ec to becfa82 Compare June 27, 2023 14:35

0xmaayan linked an issue Jun 27, 2023 that may be closed by this pull request

[TS SDK] transaction management layer #6881

Closed

0xmaayan requested a review from junkil-park July 5, 2023 16:25

0xmigo reviewed Jul 13, 2023

View reviewed changes

0xmigo approved these changes Jul 14, 2023

View reviewed changes

0xmaayan force-pushed the transaction_management branch 3 times, most recently from 303b340 to 4759193 Compare July 14, 2023 19:45

davidiw reviewed Jul 15, 2023

View reviewed changes

0xmaayan force-pushed the transaction_management branch 2 times, most recently from 30507bb to acc32e8 Compare July 17, 2023 17:36

davidiw reviewed Jul 18, 2023

View reviewed changes

sionescu requested a review from a team as a code owner July 18, 2023 17:17

0xmaayan force-pushed the transaction_management branch 5 times, most recently from ef8f9e8 to d8d1b36 Compare July 19, 2023 00:51

jillxuu reviewed Jul 20, 2023

View reviewed changes

ecosystem/typescript/sdk/src/transactions/transaction_worker.ts Outdated Show resolved Hide resolved

0xmaayan force-pushed the transaction_management branch 2 times, most recently from 37d2920 to 19d1543 Compare July 20, 2023 16:48

davidiw reviewed Jul 21, 2023

View reviewed changes

0xmaayan force-pushed the transaction_management branch 2 times, most recently from fb45c61 to e9083be Compare July 21, 2023 16:48

davidiw approved these changes Jul 26, 2023

View reviewed changes

0xmaayan added 5 commits July 27, 2023 15:21

add account transaction management

fb959bd

transaction management layer add sequnce number tests transaction worker class add transaction worker and queue class add start and stop to transaction worker add txn worker tests write full flow test add account transactions management

add method comments

58e1cbe

address comments

b7815ad

implement async queue

49cf5cf

events as enum

f64b9b8

0xmaayan force-pushed the transaction_management branch from e9083be to 2583fc8 Compare July 27, 2023 19:52

address feedback

ebf3233

0xmaayan force-pushed the transaction_management branch from 2583fc8 to ebf3233 Compare July 27, 2023 21:39

0xmaayan enabled auto-merge (squash) July 27, 2023 21:39

This comment has been minimized.

Sign in to view

0xmaayan merged commit ffdeefe into main Jul 28, 2023

0xmaayan deleted the transaction_management branch July 28, 2023 00:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TS SDK] Add account transaction management #8854

[TS SDK] Add account transaction management #8854

0xmaayan commented Jun 27, 2023 •

edited

Loading

0xmigo left a comment

davidiw left a comment

davidiw left a comment

davidiw Jul 18, 2023

davidiw Jul 18, 2023

davidiw left a comment

davidiw Jul 21, 2023

0xmaayan Jul 21, 2023

davidiw Jul 21, 2023

davidiw left a comment

davidiw Jul 26, 2023

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jul 27, 2023

github-actions bot commented Jul 27, 2023

github-actions bot commented Jul 27, 2023

[TS SDK] Add account transaction management #8854

[TS SDK] Add account transaction management #8854

Conversation

0xmaayan commented Jun 27, 2023 • edited Loading

Description

Test Plan

0xmigo left a comment

Choose a reason for hiding this comment

davidiw left a comment

Choose a reason for hiding this comment

davidiw left a comment

Choose a reason for hiding this comment

davidiw Jul 18, 2023

Choose a reason for hiding this comment

davidiw Jul 18, 2023

Choose a reason for hiding this comment

davidiw left a comment

Choose a reason for hiding this comment

davidiw Jul 21, 2023

Choose a reason for hiding this comment

0xmaayan Jul 21, 2023

Choose a reason for hiding this comment

davidiw Jul 21, 2023

Choose a reason for hiding this comment

davidiw left a comment

Choose a reason for hiding this comment

davidiw Jul 26, 2023

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jul 27, 2023

✅ Forge suite compat success on aptos-node-v1.5.1 ==> ebf3233569b4b6f7e5c0ffd36656f86321fe1a14

github-actions bot commented Jul 27, 2023

✅ Forge suite realistic_env_max_load success on ebf3233569b4b6f7e5c0ffd36656f86321fe1a14

github-actions bot commented Jul 27, 2023

✅ Forge suite framework_upgrade success on aptos-node-v1.5.1 ==> ebf3233569b4b6f7e5c0ffd36656f86321fe1a14

0xmaayan commented Jun 27, 2023 •

edited

Loading

✅ Forge suite `compat` success on `aptos-node-v1.5.1` ==> `ebf3233569b4b6f7e5c0ffd36656f86321fe1a14`

✅ Forge suite `realistic_env_max_load` success on `ebf3233569b4b6f7e5c0ffd36656f86321fe1a14`

✅ Forge suite `framework_upgrade` success on `aptos-node-v1.5.1` ==> `ebf3233569b4b6f7e5c0ffd36656f86321fe1a14`