Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

State performance tests #1134

Closed
Voxelot opened this issue Apr 17, 2023 · 1 comment · Fixed by #1230
Closed

State performance tests #1134

Voxelot opened this issue Apr 17, 2023 · 1 comment · Fixed by #1230
Assignees
@bvrooman

Comments

@Voxelot
Copy link
Member

Voxelot commented Apr 17, 2023 •
edited by xgreenx
Loading

Add performance tests for contracts with lots of state and balances

  • Bench insertion of 1 key/value pair into the RocksDB table that contains [0, 1_000, 100_000, 10_000_000] entries of different contracts.
  • Bench insertion of 1 key/value pair into the RocksDB table that contains [0, 1_000, 100_000, 10_000_000] entries of same contracts.
@bvrooman bvrooman mentioned this issue Jun 22, 2023
Merged
@bvrooman
Copy link
Contributor

bvrooman commented Jun 23, 2023 •
edited
Loading

I'm currently benchmarking with these configurations, but going forward, I propose that we reduce the benchmark with 10,000,000 entries to 1,000,000, or remove it entirely, because the setup takes hours on my M1 machine.

@bvrooman bvrooman mentioned this issue Jun 28, 2023
Merged
bvrooman pushed a commit that referenced this issue Jun 29, 2023
test: State benchmarks (#1226)
633de7c 
Related issues:
- Ref #1134

These benches demonstrate the performance of:

- Inserting 1 key/value pair into the RocksDB table that contains `n`
state entries for a single contracts. `n = [0, 1, 10, 100, 1,000,
10,000, 100,000, 1,000,000]`
- Inserting 1 key/value pair into the RocksDB table that contains 1
state entries with `n` existing contracts. `n = [0, 1, 10, 100, 1,000,
10,000, 100,000, 1,000,000]`

Bench results:
```
state single contract/insert state with 0 preexisting entries
                        time:   [11.991 µs 12.132 µs 12.310 µs]
                        change: [-4.7172% -1.6523% +1.0884%] (p = 0.29 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe
state single contract/insert state with 1 preexisting entries
                        time:   [25.009 µs 28.694 µs 32.962 µs]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 10 preexisting entries
                        time:   [39.002 µs 41.617 µs 44.324 µs]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
state single contract/insert state with 100 preexisting entries
                        time:   [60.968 µs 63.985 µs 67.259 µs]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 1,000 preexisting entries
                        time:   [87.465 µs 91.207 µs 95.394 µs]
                        change: [+0.4239% +4.5684% +8.9542%] (p = 0.04 < 0.05)
                        Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
  6 (6.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 10,000 preexisting entries
                        time:   [117.22 µs 121.38 µs 125.93 µs]
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild
state single contract/insert state with 100,000 preexisting entries
                        time:   [145.05 µs 148.30 µs 151.60 µs]
                        change: [-3.4471% -0.8145% +1.9661%] (p = 0.56 > 0.05)
                        No change in performance detected.
state single contract/insert state with 1,000,000 preexisting entries
                        time:   [186.36 µs 190.05 µs 193.80 µs]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

state multiple contracts/insert state with 0 preexisting entries
                        time:   [11.957 µs 12.080 µs 12.237 µs]
                        change: [-19.431% -7.6642% +0.8636%] (p = 0.27 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe
state multiple contracts/insert state with 1 preexisting entries
                        time:   [11.872 µs 12.008 µs 12.199 µs]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 10 preexisting entries
                        time:   [12.189 µs 12.252 µs 12.320 µs]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 100 preexisting entries
                        time:   [12.538 µs 12.691 µs 12.867 µs]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 1,000 preexisting entries
                        time:   [13.041 µs 13.278 µs 13.640 µs]
                        change: [-1.9317% +1.1895% +4.0288%] (p = 0.46 > 0.05)
                        No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe
state multiple contracts/insert state with 10,000 preexisting entries
                        time:   [13.350 µs 13.529 µs 13.729 µs]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
state multiple contracts/insert state with 100,000 preexisting entries
                        time:   [13.973 µs 14.312 µs 14.719 µs]
                        change: [+4.3928% +9.0688% +14.995%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe
state multiple contracts/insert state with 1,000,000 preexisting entries
                        time:   [13.891 µs 14.096 µs 14.314 µs]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
```

**Conclusion:**

For inserting `n` state entries for the same contract, the performance
appears to be logarithmic with `n`.

For inserting 1 state entry for a contract, with `n` existing contracts,
the performance appears to be fairly constant, i.e., the number of
existing contracts does not have an impact on the performance of
inserting state for a contract.

Note that these benchmarks were taken after the change to use RocksDB in
benches (#1227).
bvrooman pushed a commit that referenced this issue Jul 4, 2023
test: Benchmark contract state insertions with DB vs. DB transactions (…
45982da 
…#1230)

Related issues:
- Closes #1134 

This PR adds additional benches to compare the performance of inserting
state when the existing data is 1) written to the database, and 2)
stored in-memory in a transaction. The results demonstrate that when
preexisting data is held in a transaction, inserting state has constant
complexity and is generally faster, and performance degradation comes
from the logarithmic complexity of Merkle tree insertion.

```
insert state single contract database/insert state with 0 preexisting entries
                        time:   [3.0283 µs 3.0349 µs 3.0417 µs]
insert state single contract database/insert state with 1 preexisting entry
                        time:   [6.4704 µs 6.9960 µs 7.6233 µs]
insert state single contract database/insert state with 10 preexisting entries
                        time:   [15.750 µs 16.668 µs 17.636 µs]
insert state single contract database/insert state with 100 preexisting entries
                        time:   [29.102 µs 30.128 µs 31.204 µs]
insert state single contract database/insert state with 1,000 preexisting entries
                        time:   [45.264 µs 46.447 µs 47.666 µs]
insert state single contract database/insert state with 10,000 preexisting entries
                        time:   [63.839 µs 65.277 µs 66.784 µs]
insert state single contract database/insert state with 100,000 preexisting entries
                        time:   [85.863 µs 87.430 µs 89.119 µs]

insert state single contract transaction/insert state with 0 preexisting entries
                        time:   [2.4829 µs 2.8342 µs 3.4867 µs]
insert state single contract transaction/insert state with 1 preexisting entry
                        time:   [5.4261 µs 5.8455 µs 6.2946 µs]
insert state single contract transaction/insert state with 10 preexisting entries
                        time:   [11.104 µs 11.721 µs 12.352 µs]
insert state single contract transaction/insert state with 100 preexisting entries
                        time:   [20.425 µs 21.213 µs 21.987 µs]
insert state single contract transaction/insert state with 1,000 preexisting entries
                        time:   [28.998 µs 29.852 µs 30.714 µs]
insert state single contract transaction/insert state with 10,000 preexisting entries
                        time:   [38.645 µs 39.401 µs 40.206 µs]
insert state single contract transaction/insert state with 100,000 preexisting entries
                        time:   [49.969 µs 50.776 µs 51.611 µs]

insert state multiple contracts database/insert state with 0 preexisting entries
                        time:   [2.7846 µs 2.7901 µs 2.7962 µs]
insert state multiple contracts database/insert state with 1 preexisting entry
                        time:   [2.9599 µs 2.9688 µs 2.9782 µs]
insert state multiple contracts database/insert state with 10 preexisting entries
                        time:   [3.1822 µs 3.2080 µs 3.2350 µs]
insert state multiple contracts database/insert state with 100 preexisting entries
                        time:   [3.7841 µs 3.9514 µs 4.1838 µs]
insert state multiple contracts database/insert state with 1,000 preexisting entries
                        time:   [4.1301 µs 4.3894 µs 4.7190 µs]
insert state multiple contracts database/insert state with 10,000 preexisting entries
                        time:   [4.4221 µs 4.5124 µs 4.6034 µs]
insert state multiple contracts database/insert state with 100,000 preexisting entries
                        time:   [4.8611 µs 4.9762 µs 5.0904 µs]

insert state multiple contracts transaction/insert state with 0 preexisting entries
                        time:   [2.8071 µs 2.8121 µs 2.8175 µs]
insert state multiple contracts transaction/insert state with 1 preexisting entry
                        time:   [2.9778 µs 2.9842 µs 2.9912 µs]
insert state multiple contracts transaction/insert state with 10 preexisting entries
                        time:   [2.9502 µs 2.9587 µs 2.9677 µs]
insert state multiple contracts transaction/insert state with 100 preexisting entries
                        time:   [2.9706 µs 2.9779 µs 2.9854 µs]
insert state multiple contracts transaction/insert state with 1,000 preexisting entries
                        time:   [2.9824 µs 2.9910 µs 2.9999 µs]
insert state multiple contracts transaction/insert state with 10,000 preexisting entries
                        time:   [2.9799 µs 2.9887 µs 2.9985 µs]
insert state multiple contracts transaction/insert state with 100,000 preexisting entries
                        time:   [2.9731 µs 2.9846 µs 2.9963 µs]
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
@crypto523
test: State benchmarks (#1226)
85e0471 
Related issues:
- Ref FuelLabs/fuel-core#1134

These benches demonstrate the performance of:

- Inserting 1 key/value pair into the RocksDB table that contains `n`
state entries for a single contracts. `n = [0, 1, 10, 100, 1,000,
10,000, 100,000, 1,000,000]`
- Inserting 1 key/value pair into the RocksDB table that contains 1
state entries with `n` existing contracts. `n = [0, 1, 10, 100, 1,000,
10,000, 100,000, 1,000,000]`

Bench results:
```
state single contract/insert state with 0 preexisting entries
                        time:   [11.991 µs 12.132 µs 12.310 µs]
                        change: [-4.7172% -1.6523% +1.0884%] (p = 0.29 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe
state single contract/insert state with 1 preexisting entries
                        time:   [25.009 µs 28.694 µs 32.962 µs]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 10 preexisting entries
                        time:   [39.002 µs 41.617 µs 44.324 µs]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
state single contract/insert state with 100 preexisting entries
                        time:   [60.968 µs 63.985 µs 67.259 µs]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 1,000 preexisting entries
                        time:   [87.465 µs 91.207 µs 95.394 µs]
                        change: [+0.4239% +4.5684% +8.9542%] (p = 0.04 < 0.05)
                        Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
  6 (6.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 10,000 preexisting entries
                        time:   [117.22 µs 121.38 µs 125.93 µs]
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild
state single contract/insert state with 100,000 preexisting entries
                        time:   [145.05 µs 148.30 µs 151.60 µs]
                        change: [-3.4471% -0.8145% +1.9661%] (p = 0.56 > 0.05)
                        No change in performance detected.
state single contract/insert state with 1,000,000 preexisting entries
                        time:   [186.36 µs 190.05 µs 193.80 µs]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

state multiple contracts/insert state with 0 preexisting entries
                        time:   [11.957 µs 12.080 µs 12.237 µs]
                        change: [-19.431% -7.6642% +0.8636%] (p = 0.27 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe
state multiple contracts/insert state with 1 preexisting entries
                        time:   [11.872 µs 12.008 µs 12.199 µs]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 10 preexisting entries
                        time:   [12.189 µs 12.252 µs 12.320 µs]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 100 preexisting entries
                        time:   [12.538 µs 12.691 µs 12.867 µs]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 1,000 preexisting entries
                        time:   [13.041 µs 13.278 µs 13.640 µs]
                        change: [-1.9317% +1.1895% +4.0288%] (p = 0.46 > 0.05)
                        No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe
state multiple contracts/insert state with 10,000 preexisting entries
                        time:   [13.350 µs 13.529 µs 13.729 µs]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
state multiple contracts/insert state with 100,000 preexisting entries
                        time:   [13.973 µs 14.312 µs 14.719 µs]
                        change: [+4.3928% +9.0688% +14.995%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe
state multiple contracts/insert state with 1,000,000 preexisting entries
                        time:   [13.891 µs 14.096 µs 14.314 µs]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
```

**Conclusion:**

For inserting `n` state entries for the same contract, the performance
appears to be logarithmic with `n`.

For inserting 1 state entry for a contract, with `n` existing contracts,
the performance appears to be fairly constant, i.e., the number of
existing contracts does not have an impact on the performance of
inserting state for a contract.

Note that these benchmarks were taken after the change to use RocksDB in
benches (FuelLabs/fuel-core#1227).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bvrooman bvrooman

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: Benchmark contract state insertions with DB vs. DB transactions FuelLabs/fuel-core
2 participants
@Voxelot @bvrooman