Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[crypto+move] add support for Blake2b-256 in Move #5436

Merged
merged 20 commits into from
Dec 20, 2022

Conversation

blasrodri
Copy link
Contributor

@blasrodri blasrodri commented Nov 3, 2022

Taken a lot of inspiration from #4181 (comment)

Description

Add Blake2b-256 hashing in Move, to support folks writing bridge contracts to other platforms, as per conversations with @JoshLind and @alinush.

  • it is feature-gated
  • it has (reasonable) gas costs for the new hash functions

Test Plan

This PR merely exports an existing hash function crate (i.e.: blake2_rfc), which are is believed to be trustworthy implementation as it's used in Substrate (https://crates.io/crates/sp-core-hashing/4.0.0/dependencies). Therefore, we only made sure the native implementation give the expected hash values for b"", b"testing" and b"testing again".


This change is Reviewable

@alinush
Copy link
Contributor

alinush commented Nov 4, 2022

Thank you for the PR @blasrodri!

There are two flavors of Blake: Blake2b, Blake2s (see https://www.blake2.net/).

Can you please update the PR title and all of the code (e.g., function names, gas parameter names, comments, etc.) to clearly indicate which one you are implementing?

@alinush
Copy link
Contributor

alinush commented Nov 4, 2022

Thank you for the update @blasrodri! Can you include in the PR's description the performance impact of going from blake2 crate to blake2-rfc?

@blasrodri
Copy link
Contributor Author

Thank you for the update @blasrodri! Can you include in the PR's description the performance impact of going from blake2 crate to blake2-rfc?

Oh it's just because i checked the Substrate implemention and it uses blake2_rfc. But ill do a quick benchmark, and post the results. So that we can pick the better one ☺️

@blasrodri
Copy link
Contributor Author

blake2 seems to be faster than blake2-rfc

use blake2::{
    digest::{Update, VariableOutput},
    Blake2bVar,
};
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn blake2_blake2b_256(data: &[u8]) -> Vec<u8> {
    // Blake2bVarCore::new_with_params(salt, persona, key_size, output_size)
    let mut hasher = Blake2bVar::new(32).unwrap();
    hasher.update(data);
    let mut buf = vec![0u8; 32];
    hasher.finalize_variable(&mut buf).unwrap();
    buf
}

fn blake2_rfc_blake2b_256(data: &[u8]) -> Vec<u8> {
    blake2_blake2b_256(data).to_vec()
}

fn criterion_benchmark(c: &mut Criterion) {
    assert_eq!(
        blake2_blake2b_256(b"testing"),
        blake2_rfc_blake2b_256(b"testing")
    );
    c.bench_function("blake2_blake2b_256", |b| {
        b.iter(|| blake2_blake2b_256(black_box(b"testing")))
    });
    c.bench_function("blake2_rfc_blake2b_256", |b| {
        b.iter(|| blake2_rfc_blake2b_256(black_box(b"testing")))
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);
blas@f1:~/foss/blak2-benchmarks$ cargo bench
   Compiling blak2-benchmarks v0.1.0 (/home/blas/foss/blak2-benchmarks)
    Finished bench [optimized] target(s) in 0.79s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [123.94 ns 124.07 ns 124.21 ns]
                        change: [-0.2375% +0.0861% +0.3560%] (p = 0.59 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

blake2_rfc_blake2b_256  time:   [142.25 ns 142.37 ns 142.51 ns]
                        change: [-0.7492% -0.6241% -0.4932%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

blas@f1:~/foss/blak2-benchmarks$ cargo bench
    Finished bench [optimized] target(s) in 0.02s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [122.38 ns 122.51 ns 122.64 ns]
                        change: [-1.5252% -1.3730% -1.2095%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

blake2_rfc_blake2b_256  time:   [139.35 ns 139.51 ns 139.71 ns]
                        change: [-1.8661% -1.6701% -1.4750%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

@alinush
Copy link
Contributor

alinush commented Nov 5, 2022

blake2 seems to be faster than blake2-rfc

use blake2::{
    digest::{Update, VariableOutput},
    Blake2bVar,
};
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn blake2_blake2b_256(data: &[u8]) -> Vec<u8> {
    // Blake2bVarCore::new_with_params(salt, persona, key_size, output_size)
    let mut hasher = Blake2bVar::new(32).unwrap();
    hasher.update(data);
    let mut buf = vec![0u8; 32];
    hasher.finalize_variable(&mut buf).unwrap();
    buf
}

fn blake2_rfc_blake2b_256(data: &[u8]) -> Vec<u8> {
    blake2_blake2b_256(data).to_vec()
}

fn criterion_benchmark(c: &mut Criterion) {
    assert_eq!(
        blake2_blake2b_256(b"testing"),
        blake2_rfc_blake2b_256(b"testing")
    );
    c.bench_function("blake2_blake2b_256", |b| {
        b.iter(|| blake2_blake2b_256(black_box(b"testing")))
    });
    c.bench_function("blake2_rfc_blake2b_256", |b| {
        b.iter(|| blake2_rfc_blake2b_256(black_box(b"testing")))
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);
blas@f1:~/foss/blak2-benchmarks$ cargo bench
   Compiling blak2-benchmarks v0.1.0 (/home/blas/foss/blak2-benchmarks)
    Finished bench [optimized] target(s) in 0.79s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [123.94 ns 124.07 ns 124.21 ns]
                        change: [-0.2375% +0.0861% +0.3560%] (p = 0.59 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

blake2_rfc_blake2b_256  time:   [142.25 ns 142.37 ns 142.51 ns]
                        change: [-0.7492% -0.6241% -0.4932%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

blas@f1:~/foss/blak2-benchmarks$ cargo bench
    Finished bench [optimized] target(s) in 0.02s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [122.38 ns 122.51 ns 122.64 ns]
                        change: [-1.5252% -1.3730% -1.2095%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

blake2_rfc_blake2b_256  time:   [139.35 ns 139.51 ns 139.71 ns]
                        change: [-1.8661% -1.6701% -1.4750%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Thank you @blasrodri! Let's stick to blake2 in that case.

Could you please add your benchmarks to this hash.rs file in crates/aptos-crypto/benches/?

This will help me estimate gas costs more accurately.

@blasrodri
Copy link
Contributor Author

blake2 seems to be faster than blake2-rfc

use blake2::{
    digest::{Update, VariableOutput},
    Blake2bVar,
};
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn blake2_blake2b_256(data: &[u8]) -> Vec<u8> {
    // Blake2bVarCore::new_with_params(salt, persona, key_size, output_size)
    let mut hasher = Blake2bVar::new(32).unwrap();
    hasher.update(data);
    let mut buf = vec![0u8; 32];
    hasher.finalize_variable(&mut buf).unwrap();
    buf
}

fn blake2_rfc_blake2b_256(data: &[u8]) -> Vec<u8> {
    blake2_blake2b_256(data).to_vec()
}

fn criterion_benchmark(c: &mut Criterion) {
    assert_eq!(
        blake2_blake2b_256(b"testing"),
        blake2_rfc_blake2b_256(b"testing")
    );
    c.bench_function("blake2_blake2b_256", |b| {
        b.iter(|| blake2_blake2b_256(black_box(b"testing")))
    });
    c.bench_function("blake2_rfc_blake2b_256", |b| {
        b.iter(|| blake2_rfc_blake2b_256(black_box(b"testing")))
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);
blas@f1:~/foss/blak2-benchmarks$ cargo bench
   Compiling blak2-benchmarks v0.1.0 (/home/blas/foss/blak2-benchmarks)
    Finished bench [optimized] target(s) in 0.79s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [123.94 ns 124.07 ns 124.21 ns]
                        change: [-0.2375% +0.0861% +0.3560%] (p = 0.59 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

blake2_rfc_blake2b_256  time:   [142.25 ns 142.37 ns 142.51 ns]
                        change: [-0.7492% -0.6241% -0.4932%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

blas@f1:~/foss/blak2-benchmarks$ cargo bench
    Finished bench [optimized] target(s) in 0.02s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [122.38 ns 122.51 ns 122.64 ns]
                        change: [-1.5252% -1.3730% -1.2095%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

blake2_rfc_blake2b_256  time:   [139.35 ns 139.51 ns 139.71 ns]
                        change: [-1.8661% -1.6701% -1.4750%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Thank you @blasrodri! Let's stick to blake2 in that case.

Could you please add your benchmarks to this hash.rs file in crates/aptos-crypto/benches/?

This will help me estimate gas costs more accurately.

I made a mistake on the previous script. Now, I've added the two implementations on the hash.rs bench, and these
are the results:

hash/BLAKE2B-256-blake2-crate/0                                                                             
                        time:   [156.02 ns 156.21 ns 156.39 ns]
                        thrpt:  [0.0000   B/s 0.0000   B/s 0.0000   B/s]
                 change:
                        time:   [+0.3640% +0.5418% +0.7898%] (p = 0.00 < 0.05)
                        thrpt:  [-0.7836% -0.5389% -0.3627%]
                        Change within noise threshold.
Found 32 outliers among 1000 measurements (3.20%)
  16 (1.60%) high mild
  16 (1.60%) high severe
hash/BLAKE2B-256-blake2-rfc-crate/0                                                                             
                        time:   [125.81 ns 125.89 ns 125.97 ns]
                        thrpt:  [0.0000   B/s 0.0000   B/s 0.0000   B/s]
                 change:
                        time:   [+0.2890% +0.4097% +0.5219%] (p = 0.00 < 0.05)
                        thrpt:  [-0.5192% -0.4080% -0.2882%]
                        Change within noise threshold.

hash/BLAKE2B-256-blake2-crate/1                                                                             
                        time:   [171.15 ns 171.28 ns 171.40 ns]
                        thrpt:  [5.5640 MiB/s 5.5681 MiB/s 5.5720 MiB/s]
                 change:
                        time:   [+10.641% +10.729% +10.818%] (p = 0.00 < 0.05)
                        thrpt:  [-9.7620% -9.6898% -9.6175%]
                        Performance has regressed.
Found 26 outliers among 1000 measurements (2.60%)
  22 (2.20%) high mild
  4 (0.40%) high severe
hash/BLAKE2B-256-blake2-rfc-crate/1                                                                             
                        time:   [134.56 ns 134.72 ns 134.90 ns]
                        thrpt:  [7.0693 MiB/s 7.0788 MiB/s 7.0874 MiB/s]
                 change:
                        time:   [-1.2537% -1.0534% -0.8347%] (p = 0.00 < 0.05)
                        thrpt:  [+0.8417% +1.0646% +1.2696%]
                        Change within noise threshold.
Found 11 outliers among 1000 measurements (1.10%)
  8 (0.80%) high mild
  3 (0.30%) high severe

@alinush
Copy link
Contributor

alinush commented Nov 6, 2022

Sweet! I forgot to ask the most obvious question: are these numbers for x86_64?

@blasrodri
Copy link
Contributor Author

blasrodri commented Nov 6, 2022

Sweet! I forgot to ask the most obvious question: are these numbers for x86_64?

Yes ^_^

blas@f1:~$ uname -m
x86_64

@alinush alinush changed the title [crypto+move] added support for Blake2 256 in Move [crypto+move] added support for Blake2b-256 in Move Nov 7, 2022
@alinush alinush changed the title [crypto+move] added support for Blake2b-256 in Move [crypto+move] add support for Blake2b-256 in Move Nov 7, 2022
/// Whether the new BLAKE2B-256 hash function natives is enabled.
/// This is needed because of the introduction of new native functions.
/// Lifetime: transient
const BLAKE2B_256_NATIVE: u64 = 5;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a heads up that this might need to be further bumped up given recent changes. We'll have to see.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see no issues here.

@alinush
Copy link
Contributor

alinush commented Nov 18, 2022

One more request, since you are adding new gas parameters, you will need to bump up the LATEST_GAS_FEATURE_VERSION in aptos-move/aptos-gas/src/gas_meter.rs (e.g., see this).

Right, @vgao1996? (Sorry, I keep forgetting this.)

@alinush
Copy link
Contributor

alinush commented Nov 18, 2022

Sweet! I forgot to ask the most obvious question: are these numbers for x86_64?

Yes ^_^

blas@f1:~$ uname -m
x86_64

Another request: could you re-run benchmarks both for SHA2-256 and your newly-added Blake2b function and post the results here? This will allow us to calibrate the gas costs a little better.

The SHA2-256 ones are also in aptos-crypto/benches/hash.rs

@alinush alinush requested a review from zjma November 18, 2022 19:52
@blasrodri
Copy link
Contributor Author

Sweet! I forgot to ask the most obvious question: are these numbers for x86_64?

Yes ^_^

blas@f1:~$ uname -m
x86_64

Another request: could you re-run benchmarks both for SHA2-256 and your newly-added Blake2b function and post the results here? This will allow us to calibrate the gas costs a little better.

The SHA2-256 ones are also in aptos-crypto/benches/hash.rs

Sure! Will do it today

@blasrodri
Copy link
Contributor Author

SHA2

hash/SHA2-256/0         time:   [71.395 ns 71.552 ns 71.720 ns]                             
                        thrpt:  [0.0000   B/s 0.0000   B/s 0.0000   B/s]
                 change:
                        time:   [+4.6839% +4.9618% +5.2726%] (p = 0.00 < 0.05)
                        thrpt:  [-5.0086% -4.7273% -4.4743%]
                        Performance has regressed.
Found 53 outliers among 1000 measurements (5.30%)
  19 (1.90%) high mild
  34 (3.40%) high severe
hash/SHA2-256/8         time:   [72.274 ns 72.323 ns 72.374 ns]                             
                        thrpt:  [105.42 MiB/s 105.49 MiB/s 105.56 MiB/s]
                 change:
                        time:   [+0.9191% +1.1247% +1.3492%] (p = 0.00 < 0.05)
                        thrpt:  [-1.3312% -1.1122% -0.9108%]
                        Change within noise threshold.
Found 4 outliers among 1000 measurements (0.40%)
  2 (0.20%) high mild
  2 (0.20%) high severe
hash/SHA2-256/16        time:   [67.068 ns 67.177 ns 67.301 ns]                              
                        thrpt:  [226.72 MiB/s 227.14 MiB/s 227.51 MiB/s]
                 change:
                        time:   [-5.5021% -5.3377% -5.1669%] (p = 0.00 < 0.05)
                        thrpt:  [+5.4484% +5.6387% +5.8225%]
                        Performance has improved.
Found 94 outliers among 1000 measurements (9.40%)


BLAKE2B

hash/BLAKE2B-256-blake2-crate/2                                                                             
                        time:   [169.94 ns 170.12 ns 170.33 ns]
                        thrpt:  [11.198 MiB/s 11.212 MiB/s 11.223 MiB/s]
                 change:
                        time:   [-2.9107% -2.7912% -2.6447%] (p = 0.00 < 0.05)
                        thrpt:  [+2.7166% +2.8713% +2.9980%]
                        Performance has improved.
Found 23 outliers among 1000 measurements (2.30%)
  10 (1.00%) low mild
  7 (0.70%) high mild
  6 (0.60%) high severe

hash/BLAKE2B-256-blake2-rfc-crate/8                                                                             
                        time:   [132.83 ns 132.88 ns 132.94 ns]
                        thrpt:  [57.392 MiB/s 57.415 MiB/s 57.439 MiB/s]
                 change:
                        time:   [-4.1867% -3.7555% -3.3995%] (p = 0.00 < 0.05)
                        thrpt:  [+3.5191% +3.9020% +4.3696%]
                        Performance has improved.
Found 7 outliers among 1000 measurements (0.70%)

aptos-move/framework/Cargo.toml Outdated Show resolved Hide resolved
Signed-off-by: Blas Rodriguez Irizar <[email protected]>
Signed-off-by: Blas Rodriguez Irizar <[email protected]>
@davidiw davidiw enabled auto-merge (squash) December 18, 2022 04:48
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@davidiw davidiw disabled auto-merge December 18, 2022 05:35
@davidiw
Copy link
Contributor

davidiw commented Dec 18, 2022

This branch has conflicts that must be resolved

Please rebase and fix on your end. I can potentially help, but I think there might be some gnarly changes.

@davidiw davidiw closed this Dec 18, 2022
@davidiw davidiw reopened this Dec 18, 2022
@alinush
Copy link
Contributor

alinush commented Dec 19, 2022

I'm coordinating with Blas on re-basing and merging. No worries @davidiw.

@alinush alinush enabled auto-merge (squash) December 20, 2022 10:14
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

✅ Forge suite land_blocking success on 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c

performance benchmark with full nodes : 6665 TPS, 5953 ms latency, 9000 ms p99 latency,no expired txns
Test Ok

@github-actions
Copy link
Contributor

✅ Forge suite compat success on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7284 TPS, 5331 ms latency, 7000 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c
compatibility::simple-validator-upgrade::single-validator-upgrade : 4545 TPS, 8820 ms latency, 12600 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c
compatibility::simple-validator-upgrade::half-validator-upgrade : 4296 TPS, 9750 ms latency, 12800 ms p99 latency,no expired txns
4. upgrading second batch to new version: 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c
compatibility::simple-validator-upgrade::rest-validator-upgrade : 6189 TPS, 6201 ms latency, 9200 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c passed
Test Ok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants