[move][stdlib] Implement mem::swap native move call #14786

igor-aptos · 2024-09-27T23:01:34Z

Description

If we have a field that contains non-copyable type, it is impossible to change it, and get the old value back.

Adding two methods:

native mem::swap, that implements swap of contents of two mutable references
mem::replace, as a simple wrapper based on mem::swap

How Has This Been Tested?

provided unit tests

Type of Change

New feature

Which Components or Systems Does This Change Impact?

Move/Aptos Virtual Machine

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-09-27T23:01:38Z

⏱️ 1h 45m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
rust-move-unit-coverage	16m	🟩
rust-move-unit-coverage	11m	🟩
rust-move-unit-coverage	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-unit-coverage	10m	🟩
rust-move-tests	10m	🟥
rust-move-tests	9m	🟥
rust-cargo-deny	7m	🟩 🟩 🟩 🟩
check-dynamic-deps	4m	🟩 🟩 🟩 🟩
general-lints	3m	🟩 🟩 🟩 🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩
file_change_determinator	52s	🟩 🟩 🟩 🟩
permission-check	22s	🟩 🟩 🟩 🟩 🟩
file_change_determinator	20s	🟩 🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

codecov · 2024-09-27T23:12:00Z

Codecov Report

Attention: Patch coverage is 80.08658% with 46 lines in your changes missing coverage. Please review.

Project coverage is 60.1%. Comparing base (0e4f5df) to head (8b0d35b).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...party/move/move-vm/types/src/values/values_impl.rs	72.5%	37 Missing ⚠️
...ptos-move/framework/move-stdlib/src/natives/mem.rs	75.0%	9 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #14786    +/-   ##
========================================
  Coverage    60.1%    60.1%            
========================================
  Files         856      857     +1     
  Lines      211110   211341   +231     
========================================
+ Hits       126962   127147   +185     
- Misses      84148    84194    +46

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lightmark · 2024-10-03T19:14:44Z

What's this for?

igor-aptos · 2024-10-03T19:44:01Z

This is needed to be able to modify fields, when they hold a non-copyable non-dropable type.

For example, you cannot do this today:

struct A<T> {
   field: T,
}

public fun swap<T>(self: &mut A<T>, value: T): T {
   let old_value = self.field;  <-- doesn't compile, expects copy
   self.field = value;          <-- doesn't compile, expects drop
   old_value
}

https://aptos-org.slack.com/archives/C05V6JEKZQ9/p1727393314700309

vgao1996 · 2024-10-10T00:37:25Z

@wrwg I'm also wondering if we should just make this a bytecode instruction. Can probably do it later though.

igor-aptos

is there a way to unit-test call_native?

i.e. we are relying for a lot of tests in third_party/move/move-vm/types/src/values/value_tests.rs that are not failing checks internally, to fail checks when native call types get checked.

Otherwise I am not sure how to have that test coverage, as in end-to-end tests - such code would not compile.

igor-aptos · 2024-11-08T23:23:21Z

addressed all comments and added more tests

georgemitenkov

Pass 1: everything but the values

georgemitenkov · 2024-10-04T17:13:20Z

aptos-move/framework/move-stdlib/sources/mem.move

@@ -0,0 +1,27 @@
+/// Module with methods for safe memory manipulation.
+/// I.e. swapping/replacing non-copyable/non-droppable types.


nit: I.e. reads a bit weird, maybe For example? Also why break into 2 sentences?

I think the example is just redundant, removing. swap below has the docstring as well.

georgemitenkov · 2024-10-24T04:15:51Z

aptos-move/aptos-gas-schedule/src/ver.rs

@@ -69,7 +69,7 @@
 ///       global operations.
 /// - V1
 ///   - TBA
-pub const LATEST_GAS_FEATURE_VERSION: u64 = gas_feature_versions::RELEASE_V1_22;
+pub const LATEST_GAS_FEATURE_VERSION: u64 = gas_feature_versions::RELEASE_V1_23;


Please add documentation to new 1.23 in the comment above that new natives have been added

aptos-move/aptos-release-builder/src/components/feature_flags.rs

georgemitenkov · 2024-10-24T04:21:48Z

aptos-move/framework/move-stdlib/src/natives/mem.rs

+// Copyright © Aptos Foundation
+// SPDX-License-Identifier: Apache-2.0
+
+// Copyright (c) The Diem Core Contributors


I think this is wrong, multiple licenses and why we even have Diem here?

georgemitenkov · 2024-10-24T04:23:25Z

aptos-move/framework/move-stdlib/src/natives/mem.rs

+
+/// The feature is not enabled.
+/// (0xD is unavailable)
+pub const EFEATURE_NOT_ENABLED: u64 = 0x0D_0001;


Can we document this in Move code? Running into these errors is such a pain because you need to search through aptos-core hoping to get the right constant....

Also, don't we have errors::unavailable Rust helper already? I think it exists in aptos-types though...

types/src/on_chain_config/aptos_features.rs

georgemitenkov · 2024-10-24T04:28:40Z

types/src/on_chain_config/aptos_features.rs

@@ -317,6 +320,10 @@ impl Features {
        self.is_enabled(FeatureFlag::TRANSACTION_SIMULATION_ENHANCEMENT)
    }

+    pub fn is_native_memory_operations_enabled(&self) -> bool {


nit: are instead of is

everything is using is?

It just reads weird to me because it is plural, but maybe a personal taste

georgemitenkov · 2024-10-24T04:32:54Z

third_party/move/move-vm/types/src/values/values_impl.rs

+    fn swap_contents(&self, other: &Self) -> PartialVMResult<()> {
+        use Container::*;
+
+        // TODO: check if unique ownership?


If you add a todo, add a github handle or create a ticket to track this so someone else can have more context about it.

oh this was from Victor's change, missed it. removing, and adding comment.

georgemitenkov · 2024-11-12T17:07:09Z

aptos-move/framework/move-stdlib/sources/mem.move

+    /// Swap contents of two passed mutable references.
+    public(friend) native fun swap<T>(left: &mut T, right: &mut T);
+
+    /// Replace value reference points to with the given new value,


I think articles are missing, but I am not a native speaker 😄

georgemitenkov · 2024-11-12T17:14:41Z

aptos-move/framework/move-stdlib/src/natives/mem.rs

+    debug_assert!(args.len() == 2);
+
+    if args.len() != 2 {
+        return Err(SafeNativeError::InvariantViolation(PartialVMError::new(


Add a message?

aptos-move/framework/move-stdlib/tests/mem_tests.move

igor-aptos

oh I missed your previous review, addressing that one and the new one

aptos-move/aptos-release-builder/src/components/feature_flags.rs

igor-aptos · 2024-11-12T17:41:16Z

aptos-move/framework/move-stdlib/sources/mem.move

@@ -0,0 +1,27 @@
+/// Module with methods for safe memory manipulation.
+/// I.e. swapping/replacing non-copyable/non-droppable types.


I think the example is just redundant, removing. swap below has the docstring as well.

aptos-move/framework/move-stdlib/tests/mem_tests.move

igor-aptos · 2024-11-12T17:52:29Z

third_party/move/move-vm/types/src/values/values_impl.rs

+    fn swap_contents(&self, other: &Self) -> PartialVMResult<()> {
+        use Container::*;
+
+        // TODO: check if unique ownership?


oh this was from Victor's change, missed it. removing, and adding comment.

types/src/on_chain_config/aptos_features.rs

igor-aptos · 2024-11-12T17:54:08Z

types/src/on_chain_config/aptos_features.rs

@@ -317,6 +320,10 @@ impl Features {
        self.is_enabled(FeatureFlag::TRANSACTION_SIMULATION_ENHANCEMENT)
    }

+    pub fn is_native_memory_operations_enabled(&self) -> bool {


everything is using is?

ziaptos

Few minor comments

ziaptos · 2024-11-14T01:17:19Z

aptos-move/framework/move-stdlib/src/natives/mem.rs

+ *   gas cost: MEM_SWAP_BASE
+ *
+ **************************************************************************************************/
+fn native_swap(


maybe add a comment that function pre-condition is that the two references are not the same.

here we generally don't put much. but added into mem.move and into value_impl.rs, same places as we did for vector::move_range.

ziaptos · 2024-11-14T01:19:22Z

aptos-move/framework/move-stdlib/tests/mem_tests.move

@@ -0,0 +1,65 @@
+#[test_only]
+module std::mem_tests {


@igor-aptos
specialized indexed references are references to vec, vec, etc....
indexed ref is a reference to a, for instance, struct member, and vec

ziaptos · 2024-11-14T01:20:40Z

aptos-move/framework/move-stdlib/tests/mem_tests.move

+        let a = 1;
+        let b = 2;
+        let v = vector[3, 4, 5, 6];
+


Maybe add a struct with an integral member, and test swap with the member?

ziaptos · 2024-11-14T01:24:49Z

third_party/move/move-vm/types/src/values/values_impl.rs

+            }};
+        }
+
+        macro_rules! swap_g_s {


swap_general_with_specialized, etc...
let's lower the future code readers' cognitive load.

addressed

igor-aptos · 2024-11-14T07:40:08Z

aptos-move/framework/move-stdlib/tests/mem_tests.move

+        assert!(&s2 == &SomeStruct { f: 8, v: vector[9, 10]}, 6);
+        assert!(vector::borrow(&vs, 0) == &SomeStruct { f: 2, v: vector[3, 4]}, 7);
+
+        swap(&mut s1.f, vector::borrow_mut(&mut v, 0));


Maybe add a struct with an integral member, and test swap with the member?

this line should cover that?

igor-aptos · 2024-11-14T07:40:15Z

aptos-move/framework/move-stdlib/tests/mem_tests.move

+        let a = 1;
+        let b = 2;
+        let v = vector[3, 4, 5, 6];
+


igor-aptos · 2024-11-14T07:40:55Z

aptos-move/framework/move-stdlib/tests/mem_tests.move

@@ -0,0 +1,65 @@
+#[test_only]
+module std::mem_tests {


I think tests now mostly cover things, but if you see anything that would be useful to add, let me know

igor-aptos · 2024-11-14T07:47:55Z

aptos-move/framework/move-stdlib/src/natives/mem.rs

+ *   gas cost: MEM_SWAP_BASE
+ *
+ **************************************************************************************************/
+fn native_swap(


here we generally don't put much. but added into mem.move and into value_impl.rs, same places as we did for vector::move_range.

github-actions · 2024-11-14T08:28:22Z

✅ Forge suite `realistic_env_max_load` success on `ac25522950f1b46ab74e8d3167a5d6aabecdd8fb`

two traffics test: inner traffic : committed: 14563.63 txn/s, latency: 2731.16 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3000 ms), latency samples: 5537520
two traffics test : committed: 99.97 txn/s, latency: 1403.33 ms, (p50: 1300 ms, p70: 1400, p90: 1500 ms, p99: 3000 ms), latency samples: 1720
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.977, avg: 1.562", "ConsensusProposalToOrdered: max: 0.319, avg: 0.291", "ConsensusOrderedToCommit: max: 0.362, avg: 0.351", "ConsensusProposalToCommit: max: 0.656, avg: 0.643"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.07s no progress at version 1988630 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.55s no progress at version 1988628 (avg 7.73s) [limit 15].
Test Ok

github-actions · 2024-11-14T08:29:24Z

✅ Forge suite `framework_upgrade` success on `a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe` ==> `ac25522950f1b46ab74e8d3167a5d6aabecdd8fb`

Compatibility test results for a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> ac25522950f1b46ab74e8d3167a5d6aabecdd8fb (PR)
Upgrade the nodes to version: ac25522950f1b46ab74e8d3167a5d6aabecdd8fb
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1378.04 txn/s, submitted: 1382.95 txn/s, failed submission: 4.91 txn/s, expired: 4.91 txn/s, latency: 2276.22 ms, (p50: 2100 ms, p70: 2400, p90: 3300 ms, p99: 4500 ms), latency samples: 123520
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1260.07 txn/s, submitted: 1261.88 txn/s, failed submission: 1.80 txn/s, expired: 1.80 txn/s, latency: 2441.15 ms, (p50: 2400 ms, p70: 2700, p90: 3600 ms, p99: 5400 ms), latency samples: 111780
5. check swarm health
Compatibility test for a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> ac25522950f1b46ab74e8d3167a5d6aabecdd8fb passed
Upgrade the remaining nodes to version: ac25522950f1b46ab74e8d3167a5d6aabecdd8fb
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1376.45 txn/s, submitted: 1379.11 txn/s, failed submission: 2.66 txn/s, expired: 2.66 txn/s, latency: 2191.84 ms, (p50: 2100 ms, p70: 2400, p90: 3300 ms, p99: 4200 ms), latency samples: 124060
Test Ok

github-actions · 2024-11-14T08:29:28Z

✅ Forge suite `compat` success on `a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe` ==> `ac25522950f1b46ab74e8d3167a5d6aabecdd8fb`

Compatibility test results for a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> ac25522950f1b46ab74e8d3167a5d6aabecdd8fb (PR)
1. Check liveness of validators at old version: a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe
compatibility::simple-validator-upgrade::liveness-check : committed: 14995.31 txn/s, latency: 2130.58 ms, (p50: 2000 ms, p70: 2100, p90: 2200 ms, p99: 7600 ms), latency samples: 569440
2. Upgrading first Validator to new version: ac25522950f1b46ab74e8d3167a5d6aabecdd8fb
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7326.91 txn/s, latency: 3840.11 ms, (p50: 4200 ms, p70: 4400, p90: 4900 ms, p99: 5100 ms), latency samples: 144680
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7836.42 txn/s, latency: 4069.80 ms, (p50: 4100 ms, p70: 4300, p90: 6300 ms, p99: 6500 ms), latency samples: 257600
3. Upgrading rest of first batch to new version: ac25522950f1b46ab74e8d3167a5d6aabecdd8fb
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7455.23 txn/s, latency: 3906.61 ms, (p50: 4300 ms, p70: 4500, p90: 4600 ms, p99: 4700 ms), latency samples: 143220
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7009.11 txn/s, latency: 4554.19 ms, (p50: 4500 ms, p70: 4600, p90: 6700 ms, p99: 7000 ms), latency samples: 264860
4. upgrading second batch to new version: ac25522950f1b46ab74e8d3167a5d6aabecdd8fb
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11352.23 txn/s, latency: 2395.28 ms, (p50: 2300 ms, p70: 2500, p90: 3700 ms, p99: 4100 ms), latency samples: 196200
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11018.49 txn/s, latency: 2835.04 ms, (p50: 2600 ms, p70: 2700, p90: 5100 ms, p99: 6500 ms), latency samples: 357360
5. check swarm health
Compatibility test for a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> ac25522950f1b46ab74e8d3167a5d6aabecdd8fb passed
Test Ok

## Description If we have a field that contains non-copyable type, it is impossible to change it, and get the old value back. Adding two methods: * native mem::swap, that implements swap of contents of two mutable references * mem::replace, as a simple wrapper based on mem::swap ## How Has This Been Tested? provided unit tests ## Type of Change - [x] New feature ## Which Components or Systems Does This Change Impact? - [x] Move/Aptos Virtual Machine

igor-aptos requested review from runtian-zhou, wrwg, georgemitenkov and ziaptos September 27, 2024 23:01

igor-aptos requested review from davidiw and movekevin as code owners September 27, 2024 23:01

wrwg approved these changes Oct 3, 2024

View reviewed changes

igor-aptos force-pushed the igor/move_mem_swap branch from 093a454 to 06d6cb3 Compare October 3, 2024 20:04

igor-aptos requested review from alinush, heliuchuan, msmouse, lightmark, grao1991, hariria, a team, banool, gregnazario, bchocho, sasha8, gelash, zekun000, ibalajiarun, JoshLind and 0xmaayan as code owners October 3, 2024 20:04

igor-aptos changed the base branch from igor/native_compare to main October 3, 2024 20:04

igor-aptos force-pushed the igor/move_mem_swap branch from efbef47 to 7c0fbc1 Compare October 10, 2024 18:09

igor-aptos requested a review from vgao1996 October 11, 2024 05:37

igor-aptos force-pushed the igor/move_mem_swap branch 3 times, most recently from c5f381e to 8b0d35b Compare October 16, 2024 21:17

igor-aptos commented Oct 21, 2024

View reviewed changes

igor-aptos force-pushed the igor/move_mem_swap branch from 8b0d35b to 170eca3 Compare October 21, 2024 23:10

igor-aptos force-pushed the igor/move_mem_swap branch from 170eca3 to a99eccd Compare November 8, 2024 23:21

georgemitenkov reviewed Nov 12, 2024

View reviewed changes

aptos-move/framework/move-stdlib/tests/mem_tests.move Show resolved Hide resolved

igor-aptos commented Nov 12, 2024

View reviewed changes

igor-aptos force-pushed the igor/move_mem_swap branch from 23c63a3 to c95646e Compare November 12, 2024 18:20

igor-aptos requested a review from georgemitenkov November 12, 2024 22:09

ziaptos approved these changes Nov 14, 2024

View reviewed changes

[move][stdlib] Implement mem::swap native move call

ac25522

igor-aptos force-pushed the igor/move_mem_swap branch from c95646e to ac25522 Compare November 14, 2024 07:58

igor-aptos enabled auto-merge (squash) November 14, 2024 08:00

This comment has been minimized.

Sign in to view

igor-aptos commented Nov 14, 2024

View reviewed changes

igor-aptos merged commit 6255194 into main Nov 14, 2024
87 of 92 checks passed

igor-aptos deleted the igor/move_mem_swap branch November 14, 2024 08:33

		@@ -0,0 +1,27 @@
		/// Module with methods for safe memory manipulation.
		/// I.e. swapping/replacing non-copyable/non-droppable types.

[move][stdlib] Implement mem::swap native move call #14786

[move][stdlib] Implement mem::swap native move call #14786

Conversation

igor-aptos commented Sep 27, 2024

Description

How Has This Been Tested?

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Sep 27, 2024 • edited Loading

codecov bot commented Sep 27, 2024 • edited Loading

Codecov Report

lightmark commented Oct 3, 2024

igor-aptos commented Oct 3, 2024

vgao1996 commented Oct 10, 2024

igor-aptos left a comment

Choose a reason for hiding this comment

igor-aptos commented Nov 8, 2024

georgemitenkov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

igor-aptos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ziaptos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 14, 2024

✅ Forge suite realistic_env_max_load success on ac25522950f1b46ab74e8d3167a5d6aabecdd8fb

github-actions bot commented Nov 14, 2024

✅ Forge suite framework_upgrade success on a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> ac25522950f1b46ab74e8d3167a5d6aabecdd8fb

github-actions bot commented Nov 14, 2024

✅ Forge suite compat success on a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe ==> ac25522950f1b46ab74e8d3167a5d6aabecdd8fb

trunk-io bot commented Sep 27, 2024 •

edited

Loading

codecov bot commented Sep 27, 2024 •

edited

Loading

✅ Forge suite `realistic_env_max_load` success on `ac25522950f1b46ab74e8d3167a5d6aabecdd8fb`

✅ Forge suite `framework_upgrade` success on `a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe` ==> `ac25522950f1b46ab74e8d3167a5d6aabecdd8fb`

✅ Forge suite `compat` success on `a0ec6ba11bfe4cfc5b586edc9e227aba4909e8fe` ==> `ac25522950f1b46ab74e8d3167a5d6aabecdd8fb`