[compiler-v2] Ast generator from stackless bytecode, Source Gen from Ast #14494

wrwg · 2024-09-02T06:57:46Z

Description

Converter from stackless bytecode into Model AST ('astifier'). (We already have one from binary format to stackless.) Also adds a converter from AST to source ('sourcifier').

This adds the major pieces for a working decompiler. However, the pieces are not fully connected to a tool yet, as more sidework needs to be done first. Specifically, this PR adds breaks/continues to labeled outer loops, which are not yet supported by the compiler.

How is this tested

There are some tests in a new test crate ast-generator-tests. These call the v2 compiler to compile up to file format, then decompile from there into stackless bytecode, into AST, and via sourcifier back to source. However, desired execution roundtrip comparison tests can only be implemented once the full toolchain is ready.

Another set of tests for the 'sourcifier' is via compiler-v2 baseline files. Where AST dump is requested in those tests, we dump now also the sourcified AST. This can live side-by-side for a while until we may decide to remove one of the outputs.

Type of Change

Which Components or Systems Does This Change Impact?

trunk-io · 2024-09-02T06:57:50Z

⏱️ 49m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-move-unit-coverage	14m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-unit-coverage	8m	🟩
general-lints	4m	🟩 🟩
rust-cargo-deny	3m	🟩 🟥
check-dynamic-deps	2m	🟩 🟩
semgrep/ci	49s	🟩 🟩
file_change_determinator	24s	🟩 🟩
file_change_determinator	20s	🟩 🟩
permission-check	7s	🟩 🟩
permission-check	5s	🟩 🟩
permission-check	5s	🟩 🟩
permission-check	4s	🟩 🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

wrwg · 2024-09-02T06:58:03Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @wrwg and the rest of your teammates on Graphite

codecov · 2024-09-02T07:07:56Z

Codecov Report

Attention: Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 60.0%. Comparing base (800bacf) to head (11cc61f).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...ty/move/move-compiler-v2/src/bytecode_generator.rs	50.0%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #14494       +/-   ##
===========================================
- Coverage    72.6%    60.0%    -12.6%     
===========================================
  Files        2400      856     -1544     
  Lines      485594   210624   -274970     
===========================================
- Hits       352847   126555   -226292     
+ Misses     132747    84069    -48678

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

brmataptos

I still need to read astifier.rs, but here are a few comments.

third_party/move/move-model/bytecode/ast-generator-tests/src/lib.rs

third_party/move/move-model/src/ast.rs

third_party/move/move-model/src/exp_builder.rs

third_party/move/move-model/src/ast.rs

third_party/move/move-model/src/sourcifier.rs

third_party/move/move-model/bytecode/src/astifier.rs

third_party/move/move-compiler-v2/tests/checking/typing/return_any_type.exp

third_party/move/move-compiler-v2/tests/checking/typing/main_call_entry.exp

third_party/move/move-compiler-v2/tests/checking/typing/dummy_field.exp

third_party/move/move-compiler-v2/tests/checking/typing/constant_all_valid_types.exp

wrwg

Thanks for the review!

third_party/move/move-model/src/ast.rs

third_party/move/move-model/src/sourcifier.rs

third_party/move/move-model/bytecode/src/stackless_control_flow_graph.rs

third_party/move/move-compiler-v2/tests/checking/typing/return_any_type.exp

third_party/move/move-compiler-v2/tests/checking/typing/main_call_entry.exp

third_party/move/move-compiler-v2/tests/checking/typing/dummy_field.exp

third_party/move/move-compiler-v2/tests/checking/typing/constant_all_valid_types.exp

Converter from stackless bytecode into Model AST ('astifier'). (We already have one from binary format to stackless.) Also adds a converter from AST to source ('sourcifier'). This adds the major pieces for a working decompiler. However, the pieces are not fully connected to a tool yet, as more sidework needs to be done first. Specifically, this PR adds breaks/continues to labeled outer loops, which are not yet supported by the compiler. There are some tests in a new test crate `ast-generator-tests`. These call the v2 compiler to compile up to file format, then decompile from there into stackless bytecode, into AST, and via sourcifier back to source. However, desired execution roundtrip comparison tests can only be implemented once the full toolchain is ready. Another set of tests for the 'sourcifier' is via compiler-v2 baseline files. Where AST dump is requested in those tests, we dump now also the sourcified AST. This can live side-by-side for a while until we may decide to remove one of the outputs.

github-actions · 2024-10-09T04:14:11Z

✅ Forge suite `realistic_env_max_load` success on `11cc61fa8213686dcb9658571ee743fa35e54c3b`

two traffics test: inner traffic : committed: 12482.38 txn/s, latency: 3184.14 ms, (p50: 2900 ms, p70: 3000, p90: 4200 ms, p99: 7500 ms), latency samples: 4746080
two traffics test : committed: 100.03 txn/s, latency: 3195.94 ms, (p50: 2500 ms, p70: 2900, p90: 5200 ms, p99: 11800 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.238, avg: 0.225", "QsPosToProposal: max: 0.359, avg: 0.286", "ConsensusProposalToOrdered: max: 0.321, avg: 0.314", "ConsensusOrderedToCommit: max: 0.490, avg: 0.459", "ConsensusProposalToCommit: max: 0.809, avg: 0.773"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.41s no progress at version 2483747 (avg 0.22s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.72s no progress at version 2483745 (avg 7.72s) [limit 15].
Test Ok

github-actions · 2024-10-09T04:15:20Z

✅ Forge suite `compat` success on `46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775` ==> `11cc61fa8213686dcb9658571ee743fa35e54c3b`

Compatibility test results for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 11cc61fa8213686dcb9658571ee743fa35e54c3b (PR)
1. Check liveness of validators at old version: 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775
compatibility::simple-validator-upgrade::liveness-check : committed: 11121.49 txn/s, latency: 3076.97 ms, (p50: 2200 ms, p70: 2700, p90: 6700 ms, p99: 22300 ms), latency samples: 410140
2. Upgrading first Validator to new version: 11cc61fa8213686dcb9658571ee743fa35e54c3b
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7449.83 txn/s, latency: 3776.56 ms, (p50: 4300 ms, p70: 4500, p90: 4600 ms, p99: 4700 ms), latency samples: 139460
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7434.24 txn/s, latency: 4338.23 ms, (p50: 4700 ms, p70: 4900, p90: 5000 ms, p99: 5000 ms), latency samples: 249300
3. Upgrading rest of first batch to new version: 11cc61fa8213686dcb9658571ee743fa35e54c3b
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6045.54 txn/s, latency: 4390.12 ms, (p50: 4900 ms, p70: 5100, p90: 5400 ms, p99: 5700 ms), latency samples: 128300
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6475.66 txn/s, latency: 4878.51 ms, (p50: 5000 ms, p70: 5300, p90: 7200 ms, p99: 7700 ms), latency samples: 219920
4. upgrading second batch to new version: 11cc61fa8213686dcb9658571ee743fa35e54c3b
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10496.01 txn/s, latency: 2597.15 ms, (p50: 2800 ms, p70: 3000, p90: 3100 ms, p99: 3700 ms), latency samples: 185020
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10585.03 txn/s, latency: 2995.96 ms, (p50: 2900 ms, p70: 3000, p90: 3600 ms, p99: 5100 ms), latency samples: 347060
5. check swarm health
Compatibility test for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 11cc61fa8213686dcb9658571ee743fa35e54c3b passed
Test Ok

github-actions · 2024-10-09T04:16:25Z

✅ Forge suite `framework_upgrade` success on `46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775` ==> `11cc61fa8213686dcb9658571ee743fa35e54c3b`

Compatibility test results for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 11cc61fa8213686dcb9658571ee743fa35e54c3b (PR)
Upgrade the nodes to version: 11cc61fa8213686dcb9658571ee743fa35e54c3b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1179.88 txn/s, submitted: 1182.52 txn/s, failed submission: 2.64 txn/s, expired: 2.64 txn/s, latency: 2510.41 ms, (p50: 2400 ms, p70: 2700, p90: 3600 ms, p99: 5100 ms), latency samples: 107220
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1092.80 txn/s, submitted: 1095.03 txn/s, failed submission: 2.23 txn/s, expired: 2.23 txn/s, latency: 2726.69 ms, (p50: 2500 ms, p70: 3000, p90: 4500 ms, p99: 6500 ms), latency samples: 97880
5. check swarm health
Compatibility test for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 11cc61fa8213686dcb9658571ee743fa35e54c3b passed
Upgrade the remaining nodes to version: 11cc61fa8213686dcb9658571ee743fa35e54c3b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1177.93 txn/s, submitted: 1180.13 txn/s, failed submission: 2.20 txn/s, expired: 2.20 txn/s, latency: 2551.87 ms, (p50: 2400 ms, p70: 2900, p90: 3600 ms, p99: 5700 ms), latency samples: 107000
Test Ok

wrwg mentioned this pull request Sep 2, 2024

[compiler-v2][decompiler] Pulling fat-loop algorithm out of the prover #14488

Merged

15 tasks

wrwg force-pushed the wrwg/astifier branch from 8394da1 to 29e2d8d Compare September 2, 2024 07:19

wrwg force-pushed the wrwg/fat-loop branch from 2dc4e05 to 8d449bb Compare September 23, 2024 06:48

wrwg force-pushed the wrwg/astifier branch from 29e2d8d to be57207 Compare September 23, 2024 06:48

wrwg changed the title ~~[WIP][compiler-v2] Ast generator from stackless bytecode~~ [compiler-v2] Ast generator from stackless bytecode Sep 23, 2024

wrwg requested review from brmataptos, vineethk, rahxephon89 and fEst1ck September 23, 2024 08:24

wrwg marked this pull request as ready for review September 23, 2024 08:25

wrwg force-pushed the wrwg/fat-loop branch from 8d449bb to 7a3da0a Compare September 23, 2024 18:15

wrwg force-pushed the wrwg/astifier branch from be57207 to f3f4275 Compare September 23, 2024 18:15

Base automatically changed from wrwg/fat-loop to main September 23, 2024 21:15

wrwg force-pushed the wrwg/astifier branch 6 times, most recently from f32211e to 04a48c5 Compare September 26, 2024 15:13

brmataptos reviewed Sep 30, 2024

View reviewed changes

third_party/move/move-model/bytecode/ast-generator-tests/src/lib.rs Show resolved Hide resolved

third_party/move/move-model/src/ast.rs Show resolved Hide resolved

third_party/move/move-model/src/exp_builder.rs Outdated Show resolved Hide resolved

brmataptos self-requested a review September 30, 2024 18:19

wrwg force-pushed the wrwg/astifier branch from 04a48c5 to cce4419 Compare October 3, 2024 06:25

wrwg changed the title ~~[compiler-v2] Ast generator from stackless bytecode~~ [compiler-v2] Ast generator from stackless bytecode, Source Gen from Ast Oct 3, 2024

wrwg force-pushed the wrwg/astifier branch 3 times, most recently from d74e23c to 18b921d Compare October 4, 2024 06:54

wrwg mentioned this pull request Oct 4, 2024

[compiler-v2] Add loop labels to the language #14868

Merged

16 tasks

vineethk approved these changes Oct 7, 2024

View reviewed changes

wrwg commented Oct 8, 2024

View reviewed changes

wrwg force-pushed the wrwg/astifier branch 4 times, most recently from b0238dd to 4352c47 Compare October 9, 2024 02:42

rahxephon89 approved these changes Oct 9, 2024

View reviewed changes

wrwg and others added 2 commits October 8, 2024 19:49

Implement positional layouts in sourcifier

7edbc85

wrwg force-pushed the wrwg/astifier branch from 4352c47 to ae2049a Compare October 9, 2024 02:49

wrwg enabled auto-merge (squash) October 9, 2024 02:54

This comment has been minimized.

Sign in to view

Fixing type display context for multiple usage scenarios

11cc61f

wrwg force-pushed the wrwg/astifier branch from ae2049a to 11cc61f Compare October 9, 2024 03:56

This comment has been minimized.

Sign in to view

wrwg merged commit cbb4431 into main Oct 9, 2024
52 checks passed

wrwg deleted the wrwg/astifier branch October 9, 2024 04:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[compiler-v2] Ast generator from stackless bytecode, Source Gen from Ast #14494

[compiler-v2] Ast generator from stackless bytecode, Source Gen from Ast #14494

wrwg commented Sep 2, 2024 •

edited

Loading

trunk-io bot commented Sep 2, 2024 •

edited

Loading

wrwg commented Sep 2, 2024 •

edited

Loading

codecov bot commented Sep 2, 2024 •

edited

Loading

brmataptos left a comment

wrwg left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

[compiler-v2] Ast generator from stackless bytecode, Source Gen from Ast #14494

[compiler-v2] Ast generator from stackless bytecode, Source Gen from Ast #14494

Conversation

wrwg commented Sep 2, 2024 • edited Loading

Description

How is this tested

Type of Change

Which Components or Systems Does This Change Impact?

trunk-io bot commented Sep 2, 2024 • edited Loading

wrwg commented Sep 2, 2024 • edited Loading

codecov bot commented Sep 2, 2024 • edited Loading

Codecov Report

brmataptos left a comment

Choose a reason for hiding this comment

wrwg left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Oct 9, 2024

✅ Forge suite realistic_env_max_load success on 11cc61fa8213686dcb9658571ee743fa35e54c3b

github-actions bot commented Oct 9, 2024

✅ Forge suite compat success on 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 11cc61fa8213686dcb9658571ee743fa35e54c3b

github-actions bot commented Oct 9, 2024

✅ Forge suite framework_upgrade success on 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 11cc61fa8213686dcb9658571ee743fa35e54c3b

wrwg commented Sep 2, 2024 •

edited

Loading

trunk-io bot commented Sep 2, 2024 •

edited

Loading

wrwg commented Sep 2, 2024 •

edited

Loading

codecov bot commented Sep 2, 2024 •

edited

Loading

✅ Forge suite `realistic_env_max_load` success on `11cc61fa8213686dcb9658571ee743fa35e54c3b`

✅ Forge suite `compat` success on `46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775` ==> `11cc61fa8213686dcb9658571ee743fa35e54c3b`

✅ Forge suite `framework_upgrade` success on `46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775` ==> `11cc61fa8213686dcb9658571ee743fa35e54c3b`