feat(parser): Generate Serialize impls in ast_tools

[ottomated]

Beginning of #6347. Instead of using serde-derive, we generate Serialize impls manually.

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “0-merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[Boshen]

Nice, I believe this'll make compilation faster as well, because serde no longer runs syn.

[ottomated]

Tested on checker.ts. Diff below. The only changes are in regex serialization which is incorrect anyway so shouldn't be an issue.

--- a/expected.json
+++ b/actual.json
@@ -20457,11 +20457,9 @@
             "type": "RegExpLiteral",
             "start": 29848,
             "end": 29856,
-            "value": null,
+            "value": {},
             "regex": {
-              "pattern": {
-                "Raw": "^\".+\"$"
-              },
+              "pattern": "^\".+\"$",
               "flags": ""
             }
           },
@@ -247915,11 +247913,9 @@
                                                                     "type": "RegExpLiteral",
                                                                     "start": 404540,
                                                                     "end": 404545,
-                                                                    "value": null,
+                                                                    "value": {},
                                                                     "regex": {
-                                                                      "pattern": {
-                                                                        "Raw": "\\n"
-                                                                      },
+                                                                      "pattern": "\\n",
                                                                       "flags": "g"
                                                                     }
                                                                   },
@@ -256454,11 +256450,9 @@
                                                   "type": "RegExpLiteral",
                                                   "start": 417226,
                                                   "end": 417238,
-                                                  "value": null,
+                                                  "value": {},
                                                   "regex": {
-                                                    "pattern": {
-                                                      "Raw": "\\r\\n|\\n|\\r"
-                                                    },
+                                                    "pattern": "\\r\\n|\\n|\\r",
                                                     "flags": ""
                                                   }
                                                 }
@@ -256546,11 +256540,9 @@
                                                           "type": "RegExpLiteral",
                                                           "start": 417265,
                                                           "end": 417271,
-                                                          "value": null,
+                                                          "value": {},
                                                           "regex": {
-                                                            "pattern": {
-                                                              "Raw": "^\\s+"
-                                                            },
+                                                            "pattern": "^\\s+",
                                                             "flags": ""
                                                           }
                                                         },
@@ -287745,11 +287737,9 @@
                                                         "type": "RegExpLiteral",
                                                         "start": 470730,
                                                         "end": 470736,
-                                                        "value": null,
+                                                        "value": {},
                                                         "regex": {
-                                                          "pattern": {
-                                                            "Raw": "\\\\."
-                                                          },
+                                                          "pattern": "\\\\.",
                                                           "flags": "g"
                                                         }
                                                       },
@@ -321794,11 +321784,9 @@
                                                               "type": "RegExpLiteral",
                                                               "start": 536344,
                                                               "end": 536349,
-                                                              "value": null,
+                                                              "value": {},
                                                               "regex": {
-                                                                "pattern": {
-                                                                  "Raw": "\\n"
-                                                                },
+                                                                "pattern": "\\n",
                                                                 "flags": "g"
                                                               }
                                                             },
@@ -358021,11 +358009,9 @@
                                             "type": "RegExpLiteral",
                                             "start": 606654,
                                             "end": 606667,
-                                            "value": null,
+                                            "value": {},
                                             "regex": {
-                                              "pattern": {
-                                                "Raw": "[^a-z0-9]"
-                                              },
+                                              "pattern": "[^a-z0-9]",
                                               "flags": "gi"
                                             }
                                           },
@@ -1212733,11 +1212719,9 @@
                                             "type": "RegExpLiteral",
                                             "start": 2037983,
                                             "end": 2038033,
-                                            "value": null,
+                                            "value": {},
                                             "regex": {
-                                              "pattern": {
-                                                "Raw": "^(?:EventTarget|Node|(?:HTML[a-zA-Z]*)?Element)$"
-                                              },
+                                              "pattern": "^(?:EventTarget|Node|(?:HTML[a-zA-Z]*)?Element)$",
                                               "flags": ""
                                             }
                                           },

[ottomated]

Important changes:

• #[derive(Serialize)] -> #[generate_derive(ESTree)]
• #[serde(...)] -> #[estree(...)]
  • Supported on fields: skip, flatten, rename
  • Supported on enums: rename_all, untagged (in the future, I propose renaming this to something like #[estree(transparent)] for untagged enums, and maybe #[estree(type_enum)] for enums that shouldn't be camelCase)
  • Supported on structs: type = "Identifier", no_type (for things that aren't really AST nodes and don't need a "type" field on the JSON
• By default, structs get the "type" field from the following (prioritized in this order):

1. #[estree(type = "X")] / #[estree(no_type)]
2. If the struct has an r#type field, it'll use that as the value
3. The name of the struct
   Therefore, no need for tag = "type".

• Struct fields are converted to camelCase by default. I verified that any struct missing rename_all = "camelCase" already had all its fields in camelCase format so this should not change anything.
• New oxc_estree crate that currently only contains an empty trait. I envision this being used for custom serialization and type generation in the future, but maybe a different approach is warranted.

[codspeed-hq]

CodSpeed Performance Report

Merging #6404 will not alter performance

_{Comparing ottomated:manual-serialize (7639831) with main (b0b6ac7)}

Summary

✅ 30 untouched benchmarks

[Boshen]

I like the direction this is going, good job 👍

[overlookmotel]

Sorry it's taken me a while to get to reviewing this.

Excellent work. Really pleased to see this moving forwards. Thank you.

All looks good with the serialized output. As you say, the output changing for RegExp doesn't matter.

However, we do have one problem: The TS type defs generated for WASM are now inaccurate, because Tsify derive macro relies on #[serde] attrs, and they've been replaced with #[estree], which Tsify doesn't understand.

Old TS def:

export interface NumericLiteral extends Span {
    type: "NumericLiteral";
    value: number;
    raw: string;
}

After this PR:

export interface NumericLiteral {
    span: Span;
    value: number;
    raw: string;
    base: NumberBase;
}

To see the generated TS types:

cd wasm/parser
pnpm install
pnpm run build

Type defs are generated at npm/parser-wasm/node/oxc_parser_wasm.d.ts.

I can see 2 ways to solve this:

1. Go back to #[serde]

Scale back the scope of this PR a little. Revert #[estree] to #[serde].

Unfortunately this would also involve reverting the way type is dealt with, which is definitely an improvement, but is not compatible with Tsify.

We'd then try to replace Tsify as the next step, and then bring in #[estree] again.

2. Replace Tsify as well now

Replace Tsify and generate type defs in oxc_ast_tools instead.

I'm not sure how much work this would be. It's definitely not trivial, but maybe it's less hard than it initially looks.

Tsify derive macro produces a lot of code, but as we don't use wasm-bindgen for transferring these types to JS, I think the only part of Tsify's output that we actually need is:

#[wasm_bindgen(typescript_custom_section)]
const TS_APPEND_CONTENT: &'static str =
    "export interface NumericLiteral extends Span {\n    value: number;\n    raw: string;\n}";

Which?

Usually we try to keep PRs small and work incrementally, rather than long-lived branches which tend to acquire merge conflicts, and are hard to review. So that would point more towards approach (1), and that'd be my preference. However, @ottomated if you feel that 2nd approach is less work and you'd prefer to go that way, happy to go with that.

What do you think?

[ottomated]

@overlookmotel yep, already noticed this and started work on replacing tsify. I was wondering, however, if anything's actually depending on the wasm types right now — are they published, could they be temporarily disabled? I should have the new type generation work done this week.

[overlookmotel]

I've pushed a few more commits. @ottomated I hope you don't mind. It took me most of yesterday to review this PR, so I didn't want to request changes and end up going through the whole diff again to re-review. If anything you disagree with, feel free to open additional PRs to revert my changes.

Most are cosmetic. The ones which aren't are:

• fix: type field in TS defs on structs with #[estree(custom_serialize)] attr

TS type defs were missing the type field for ArrayAssignmentTarget etc.

Note: I checked all the type defs. These were the only significant errors I found (comparing to the old .d.ts). Will submit separate PRs to fix another couple of less important differences.

• fix: restore features = ["derive"] for serde in oxc_ast crate

We still have some #[derive(Serialize)] usage in oxc_ast/src/serialize.rs. Surprised Rust didn't complain. I guess it worked because oxc_span still had the feature enabled and oxc_ast depends on oxc_span.

• refactor: remove unused TypescriptGenerator
• refactor: revert GeneratorOutput

I removed these to reduce the diff. I'll resubmit those changes as separate PRs.

[overlookmotel]

Merge activity

• Oct 19, 4:26 AM EDT: The merge label '0-merge' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.
• Oct 19, 4:27 AM EDT: A user added this pull request to the Graphite merge queue.
• Oct 19, 4:27 AM EDT: The Graphite merge queue couldn't merge this PR because it failed for an unknown reason (Stack merges are not currently supported for forked repositories. Please create a branch in the target repository in order to merge).