WIP: dump asts

[nojaf]

When calling bunx rescript-tools.exe doc MyFile.res I would like to have some more details about the binding signatures. Mainly what the parameter types and return type is.

Currently I get:

{
  "signature": "let createOrder: FirebaseFunctions.callableFunction<\n  Domain.createOrderRequest,\n  string,\n>",
}

After this PR:

{

 "signature": "let createOrder: FirebaseFunctions.callableFunction<\n  Domain.createOrderRequest,\n  string,\n>",
 "detail": {
      "returnType": {
        "path": "FirebaseFunctions.callableFunction",
        "genericTypeParameters": [{
          "path": "Domain.createOrderRequest"
        }, {
          "path": "string"
        }]
      }
    }
}

Using Tools_Docgen I want to detect FirebaseFunctions.callableFunction and what the type parameters are to generate client ReScript code in my project.
I currently use a Regex to process this, but I would like to grab that information more easily from the JSON.

In order to figure out what on earth is present in Types.type_expr I wrote a helper file to dump the contents in friendly, JSON esque way:

Example:

type_desc.Tconstr(
  path = function$
  ts = [
    type_desc.Tarrow(
      t1 = type_desc.Tconstr(path = string, ts = [])
      t2 = type_desc.Tconstr(path = string, ts = [])
    )
    type_desc.Tvariant({
      row_fields = [ (label = Has_arity1, row_field = row_field.Rpresent) ]
      row_more = type_desc.Tnil
      row_closed = true
      row_fixed = false
    })
  ]
)

This is inspired by what we do in our F# ast viewer.

@zth although this is still WIP, I could already use a review to find out if I'm going in the right direction with this.

[nojaf]

Added some more code to do a dump of the full file.

{
  package = { genericJsxModule = None }
  file = {
    uri = 
      "file:///home/nojaf/projects/tenmileselverdinge-tickets/functions/src/CreateOrder.res"
    moduleName = "CreateOrder"
    structure = {
      name = "CreateOrder"
      docstring = []
      items = [
        {
          kind = 
            SharedTypes.Module.Value(
              type_desc.Tconstr(
                path = function$
                ts = [
                  type_desc.Tarrow(
                    t1 = type_desc.Tconstr(path = string, ts = [])
                    t2 = type_desc.Tconstr(path = string, ts = [])
                  )
                  type_desc.Tvariant({
                    row_fields = [ (label = Has_arity1, row_field = row_field.Rpresent) ]
                    row_more = type_desc.Tnil
                    row_closed = true
                    row_fixed = false
                  })
                ]
              )
            )
          name = "encodeURIComponent"
          docstring = []
          deprecated = None
        }
        {
          kind = 
            SharedTypes.Module.Value(
              type_desc.Tlink(
                type_desc.Tconstr(
                  path = JsonCombinators.Json.Decode.t
                  ts = [ type_desc.Tlink(type_desc.Tconstr(path = bool, ts = [])) ]
                )
              )
            )
          name = "decodeTurnTile"
          docstring = []
          deprecated = None
        }
        {
          kind = 
            SharedTypes.Module.Value(
              type_desc.Tlink(
                type_desc.Tconstr(
                  path = function$
                  ts = [
                    type_desc.Tlink(
                      type_desc.Tarrow(
                        t1 = type_desc.Tlink(type_desc.Tconstr(path = string, ts = []))
                        t2 = 
                          type_desc.Tlink(
                            type_desc.Tconstr(
                              path = RescriptCore.Promise.t
                              ts = [ type_desc.Tlink(type_desc.Tconstr(path = bool, ts = [])) ]
                            )
                          )
                      )
                    )
                    type_desc.Tvariant({
                      row_fields = [ (label = Has_arity1, row_field = row_field.Rpresent) ]
                      row_more = type_desc.Tnil
                      row_closed = true
                      row_fixed = false
                    })
                  ]
                )
              )
            )
          name = "validateTurnTileToken"
          docstring = []
          deprecated = None
        }
        {
          kind = 
            SharedTypes.Module.Value(
              type_desc.Tlink(
                type_desc.Tconstr(
                  path = function$
                  ts = [
                    type_desc.Tlink(
                      type_desc.Tarrow(
                        t1 = type_desc.Tlink(type_desc.Tconstr(path = Domain.food, ts = []))
                        t2 = type_desc.Tlink(type_desc.Tconstr(path = Stripe.priceId, ts = []))
                      )
                    )
                    type_desc.Tvariant({
                      row_fields = [ (label = Has_arity1, row_field = row_field.Rpresent) ]
                      row_more = type_desc.Tnil
                      row_closed = true
                      row_fixed = false
                    })
                  ]
                )
              )
            )
          name = "stripePriceIdOfFood"
          docstring = []
          deprecated = None
        }
        {
          kind = 
            SharedTypes.Module.Value(
              type_desc.Tlink(
                type_desc.Tconstr(
                  path = function$
                  ts = [
                    type_desc.Tlink(
                      type_desc.Tarrow(
                        t1 = type_desc.Tlink(type_desc.Tconstr(path = Domain.ticket, ts = []))
                        t2 = type_desc.Tlink(type_desc.Tconstr(path = Stripe.priceId, ts = []))
                      )
                    )
                    type_desc.Tvariant({
                      row_fields = [ (label = Has_arity1, row_field = row_field.Rpresent) ]
                      row_more = type_desc.Tnil
                      row_closed = true
                      row_fixed = false
                    })
                  ]
                )
              )
            )
          name = "stripePriceIdOfTicket"
          docstring = []
          deprecated = None
        }
        {
          kind = 
            SharedTypes.Module.Value(
              type_desc.Tlink(
                type_desc.Tconstr(
                  path = FirebaseFunctions.callableFunction
                  ts = [
                    type_desc.Tlink(type_desc.Tconstr(path = Domain.createOrderRequest, ts = []))
                    type_desc.Tlink(type_desc.Tconstr(path = string, ts = []))
                  ]
                )
              )
            )
          name = "createOrder"
          docstring = []
          deprecated = None
        }
      ]
      deprecated = None
    }
  }
}

This is nice, and could hopefully someday turn into an ast viewer.
I am bumping, however, into some OCaml limitations that make the recursive processing of mk_type_desc. It doesn't stack overflow but is very slow when called from mk_item.

[nojaf]

Alright, @zth, this is ready for review!

The original goal was to include more information about a function signature in the JSON. However, I got side-tracked and added a dump command to the tool, which will pretty print the SharedTypes.full type. This was very useful for determining what I needed to pattern match on to extract the signature information.

Please let me know your thoughts on this. If you believe this addition is worthwhile, I would greatly appreciate a thorough review, as I think this is my first OCaml PR. Thanks a bunch for your time!**

[zth]

Looks good overall, thank you! A couple of questions and small nits.

[zth]

Just to make sure I understand the justification of introducing another representation - the intention here is to have a generic pretty printer that we can use for debugging, correct? Where tast is the first thing we've made a debug printer for, but we can extend it as needed.

Provided the answer to the above is "yes", here's another important question:

• We usually deal with loc:s and cursor position in the editor tooling. It's important to know whether the loc of something is a) a regular loc b) a ghost loc c) an empty/broken loc. It's also important to be able to mark in pretty printing whether the cursor is inside of the item's loc. Would it be difficult to extend this pretty printer to handle loc:s?

Even if we don't use it right now, I'd like to see a PoC that it's doable before we commit to a specific pretty printing DSL.

[nojaf]

Just to make sure I understand the justification of introducing another representation - the intention here is to have a generic pretty printer that we can use for debugging, correct? Where tast is the first thing we've made a debug printer for, but we can extend it as needed.

Yes, I wanted to pattern match in tools.ml but wasn't sure what shape I was aiming for. So, I created a new representation and wrote a printer for it. In the future, this could be reused for the untyped tree.

I believe the loc aspect can be captured as you described. It would be helpful to have a concrete example to ensure I fully understand what you would like to see.

[zth]

Sure! Have a look through this file: https://github.com/rescript-lang/rescript-vscode/blob/master/analysis/src/DumpAst.ml

That prints (parts of) the parsetree, and marks structures with whether they hold the cursor or not (or if the loc is broken).

[nojaf]

Okay, in the tree traversal you would like to pass the cursor position and have that print something special if a node contains the cursor?

[zth]

Exactly, or if the loc is broken. Reason is that both of those are very important when working with things like hovers, autocomplete, etc. I don't need to see the actual locs I think, just if the cursor is in there and/or if the loc is broken. For actual locs, it's easy enough to print via bsc.

[nojaf]

A couple of questions there:

I'm not sure if I understand what a broken loc is. What does that mean?

I don't need to see the actual locs

I'm a little surprised by this. Can you elaborate on this? Or is it more a practical thing?

A while ago you told me that bsc -dtypedtree dumps the typed tree, however, inside a real project you need to pass in the dependencies and other flags. Is there an easy way find out what other arguments would be required? Like what is bunx rescript eventually gonna pass down?

[zth]

1. Broken loc: A broken loc is what the parser inserts when a syntax error is produced that doesn't have a clear loc by itself. Remember that the parser and editor tooling does most of its work on broken rather than complete sources. So, sometimes the parser inserts things like %rescript.exprHole with a "broken" loc at a place where it couldn't figure out more about what type of expression or pattern you're trying to write. These are important cues that help when doing autocomplete.
2. Not needing the actual locs: Exactly, it's a practical thing. They are needed occasionally when debugging something intricate, but mostly what you're after when working on editor features (or debugging existing features) is figuring out where the cursor is located, not what those exact locs are.
3. When you just care about what editor sees/works on, you don't need to know that much about what bsc would run when compiling. That's when those flags are needed. If you want to see precisely what the editor sees via the parser, this command is good enough (and it requires no knowledge of libs, PPXes etc): bsc whatever.res -dparsetree -ignore-parse-errors -only-parse -bs-no-builtin-ppx -bs-loc. This will give you just the parsetree back, just as the parser sees it (which is what the editor operates on). No PPXes (including disabling the builtin PPXes) and -bs-loc will give you the actual locs.

[nojaf]

I'm considering to split this PR in two, I want to experiment a bit further with the dump tool.
Would like to extract the signature information part in a separate PR.

[zth]

I'm considering to split this PR in two, I want to experiment a bit further with the dump tool. Would like to extract the signature information part in a separate PR.

Sounds reasonable!

[nojaf]

I'm considering to split this PR in two, I want to experiment a bit further with the dump tool. Would like to extract the signature information part in a separate PR.

Sounds reasonable!

@zth I created #1043 with the signature json changes.