Add the AST json file generation feature for external analysis purpose

[jclavier44]

In order to use Teal in a project using mlua and rust, I need to extract all variables name and type from my teal files in order to automatically link lua with a typed language (Rust in my case).
I saw we have the AST result during generation in 'tl' file.
So I have added a function to convert it into json so that a 'FILE_NAME_ast.json' is generated if the config "gen_ast" is set at a number greater than 0. I have used a number for this config key in order to use it as a parameter for the max depth of the AST json generation (For the moment with gen_ast = 10 is getting enough info from the AST, but we will be able to increase it if need).
I will publish soon on my github account the node.js program which extract variables from Teal files and will add the link to "awesome-teal".

[github-actions]

Teal Playground URL: https://840--teal-playground-preview.netlify.app

[hishamhm]

Hi! Thanks for the initiative in contributing!

The AST format is internal to the compiler and hence it is not stable. For this reason I don't feel super comfortable in exposing it for third-party tools that would rely on it at this time.

The output produced by tl types, though, is intended to be stable. Is it sufficient for your purposes? If not, what kind of information specifically that you need that is absent from there? Extending that output if necessary might be a safer approach.

[jclavier44]

Here is an example of what I want from a teal file (I used tutorial example). When you mean not stable, is that it is that the structure is changing often with new features coming?
tutorial_variables_extraction.zip

[hishamhm]

When you mean not stable, is that it is that the structure is changing often with new features coming?

It means that is an internal data structure owned by the compiler. That means there is no public guarantee that it will stay stable across releases. If it changes often or rarely, that is less of the point; the point is that it can change at any time for any reason (e.g. internal refactoring) and it is not considered a bug or a breaking change.

The tl types interface, however, is intended to be stable and it is relied on by tools such as vscode-teal. That's why I asked if the information provided by it is sufficient.

I'm on my phone, sorry, I can't easily check out a zip file right now, but I'd be glad if you could summarize what is the kind of information you need (other than a full dump of the AST). Is it a list of public symbols exposed by a module and their types? From your description at the top of the message it sounds unlikely that you need every local variable in order to do language bindings.

[lenscas]

Also, generally spoken lua ends up embedded in another language like rust rather than the other way around. Wouldn't it be easier to generate the teal types and function definition based on what is defined in rust?

I know a few projects that add typing information to mlua to be collected for definition file creation.(I made one of them which can directly create definition files for teal among other things).

That sounds an easier way to do things than going from teal to rust.

[jclavier44]

No I am not going from teal to rust but well embedding teal-lua script in Rust. First being able to get all teal variables (even local ones) and types will allow me to do hot reload. Second the interest of Teal in my project is also to be able to know which exact type of variables (object) and not any table so that I know which Rust object to bind with it and dynamically. It works well manually for now by making the list of variables and their types but now I need a way to get this automatically from teal...

For example, from this teal code:
local function createCounter(): (function(): number) --() -> int
local count: number = 0 -- This variable will be remembered by the closure
-- Define the closure
return function(): number
count = count + 1
return count
end
end

local var1: number = 1
local var2: string = 'hello'
local counter = createCounter()

I need this:
[
{
"function_name": "createCounter",
"Vars": [
{
"variable_name": "count",
"type": "number"
}
],
"kind": "local_function",
"Owner": ""
},
{
"variable_name": "var1",
"type": "number",
"kind": "local_declaration"
},
{
"variable_name": "var2",
"type": "string",
"kind": "local_declaration"
}
]

Right now I know how to get it from the AST, but maybe I need to dig in the functionalities of command "types"...

[jclavier44]

Ok I think I understand a bit more the structure of the json return from the "types" command. Can you just tell me what means the second number from the symbols, 7 in this example (from a teal program with first line being "local testTypeNumber1 = 1.0"):
[
1,
7,
"testTypeNumber1",
5
]

As well, when we create local variables in a function, we find it in the symbols but I don't get how we can find in which function it is declared...

[hishamhm]

Can you just tell me what means the second number from the symbols

The four entries are line, column, symbol name and symbol type code. The symbol type codes then map to entries in the "types" map, where they have a numeric code ("t") which can be decoded based on tl.typecodes info, and a textual representation ("str").

how we can find in which function it is declared

How are you scanning your functions for hot reloading? Are you using the Lua debug API? linedefined and lastlinedefined should give you the line ranges for functions.

[hishamhm]

@jclavier44 Keep in mind that being a source-to-source compiler, sometimes Teal does create additional temporary variables in the output Lua that do not exist in the source Teal. (It does very little of that currently, mainly for providing compatibility code, such as _tl_* variables for plugging in lua-compat-5.3) But that might change in the future. Also, macro expressions are currently very tame but they may cause surprises in the future. Just giving you a heads up on things to keep an eye on!

[jclavier44]

I am currently working on locals backups and restore, but yes I plan to use lua injection like debug.getlocal and debug.setlocal to handle locals in functions/programs.
From the output json of types command, I don't see linedefined and lastlinedefined giving me info of line ranges of functions

[hishamhm]

From the output json of types command, I don't see linedefined and lastlinedefined giving me info of line ranges of functions

That's available from Lua itself, via debug.getlocal.

[jclavier44]

The brackets "@{" and "@}" in the symbol list of types json actually helps to see where variables are declared. I am currently trying to extract all variables from the "types" json command output. I will give you my feedback in few days to see if I manage to get all the info I need without AST file.

[jclavier44]

I managed to extract mostly what I want from the types command output. I will open issue(s) or have a look on how works types command outputs to improve some info missing that I want. You can dismiss my PR about AST extraction.

[hishamhm]

@jclavier06 That is good news! Thank you for the update, and yes, do let us know if there's useful info that's missing in the tl types output.