API: expose a way to programmatically create a typed environment for tl.load/process/etc.

[euclidianAce]

I'm currently working on my own cli to Teal for a few reasons

• Implementing some of the ideas expressed in Suggestions for the structure of the project moving forward #187
• Having a sort of testing grounds for prs to make here
• Selfishly implementing features that I currently need
• Selfishly having a place to make breaking changes to config formats when I don't like them

One of the biggest issues is that I want config files to be type checked by Teal (as well as tlconfig.lua here and an eventual build.tl). Right now the util module is pulling a lot of weight by doing some runtime type checking but this feels wrong.

So when tl.tl eventually gets split up, I think this is an important feature for tooling to have.

[lenscas]

I want config files to be type checked by Teal (as well as tlconfig.lua here and an eventual build.tl)
unless I am doing things wrong, the build.tl file should already be typed checked by teal itself rather than the cli. So should any teal files that are generated by it. If I am wrong, or do things wrong PLEASE let me know :)

As for type checking tlconfig.lua, perhaps its useful to make a library that can typecheck tables and use that as well as having the exact definition in location that can easily be used by everything? That way any tool can more easily typecheck the file.

Other alternatives I can think of is switching from lua to teal for it, which would allow the use of the compiler to typecheck it but means that the config file needs to be compiled before it can be used, which seems weird. Last option I can think of is to switch to json/some other format that supports schemas. Then any tool can typecheck the config file without having to depend on teal themselves, as a bonus vscode (and other editors) can already add autocomplete suggestions.

Personally, I'm not even sure why the config file is in lua except for it saving a dependency. It isn't like the tlconfig.lua is supposed to run code, is it? (Frankly, I don't even want to imagine the kind of horrors that people could/will turn their tlconfig.lua into if they start to use actual code in it....)

[euclidianAce]

(Sorry for the wall of text, but I've been thinking about solutions to this a lot! 😄)

Other alternatives I can think of is switching from lua to teal for it, which would allow the use of the compiler to typecheck it but means that the config file needs to be compiled before it can be used

For the compiler to type check it needs types defined somewhere, currently you can hackishly prepend some typedefs to the start of your source code before calling something like tl.process so that the type checker defines those types for you, but that's hackish.

having the exact definition in location that can easily be used by everything? That way any tool can more easily typecheck the file.

Or as you suggest we could have something like a .d.tl file for the config file types and have Teal load it before loading the config, but I am of the opinion that it would be better to define these types in statically verifiable code.
For example, I would prefer to load tlconfig.tl with something like this:

local function load_config()
   local types = tl.new_env{
      config = tl.new_type{
         kind = "record",
         keys = {
            source_dir = tl.types.string,
            build_dir = tl.types.string,
            include = tl.types.array(tl.types.string),
            -- you get the idea
         },
      },
   }
   -- somehow tell tl.load to typecheck the return value as the above type, etc., etc.
   return tl.load("./tlconfig.tl", _G, types)
end

(And somehow have the return type verified as well)
But there's not really a good way to do this from the tooling's/cli's perspective. For example, the way Teal defines the types in the standard library. Having a limited public API for this would help with programmatically generating types for use with tl.load and other more dynamic mechanisms. (Although this is maybe too dynamic for Teal's tastes), but it does take away concerns of having to rely on the filesystem to load some type definitions.

Personally, I like taking advantage of Lua's syntactic sugar for making config formats and for my cli, the config options are set by some globals that are injected into the config's environment

project "deps" {
   "luafilesystem"
}

build "options" {
   source_dir = "src",
   build_dir = "build"
}

Since the values are passed through what are basically glorified setters, they can be processed a little more nicely than iterating over a table and picking out the keys you recognize. I think type checking it is a bit easier since we don't have to tell tl.load that the return value of the chunk should be of a certian type, just that some global functions are defined of a certain type.

Personally, I'm not even sure why the config file is in lua except for it saving a dependency

Saving a dependency and Lua is good at being a configuration/data description language, it was designed with it in mind.

It isn't like the tlconfig.lua is supposed to run code, is it? (Frankly, I don't even want to imagine the kind of horrors that people could/will turn their tlconfig.lua into if they start to use actual code in it....)

I mean, Makefiles and the like could just put a sudo rm -rf / (or something equally destructive) in the middle of everything and people still use those without much trouble.
I personally just use an environment variable to store the location of wherever I've cloned the teal-types repo and use os.getenv to get it. But if things like conditional compilation and different build tools become available, why not have a turing complete config file?