Syntax for pragmas

[hishamhm]

So, I'm about to introduce pragmas to Teal (like Haskell's https://wiki.haskell.org/Language_Pragmas), to deal with the language evolution and environment differences in a more modular/gradual way (also inspired by https://github.com/seanbaxter/circle/blob/master/new-circle/README.md#versioning-with-feature-directives). See #723 (comment) for more Teal context)

But, unusually this time, I'm not feeling too inspired about the syntax bikeshedding part. I feel like it should be a "special comment" a la Haskell, but I'm not sure how they should look. (I find Haskell's uppercase LANGUAGE keyword a bit jarring)

In Teal, per-module pragmas will correspond to CLI compiler flags. For example, --feat-arity=on enables optional argument markers for stricter arity checks in function calls.

Right now it's implemented (https://github.com/teal-language/tl/pull/750/files) as

--!arity on

but I think it should be more distinct. Perhaps one of these?

--#pragma arity on
--!feat_arity on
--#feat arity on

Some combination of the above? Something else entirely? Any ideas?

[Andre-LA]

On the options indeed --#pragma arity on is my favorite.

I think using a "pragma" prefix is a more sustainable approach so I would advocate more on this specific idea.

Another possible one could be creating a specific syntax (since this isn't really just a comment) that would already imply pragmas, like taking the multicomment syntax, but using # instead of =:

--[#[
arity on
something off
another_thing on
]#]

[Andre-LA]

Just for the sake of sharing, in Nelua is literally setting a Lua table on the preprocessor (a line starting with ## runs on tue Lua preprocessor), similar to #ifdef in C.

## pragmas.nogc = true -- disables the GC

-- ...

[catwell]

I know that's not the question but I am not even sure it is a good idea to have such pragmas. I understand the arity example although you could just decide it is not optional starting from some version. Regarding the LuaJIT target I should be a build option (target) concern - similar to polyfills in JavaScript land - and not a language feature.

However if we do have pragmas I am 100% in favor of using Lua syntax for them.

[hishamhm]

Regarding the LuaJIT target I should be a build option (target) concern

The difficulty there is that there are differences in corner case semantics so using bit32 or lua-bitop changes the meaning of the Teal operators. So I want people to write down in their code "Yes, I want lua-bitop semantics in this module", so that they get lua-bitop semantics even if running the module in Lua 5.4.

If it wasn't for the semantics difference, I'd make it as an "automatic polyfill" like I've done for bit32 vs native Lua 5.3+ operators. Yes, there are differences between 32 and 64-bit semantics, but I had to draw the line at the point of word sizes, but the bit32 vs bitop differences are way more pronounced. (Remember the flame wars at the time of Lua 5.2 work versions? Rereading those messages recently was not a fun time.)

[euclidianAce]

I'm a big fan of how Circle does it so I think this is a good direction for something like Teal to take.

To suggest something non-specific, I'm not a huge fan of it being in what looks like a normal -- comment. I'd rather take some sigil that lua doesn't use like @, $, !, or `. (Though maybe not @ to not spam random people with notifications)

`feature arity on
!pragma arity on
$arity on

I'm partial to Ada's pragmas where it's just pragma followed by a procedure/function call but since pragma is a valid identifier it's probably not a good fit

pragma Feature (Arity => On)

and Rust as well

#![feature(arity)]

but I do like the function call-esque syntax of these so maybe something like

`pragma feature arity
`feature (arity)

or to go off the deep end and try to actually imitate a function call

`pragma feature { arity = true }
`pragma feature "arity"

Not super attached to any of these in particular, just my 2¢ on the bikeshed

[hishamhm]

Thank you all for the replies! The discussion helps clarifying the decision criteria!

Looking at all the suggestions, one bit that stood out to me is that one thing I would put as a requirement is that it needs to be very easy to parse at the lexer stage. What I mean by that: the Teal compiler does a lexer pass where it builds a list of tokens, then a parser pass where it builds a tree, then a typechecker pass where it traverses the tree. Even though in #750 I handled the pragma syntax at the parser pass and the pragma semantics at the typechecker pass, a pragma can potentially modify the lexer pass itself (e.g. by adding a new operator). So in practical terms, we may need to "parse during the lexer", so the simpler the structure is, the better.

I think that requirement rules out both using a plain soft-keyword such as pragma (which at the lexer stage is lexed as "identifier") and using full-blown Lua-like function syntax like feature { arity = true } which would effectively require tree-like parsing.

Now whether to do it as a pseudo-comment or not, I'm leaning on "yes" because it avoids taking up an unused sigil. And comments are handled at the lexer stage, so it fits the mental model of being something "outside of the parse tree".

Another point to keep in mind is to avoid confusion between pragmas and annotations. Things like Rust attributes (and its predecessor, C# attributes) attach to tree nodes. Pragmas are more like global compiler knobs (I mean, module-level at least).

So these options are on the table:

• using a simplified function syntax (for example, something that we could lex as marker identifier ( identifier ) in a simple 5-node state machine) or just a series of words (marker identifier identifer)
• using a name like pragma or feat or lang in the marker/next to the marker or none at all
• using on/off (to match the CLI) or true/false or just putting the pragma name without an on/off marker (but arity is on by default... even if it's --#pragma(arity) we need syntax for turning it off... --#pragma(not arity)? (I'm not sure if making it look like a full-blown expression is a good idea

As a side-note, maybe I'm just too new to Rust, but for some reason I always forget what's the right syntax for #![foo(bar)] annotations. It gives me that same type of confusion as "how do I write links in Markdown, again?" .

(One might argue that the arity pragma should operate at the lexer level and disallow the ? token for arity off, but I'm leaving it ignored in that case because it's harmless. This is intended to allow old code to run in new versions, not to help users of the new version to write code to run on older versions.)

[euclidianAce]

Of the given options I think the marker identifier identifier is probably the way to go (with room to maybe expand if needed to marker identifier simple-value-token-like-number-or-string or marker identifier ( non-paren-token+ )).

And to clarify why I don't like the sigil being so close to a comment is that it can create some confusion/ambiguity (not in the lexing/parsing sense, but the user-understanding-the-intent-of-the-feature sense). When I see --# I just assume that the pragma is just when there's a #foo in a comment and that this particular comment is just styled badly as there is usually a space after the --. This would definitely be helped by syntax highlighting, but I don't think that that is a good enough reason to use it.

My main sticking point with this is that I want malformed pragmas to be caught by the compiler, starting them with a comment sigil prevents that as they will then be semantically ignored.

Additionally, do these pragmas silently fail if they don't exist? This is helpful for backwards compatibility of things that don't change syntax but could also be a footgun in the case of something like miscapitalizing/misspelling a name:

--#pragma Some_safety_feature on

-- Oh boy, I can't wait to write code knowing that this safety
-- feature is enabled and I don't have to worry about $BAD_LANGUAGE_FEATURE now!
-- ...
-- ...Why is my code crashing? I should have gotten a compile error from this?

At the risk of more bikeshedding I think this can be helped by using an additional optional/required parameter to say, "use it if you have it" and "compile error if this is not a thing"

--#pragma Some_safety_feature on required        -- compile error since S is capital
--#pragma some_codegen_optimization on optional  -- silently fails, maybe generates a warning? (see -Wunknown-pragmas in gcc+clang)

[hishamhm]

Those are all good points.

The current implementation does fail on unknown pragmas.

As for syntax checking, the current draft PR treats --! as a single token, and then the parser consumes that and two identifiers (that's not ideal because it means the lexer/parser don't even check that they're all in the same line, but that was a PoC).

We could treat --#pragma (or --#feat...) as a single token (no space between the sigil and the word, unlike what's currently implemented in the draft PR), which would make it easy to syntax-highlight it accordingly. I think the parsing should make it behave like a single-line comment, consuming all tokens in the same line.

[catwell]

I understand that you need to process pragmas during the lexer pass, but in practice does it really matter how you parse the pragmas once you have identified it? I mean, once you have a pragma pseudo-comment, how you parse the contents of that pseudo-comment should be transparent from the point of view of the lexer. So I don't understand why it would be a problem to use Lua syntax.

And since you're running Lua anyway, you could possibly go as far as actually evaluating the "pragma section" in a separate environment (or with Teal itself, with support for pragmas disabled...), so you don't even need to deal with parsing. For instance I do something similar in my blog generator, the source material is djot and not Lua but I have blocks like this which I just parse like this.

[hishamhm]

how you parse the contents of that pseudo-comment should be transparent from the point of view of the lexer.

Yes, but it since the results of its evaluation may need to affect the lexer itself, it needs to happen at the lexer stage. I just don't want to implement a mini-Lua-function-call-parser inside the lexer. (Edit: unless it's seriously restricted!)

And since you're running Lua anyway, you could possibly go as far as actually evaluating the "pragma section" in a separate environment

Ooh, I'll avoid doing any kind of eval during the compiler execution for as long as I can. :) (That would give me so many additional things to think about; sandboxing, error handling, etc.; even if I tried to force a subset via custom lexing and environments.)

[hishamhm]

It seems people like the syntax to be more Lua-ish and possibly function-call-ish. How about this:

--#pragma arity(true)

and would people be okay with the fact this is not real Lua/Teal evaluation? (ie, that --#pragma arity(20 > 10) wouldn't work?)

[hishamhm]

Stepping back a little to look at the big picture, if this is the goal: (quoting myself)

This is intended to allow old code to run in new versions, not to help users of the new version to write code to run on older versions

...then adding a pragma might not be the right solution, because, for example, the module using the old arity rules will not have the annotation. It would make more sense to load the old module with compat options, say, just as an example, like:

-- just an example, I'm not advocating this
local old_module = require("old_module", { arity = off })

At that point, instead of being fine-grained like arity, it would be better to have something like lang = "teal2023". But I don't like the idea of extending require in a Lua-nonstardard way like that. It could be a better idea to have that kind of metadata in the project's tlconfig.lua. Both the plain tl compiler and Cyan read it, so that would be good place to specify which modules need custom lang settings.

However, another goal that inspired me to explore pragmas was to: (quoting myself again)

deal with the language evolution and environment differences in a more modular/gradual way

That is, so I can introduce experimental disabled-by-default changes to the language, and for that I think pragmas in the code would make sense.

So, right now, I'm thinking the best combo would be:

• introduce fine-grained feature pragmas like Pragmas #750
• introduce a more coarse lang flag/annotation which translates to a set of pragma settings
• metadata in tlconfig.lua to allow specifying both a project's lang and that of particular dependencies (i.e. so you can write Teal 2024 code using Teal 2023 dependencies which lack annotation themselves)

[hishamhm]

It would make more sense to load the old module with compat options

I have just updated the pragmas PR so that the following works:

--#pragma arity off
local old_module = require("old_module")
--#pragma arity on

...and the arity-off rules apply to functions exported by old_module even when called from within arity-on parts of the code (because the old_module does not have ? annotations). See the test cases for full examples.

This should make it easier for people to switch to new arity rules gradually!

[icy-lava]

I'm a bit late to the conversation, but perhaps you could take some inspiration from luacheck:

for _, elem in ipairs(node:select(query)) do -- luacheck: ignore 512
    do return elem end
end

(example from here):

-- luacheck: push ignore foo
foo() -- No warning.
-- luacheck: pop
foo() -- Warning is emitted.

So an example for teal:

-- teal: enable arity
-- teal: push disable arity
-- ... some code ...
-- teal: pop