macros: make quote a proper quasi-quoting operator

[zerbina]

Summary

quote now keeps the quoted block as is, which means that:

• no symbols are bound or mixed in automatically from the enclosing
  scopes
• identifiers in definition positions aren't turned into gensyms; they
  stay as is
• .gensym and .inject pragmas within the quoted block don't affect
  the AST

Symbols thus need to be bound or created explicitly, via bindSym and
genSym, respectively. This is a breaking change.

Details

Motivation For The Change

• symbols from the quote's scope being bound is error-prone, leading
  to confusing compilation errors
• the intention of documentation of quote already said it does quasi-
  quoting (even though it didn't)
• the implementation relied on the intricacies of templates and
  template evaluation

New Behaviour

• quoted AST is not modified. No symbols are bound and no identifiers
  are turned into gensyms

Implementation

• semQuoteAst transforms the quote call into a call to the internal
  quoteImpl procedure
• the pre-processed quoted AST is passed as the first arguments; the
  extracted unquoted expression are passed as the remaining arguments
• the internal-only evalToAst magic procedure is used for evaluating
  the unquoted expressions. newLit cannot be used here, as the trees
  it produces for object values are only valid when all the type's
  fields are exported
• placing the AST is of the evaluated unqouted expressions is handled
  in-VM, by quoteImpl

Standard Library And Test Changes

• multiple modules from the standard library relied on the previous
  symbol binding and gensym behaviour; they're changed to use bindSym
  or genSym. Outside-visible behaviour doesn't change
• the t7875.nim test relied on the gensym behaviour. The definition
  outside the macro is not relevant to the issue the test guards
  against, so it can just be removed
• the quoted ASTs in tsizeof.nim are wrapped in blocks in order to
  prevent the identifiers from colliding

[alaviss]

• identifiers in definition positions aren't turned into gensyms; they stay as is

Is there some rationale for this? I expect that this will make using quote a lot harder.

[zerbina]

Is there some rationale for this? I expect that this will make using quote a lot harder.

That's just a consequence of quote now doing quasi-quoting. Previously, quote was just sugar for a template definition + expansion, which is where the symbol binding and gensym behaviour came from.

[alaviss]

Are there plans to allow gensym to happen to unquoted identifiers in the future? quote was very handy in allowing for easy generation of helper statements without bloating my macro with a bunch of explicit genSyms

[zerbina]

Are there plans to allow gensym to happen to unquoted identifiers in the future?

No, at least not from my side. quote should only do quasi-quoting, nothing else. However, that doesn't preclude things such as allowing the .gensym pragma on definitions in macro output, or making macros open a new shadow scope by default.

Edit: For my answer, I've assumed that with "unquoted identifier" you mean "identifiers that are not accent-quoted".

In the meantime, you can either use genAst (which effectively works the same as the old quote, minus the unquoting behaviour), or use a custom quote operator, like so:

template gensymQuote(bl: untyped): untyped {.dirty.} =
  block:
    let name = genSym(nskTemplate, "templ")
    quote do:
      template `name`(): untyped =
        bl
      `name`()

Depending on the quoted AST, you can also just wrap the code in an if true: in order to confine the identifiers to their own scope.

[saem]

Per the discussion in matrix, concern around being able to get the old behaviour have been resolved per @zerbina's solution.

In the meantime, you can either use genAst (which effectively works the same as the old quote, minus the unquoting behaviour), or use a custom quote operator, like so:

template gensymQuote(bl: untyped): untyped {.dirty.} =
  block:
    let name = genSym(nskTemplate, "templ")
    quote do:
      template `name`(): untyped =
        bl
      `name`()

Depending on the quoted AST, you can also just wrap the code in an if true: in order to confine the identifiers to their own scope.

[zerbina]

/merge

[github-actions]

Merge requested by: @zerbina

Contents after the first section break of the PR description has been removed and preserved below:

---

To-Do

• write a proper commit message

Notes for Reviewers

• the test and library changes provide a good overview of how users of quote are affected