fix #8405: -d:useNimRtl now works even when {.rtl.} procs are used at compile time; CTFFI now works with {dynlib}

[timotheecour]

• nim now allows using dynamic shared libraries at compile time when compiled with -d:nimHasLibFFI, extending the scope of Foreign function interface at Compile Time #10150 to cover {.dynlib:mylib.}, not just {.importc.} which was only for symbols dlsym'd from the main running executable

this is especially useful as we can now call arbitrary compiled user code at CT imported as a shared library (eg regex, or access global variables via wrappers, or to speedup CT operations by avoiding the VM interpretation, or writing to stderr at compile time)

• fixes Compile-time usage of parts of strutils fails when using -d:useNimRtl #8405 (issue was marked as closed but wasn't actually fixed AFAIK, see Compile-time usage of parts of strutils fails when using -d:useNimRtl #8405 (comment))
  /cc @awr1 @ccll
  /cc @zah because you asked about that bug here Foreign function interface at Compile Time #10150 (comment)

#8405 has been a major blocker for writing shared libraries / plugins: it prevented using any proc marked as {.rtl.} at compile time (more generally, any proc with {.importc.}); for example simply importing pkg/regex would fail with -d:useNimRtl because somewhere in the code, strutils.repeat is used at CT. (note that #10150 mitigates but doesn't fix this as explained here: #8405 (comment))

After this PR, importc procs with a body are executed in the VM as if importc wasn't specified: the body is compiled when used at compile time, and symbol is dynamically loaded when used at runtime.

nim can now be compiled with -d:useNimRtl, eg:

./bin/nim c -d:useNimRtl --skipUserCfg --skipParentCfg -o:bin/nim_temp1 -d:leanCompiler --passL:-Wl,-rpath,lib compiler/nim.nim

(whether the produced binary is a fully working nim compiler is a separate issue that can be addressed later)

whereas this would give errors before this PR.

example usage: CT FFI, dynlib, importc; with and without VM override

# t0464b.nim dynlib compiled with: `nim c -o:/tmp/libt0464b.dylib t0464b.nim`
proc fun1() {.exportc.} =
  echo "in fun1 v1"

proc fun2() {.exportc.} =
  echo "in fun2 v1"

var myvar {.exportc.}: int = 1234

proc set_myvar(a: int) {.exportc.} =
  myvar = a

proc get_myvar(): int {.exportc.} =
  myvar

# main.nim
proc c_exp1(a: float64): float64 {.importc: "exp", header: "<math.h>".}
proc c_exp2(a: float64): float64 {.importc: "exp", header: "<math.h>".} =
  echo "override exp"
  return 1234.0


proc c_printf1(frmt: cstring): cint {.importc: "printf", header: "<stdio.h>", varargs, discardable.}
proc c_printf2(frmt: cstring): cint {.importc: "printf", header: "<stdio.h>", varargs, discardable.} =
  echo "override printf"
  return 0

{.push importc, dynlib: "/tmp/libt0464b.dylib".}
proc fun1()
proc fun2() = echo "override fun1"
proc set_myvar(a: int)
proc get_myvar(): int
{.pop.}


proc main()=
  when defined(case_nim_has_ffi):
    echo c_exp1(1.0)
    c_printf1("hello world1\n")
    fun1()
    set_myvar(123)
    doAssert get_myvar() == 123

  echo c_exp2(1.0)
  c_printf2("hello world2\n")
  fun2()

static: main()
main()

output with nim c -r main.nim

override exp
1234.0
override printf
override fun1
Hint: operation successful (51925 lines compiled; 0.377 sec total; 58.875MiB peakmem; Debug Build) [SuccessX]
2.718281828459045
hello world2
in fun2 v1

output with nim c -r --experimental:compiletimeFFI -d: case_nim_has_ffi main.nim

(and nim built with -d:nimHasLibFFI)

2.718281828459045
hello world1
in fun1 v1
override exp
1234.0
override printf
override fun1
Hint: operation successful (51925 lines compiled; 0.480 sec total; 58.879MiB peakmem; Debug Build) [SuccessX]
2.718281828459045
hello world1
in fun1 v1
2.718281828459045
hello world2
in fun2 v1

note

• without -d:leanCompiler the above command gives: I get: docutils/highlite.nim(401, 15) Error: type mismatch: got <openarray[string], string, proc (a: string, b: string): int{.cdecl, noSideEffect, gcsafe, locks: 0.}>
• without the change to lib/pure/parseopt.nim , after recompiling ./bin/nim c lib/nimrtl.nim, and recompiling bin/nim_temp1 as above, running bin/nim_temp1 would error with: could not import: npocmdLineRest
• lib/system/repr.nim was also needed to prevent: lib/system/repr.nim(306, 3) Error: undeclared identifier: 'initReprClosure'

[Araq]

n[bodyPos] should not be nil. We need to find a better way to encode this state. I'm not sure why n[bodyPos].kind != nkEmpty (<-- we have a proc body here, so we can VM it) isn't sufficient.

[timotheecour]

n[bodyPos] should not be nil. We need to find a better way to encode this state. I'm not sure why n[bodyPos].kind != nkEmpty (<-- we have a proc body here, so we can VM it) isn't sufficient.

indeed; fixed it in aba1035 (just removed 1 condition which was a noop given we're in a scope that already has n[bodyPos].kind != nkEmpty)

also, I've updated PR description with example usage , showing behavior with and without CT FFI