std/platformutils: platform reflection at CT: test whether platform has some feature (C apis, intrinsics, include files, etc)

[timotheecour]

proposal

Add a low-level (usable in other low-level modules) module std/backendutils with APIs to test whether platform has certain features (eg RDTSC, __builtin_saddll_overflow, C include files/libraries etc). The API is implemented in the compiler by executing shell commands, typically by compiling and/or running C code and checking exit code or stdout/stdin or other means, depending on the use case.

This provides similar functionality to what other build tools provide, eg cmake which runs tests like this: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/main/CMakeLists.txt producing those results: https://gist.github.com/timotheecour/8e11b7617ac332ba59f44ed829f7dc72

backendCompiles

# in std/platformutils
proc backendCompiles*(a: string, wrapInMain = false): bool {.compileTime.} =
 runnableExamples:
   assert backendCompiles("""#include <stdio.h>""")
   assert backendCompiles("""__int128 a;""", wrapInMain = true)
   assert backendCompiles("""NIM_STATIC_ASSERT(sizeof(NIM_BOOL) == 1, "");""", wrapInMain = true)

example 1:

avoid having to guess platforms on which to pass -d:nimEmulateOverflowChecks (and associated logic which spills over to nimbase.h via NIM_EmulateOverflowChecks)

instead, we'd write this code:

# lib/system/integerops.nim:
when backendCompiles("__builtin_ssubll_overflow(0, 0, NULL)", wrapInMain = true):
  proc nimSubInt64(a, b: int64; res: ptr int64): bool # implement using __builtin_ssubll_overflow
else:
  proc nimSubInt64(a, b: int64; res: ptr int64): bool = ... # fallback

example 2:

likewise, would simplify and make the detection logic more robust in bitops, and avoid the need for flags like -d:noIntrinsicsBitOpts

example 3:

check presence of RDTSC for std/cputicks (#411)

when backendCompiles("__rdtsc()", wrapInMain = true):
  # use __rdtsc timestamp counter
else:
  # use fallback implementation using `clock_gettime` or other timestamp counter

example 4:

for #399 std/int128s

when backendCompiles("""__int128 a;""", wrapInMain = true):
  # use __int128
else:
  # use emulation

other APIs

other APIs can be added in std/platformutils eg:

• to figure out a C enum value at CT, avoiding hard-coding its value in nim sources, which is fragile and tedious for cross-platform files whether the enum values differ (eg all the posix nim files)
• to figure out the OS version, nodejs version, backend C compiler version for currently selected C compiler, __STDC_VERSION__, etc

implementation

backendCompiles is implemented as a vmops which executes a shell command which compiles the provided code snippet and checks exit status, reusing the logic from extccomp.nim, in particular using same backend flags as would be used for compiling the main projects file.

This in particular is what makes this feature worth having implemented in compiler rather that letting user code call staticExec which would make it more difficult to pass the same backend flags or having to take care about caching, temporary file creation etc (the config logic shouldn't have to be duplicated by user).

caching

• the output will be cached within a nim compilation instance for a given set of arguments. As a further optional optimization, we can cache the output within a nimcache directory as a json file, so that it has essentially 0 impact on performance; we can simply store the results, possibly indexed via backend compilation flags (passc), and invalidate it when --forceBuild is passed. That said, compiling tiny C programs should not incur much overhead anyways.

[juancarlospaco]

sysutils ?.

[Varriount]

I support this, provided the module is created as an external module first, and then later evaluated for inclusion into the standard library.

[timotheecour]

it needs compiler support to implement the vmops, that's the whole point, so it can be available at CT; I don't see how an external package would work.

[Varriount]

Then perhaps some thought should be given as to how the compiler can support this kind of functionality for external modules.

How would this work for cross-compilation? Even with access to header files for another system, it's not (usually) possible to test at compile-time whether a target system supports certain intrinsics. The best you can do is generate versions of a function that are specialized for certain target systems, and dispatch to them at runtime.

[ringabout]

std/sysrand can benefit from this proposal. This proposal allows urandom fall back to other APIs. Then urandom can work in VM and won't affect users.

Ref nim-lang/Nim#17059 (which is already reverted)

[timotheecour]

Then perhaps some thought should be given as to how the compiler can support this kind of functionality for external modules.

that's precisely what {.vhmook.} is about, user-defined vmops, see timotheecour/Nim#598; however in the case of this RFC, the feature should be a compiler vmops since it's general purpose and would improve existing code in stdlib.

How would this work for cross-compilation?

cross-compilation is always going to be trickier as assumptions on target platform must be specified somehow;

• backendCompiles can still work as in this RFC (without custom logic) by calling the underlying backend with cross-compilation; this is already handled when you call nim c -d:mingw main or nim c --os:windows main anyways
• running a generated (say, C) binary (eg exe on posix) can still be done via wine/wine64 as done with nim r -d:mingw main (see my recent PR); so even this case can work when you the reflection code needs to run the binary instead of just compiling it
• in more complex cases (not captured by traditional cross-compilation toolchains) you'll have to specify assumptions on target platform/headers/cpu/libraries in some other way