Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Precompiled stdlib #768

Merged
merged 28 commits into from
Aug 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
19f6d01
compiler/driver: separate compilation for static exes
vyzo Aug 26, 2023
21f890b
build-spec: remove unnecessary cc options
vyzo Aug 27, 2023
88d6f58
std/foreign: smarter define-guard macro
vyzo Aug 27, 2023
fbe6de3
build-deps
vyzo Aug 27, 2023
ad1ebfd
script to build libgerbil.a
vyzo Aug 27, 2023
f062cd4
fix option folding
vyzo Aug 27, 2023
19dbb55
add build libgerbil step to build.sh
vyzo Aug 27, 2023
1aa6e13
compile static exes with saparete comp and libgerbil by default
vyzo Aug 27, 2023
a284da0
build libgerbil in CI
vyzo Aug 27, 2023
b8e41e4
remove unnecessary options from crypto-test compilation
vyzo Aug 27, 2023
d21e7ef
gxc: compile static exes by default
vyzo Aug 27, 2023
25a5651
gxc: fix -dynamic
vyzo Aug 27, 2023
51d5511
compiler/driver: expand output-bin path
vyzo Aug 27, 2023
6dc45e4
make: exe is now static-exe
vyzo Aug 27, 2023
cd7d2fc
os/temporaries: get rid of mktemp
vyzo Aug 27, 2023
d6255e7
tols: build dynamic executables
vyzo Aug 27, 2023
8df0a4a
build-deps
vyzo Aug 27, 2023
68e76dd
add basic gxc exe compilation tests
vyzo Aug 28, 2023
835a2f7
run gxc tests on CI
vyzo Aug 28, 2023
b788a2b
fix bug
vyzo Aug 28, 2023
83886cd
update gxc test
vyzo Aug 28, 2023
0515eae
update documentation about the new status quo
vyzo Aug 28, 2023
794609b
remove nonsensical sentence
vyzo Aug 28, 2023
e55921e
build-libgerbil: compile to .c's separately
vyzo Aug 29, 2023
aa9ab65
more separate compilation
vyzo Aug 29, 2023
0a6c867
prelude/core: add delete-file-or-directory
vyzo Aug 29, 2023
e3a1cbf
build static exe artifacts in a tmp directory to avoid races
vyzo Aug 29, 2023
51127aa
build-libgerbil: clean up intermediate compilation artifacts
vyzo Aug 29, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ jobs:
run: ./src/build.sh stage1 final
- name: Build Gerbil stdlib
run: ./src/build.sh stdlib
- name: Build Gerbil libgerbil
run: ./src/build.sh libgerbil
- name: Build Gerbil lang
run: ./src/build.sh lang
- name: Build Gerbil tools
Expand All @@ -65,4 +67,5 @@ jobs:
run: |
export GERBIL_HOME=${GITHUB_WORKSPACE}
export PATH=${GITHUB_WORKSPACE}/bin:$PATH
gxtest src/gerbil/test/...
gxtest src/std/...
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,3 @@ src/tutorial/lang/build-deps
src/tutorial/proxy/build-deps
src/tutorial/proxy/socks-proxy
src/tutorial/proxy/tcp-proxy
test/
68 changes: 1 addition & 67 deletions doc/guide/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ And we can build by invoking the script:
``` bash
$ chmod +x build.ss
$ ./build.ss
...
```

## Intermediate build scripts
Expand Down Expand Up @@ -94,70 +95,3 @@ $ ./build.ss
```

After the initial dependency graph generation, we can build during development by reusing the dependency graph and simply invoking ./build.ss. You only need to generate a new dependency graph if your import sets change.

## Building static executables

Static executables are simple to build:

- the executables are specified with the static-exe: build spec in place of exe:.
- the make invocation needs static: #t to be specified so that static compilation artifacts are built for modules.

However, there is a nuance: you usually don't want to build static executables with debug introspection as this will blow the executable size significantly.

Perhaps the simplest way to deal with the bloat issue is to have a separate step building the executables, while still compiling library modules with debug introspection for working in the repl.

The following build script breaks the build action into two steps, one for building library modules and another for building the executables:

``` scheme
#!/usr/bin/env gxi

(import :std/make)

;; the library module build specification
(def lib-build-spec
'("util"))

(def bin-build-spec
'((static-exe: "hello")))

;; the source directory anchor
(def srcdir
(path-normalize (path-directory (this-source-file))))

;; the main function of the script
(def (main . args)
(match args
(["lib"]
;; this action builds the library modules -- with static compilation artifacts
(make srcdir: srcdir
bindir: srcdir
optimize: #t
debug: 'src ; enable debugger introspection for library modules
static: #t ; generate static compilation artifacts; required!
prefix: "example"
;; build-deps: "build-deps" ; this value is the default
lib-build-spec))

(["bin"]
;; this action builds the static executables -- no debug introspection
(make srcdir: srcdir
bindir: srcdir
optimize: #t
debug: #f ; no debug bloat for executables
static: #t ; generate static compilation artifacts; required!
prefix: "example"
build-deps: "build-deps-bin" ; importantly, pick a file that differs from above
bin-build-spec))

;; this is the default action, builds libraries and executables
([]
(main "lib")
(main "bin"))))
```

Note that the `build-deps:` file is a cache that stores your project dependencies.
In large project, an up-to-date cache can save many seconds in build times.
When multiple projects share a same directory, they must be made to use separate
`build-deps:` file, or the caches will clash and be ineffective.
All but one of the projects must explicitly specify the `build-deps:` argument
to point to its own distinct file.
54 changes: 20 additions & 34 deletions doc/guide/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,12 +153,12 @@ representative for your top package namespace, like your github user id.

So, let's make a simple library module:
```
$ mkdir -p myproject/src/myuser
$ cd myproject/src/myuser
$ mkdir -p myproject
$ cd myproject

# Create a gerbil.pkg file for our project
$ cat > gerbil.pkg <<EOF
(package: myuser)
(package: myproject)
EOF

$ cat > mylib.ss <<EOF
Expand All @@ -173,67 +173,53 @@ You can change this by exporting the `GERBIL_PATH` variable.
You may also explicitly use the `-d` option;
but then you'll have to add your libdir to `GERBIL_LOADPATH`.
```
$ gxc mylib.ss
$ gxc -O mylib.ss
```

You now have a compiled module, which you can use in the interpreter:
```
$ gxi
> (import :myuser/mylib)
> (import :myproject/mylib)
> (hello "world")
hello world
```

Next let's make an executable:
```
$ cat > mybin.ss <<EOF
(import :myuser/mylib)
(import ./mylib)
(export main)
(def (main who)
(hello who))
EOF
```
and let's compile it and run it:
```
$ gxc -exe -o mybin mybin.ss
$ gxc -O -exe -o mybin mybin.ss
$ ./mybin world
hello world
```

Note that this is a dynamically linked executable, the module has been
Note that this is a statically linked executable.

If you want a dynamically linked executable, the module will be
compiled dynamically in the gerbil libdir and the executable is a stub
that loads it and executes main, which means that your `GERBIL_HOME`
(and `GERBIL_LOADPATH` if you are putting your artefacts in a different
place, like `myproject/lib`) must be set.
that loads it and executes main. This means that your `GERBIL_HOME`
must be set.

You can also compile a statically linked executable, which can work without
a local gerbil environment:
You can compile a dynamic executable with the `-dynamic` flag:
```
$ gxc -static mylib.ss # compile dependent library statically first
$ gxc -static -exe -o mybin-static mybin.ss
$ ./mybin-static world
hello world
$ gxc -dynamic -exe -o mybin mybin.ss
```

The advantage of static executables is that they can work without a local
Gerbil installation, which makes them suitable for binary distribution.
They also start a little faster, as there is no dynamic module loading at runtime.
In addition, because all dependencies from the stdlib are compiled in together, you
can apply global declarations like `(declare (not safe))` to the whole program, which
can result in significant performance gains. And as of `Gerbil-v0.13-DEV-50-gaf81bba`
the compiler performs full program optimization, resulting in further performance
And if you are willing to wait a bit for your proggram to compile, you
can specify `-full-program-optimization` which instructs the compiler
to perform full program optimization, resulting in further performance
benefits.

The downside is long compilation times and the limitation that the executable
won't be able to use the expander or the compiler, as the meta parts of the Gerbil
runtime are not linked in.

Note that when creating static executables, you will need to pass on options to
the linker if you're relying on foreign libraries. For example, to
include a dependency on `zlib`:
```
$ gxc -static -exe -o mybin-static -ld-options -lz mybin.ss
```

The `-ld-options` are being passed on to `gsc` which in turn adds the
specified options to the command that invokes the C linker.
The downside of static executables, is that they currently can't use
the gerbil expander and compiler packages. This restriction will be
lifted in a future release.
71 changes: 42 additions & 29 deletions doc/guide/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,24 +40,21 @@ hello world

The "hello world" executable:
```
$ cat > hello.ss <<EOF
package: example
$ cat > gerbil.pkg <<EOF
(package: example)
EOF

$ cat > hello.ss <<EOF
(export main)
(def (main . args)
(displayln "hello world"))
EOF
$ gxc -exe -o hello hello.ss
$ ./hello
hello world
```

And if you want a statically linked hello with no runtime dependencies to the Gerbil environment:
```
$ gxc -static -exe -o hello hello.ss
$ gxc -O -exe -o hello hello.ss
$ ./hello
hello world
```

## Core Gerbil
### Primitive forms
The standard Scheme primitive forms and macros are all supported.
Expand Down Expand Up @@ -769,8 +766,11 @@ to work.
For example, suppose we have a module example/hello.ss that we
want to compile as an executable module:
```
$ cat > example/gerbil.pkg <<EOF
(package: example)
EOF

$ cat > example/hello.ss <<EOF
package: example
(export main)
(def (main . args)
(displayln "hello world"))
Expand All @@ -783,34 +783,47 @@ runtime and module dependencies.
You can compile it to an executable with `gxc` with the
following command:
```
$ gxc -exe -o hello example/hello.ss
$ gxc -O -exe -o hello example/hello.ss
$ ./hello
hello world
```

If you want the compiler to perform full program optimization, then you can
specify the `-full-program-optimization` flag:
```
$ gxc -O -full-program-optimization -exe -o hello example/hello.ss
$ ./hello
hello world
```

You can also compile the executable with static linkage, which links
parts of the gerbil runtime and all dependent modules statically and
allows the executable to work without a local gerbil environment:
You can also compile the executable with dynamic linkage, which requires
a local gerbil installation at runtime.
```
$ gxc -static -exe -o hello example/hello.ss
$ gxc -O -dynamic -exe -o hello example/hello.ss
$ ./hello
hello world
```

The disadvantage of static linkage is that the executables are bigger,
and can only use the baseline parts of the gerbil runtime (gx-gambc0).
That means that static executables can't use the expander or the compiler,
as the meta levels of the gerbil runtime are not linked and initialized.
They also take quite longer to compile.

The advantage is that static executables don't require a local Gerbil
installation to work, which makes them suitable for binary distribution.
They also load faster, as they don't have to do any dynamic module loading.
Furthermore, because dependencies are compiled in together, you can apply
declarations like `(not safe)` to the whole program, resulting in potentially
significant performance gains. And as of `Gerbil-v0.13-DEV-50-gaf81bba` the
compiler performs full program optimization with tree shaking, which provides
further performance benefits.
The difference between the 3 executable compilation modes can be summarized as follows:
- By default, a statically linked executable is generated, linking to the precompiled
gerbil standard library. Note that the executable may some have dynamic library
dependencies from stdlib foreign code , and also links to `libgambit`.
If you have configured your gambit with `--enable-shared`, then this will be
a dynamic library dependency.
- When `-full-program-optimization` is passed to `gxc`, then the compiler will perform
full program optimization with all gerbil library dependencies. This will result
both in smaller executable size and better performance, albeit at the cost of
increased compilation time; this can be minutes for complex programs, while
separately linked executables compile in a second. Furthermore, because
dependencies are compiled in together, you can apply declarations like `(not safe)`
to the whole program using the `-prelude` directive. This can result
in potentially significant performance gains at the expense of safety.
- When `-dynamic` is passed to `gxc`, then a dynamic executable stub will be generated,
which will depend on the Gerbil runtime environment being present at execution time.
Dynamic executables do have some advantages over static executables however:
- they compile instantly and are tiny
- they can use the expander and the compiler; note that this restriction will be
lifted from static executables in a future release.

### Prelude Modules and Custom Languages

Expand Down
14 changes: 13 additions & 1 deletion src/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,14 @@ build_stdlib () {
(cd std && ./build.ss)
}

build_libgerbil () {
feedback_low "Building libgerbil"
PATH="${GERBIL_BASE}/bin:${PATH}"
GERBIL_HOME="${GERBIL_BASE}" #required for build
export PATH GERBIL_HOME
./build/build-libgerbil.ss
}

build_lang () {
feedback_low "Building gerbil languages"
PATH="${GERBIL_BASE}/bin:${PATH}"
Expand Down Expand Up @@ -226,6 +234,7 @@ build_gerbil() {
stage0 || die
stage1 final || die
build_stdlib || die
build_libgerbil || die
build_lang || die
build_r7rs_large || die
build_tools || die
Expand All @@ -252,6 +261,9 @@ else
"stdlib")
build_stdlib || die
;;
"libgerbil")
build_libgerbil || die
;;
"lang")
build_lang || die
;;
Expand All @@ -273,7 +285,7 @@ else
*)
feedback_err "Unknown command."
feedback_err \
"Correct usage: ./build.sh [gxi|stage0|stage1 [final]|stdlib|lang|r7rs-large|tools|tags]"
"Correct usage: ./build.sh [gxi|stage0|stage1 [final]|stdlib|libgerbil|lang|r7rs-large|tools|tags]"
die
;;
esac
Expand Down
Loading