Tinygo wasm file bigger than from regular go?

[refaktor]

Hi, I love what you are doing. I am making a programming language (https://github.com/refaktor/rye) in Go and I wanted to try compiling a stripped down version with tinygo, to see if it compiles. With some tweaking and removing various dependencies I managed to compile it, but it surprised me that the wasm file was 12MB in size, while wasm file compiled from regular go was 5MB. Tinygo compiles for really long time, but that doesn't bother me in this case.

Could it be that I did something wrong, os could some of the libraries I import be is somehow problematic? Is there a way to see or at least get a hint of what is taking so much space?

Best regards,
Janko

[dkegel-fastly]

Can you put the tweaked version in a branch somewhere we can try it?

[dkegel-fastly]

Also, the -size option does give some space breakdown, but requires the wasi target. Try something like

tinygo build -target wasi -size full -o main main.go

[refaktor]

Hi, thank you for your reply. I am trying to strip code of more/most deps ... to see if it makes any difference. The wasi option produces an os/signal error, even thought I don't see where I include that or parent library. If I don't figure out anything I will post stripped code online, but it's a little messy right now. Is it normal, that the compiler compiles for like half an hour (my codebase for the interpreter is not that big), or is this also the sign of the problem?

[dkegel-fastly]

At the moment, the compiler takes a lot longer than it should to run, especially when there are long init() functions.
(I put up with 10 minute builds currently.) I suspect it will be fixed sooner or later.

[refaktor]

Hi, I removed all external dependencies, even the only really cruicial one (peg-parser yhirose/go-peg) and compiled. Tingo wasm file is now 6.7MB, go wasm file is 3.7 MB. I posted this code here: https://github.com/refaktor/rye/tree/tinygo-test-purged

[aykevl]

Also, the -size option does give some space breakdown, but requires the wasi target.

It should work just fine with any WebAssembly target. If it doesn't, I'd be interested in a bug report :)

[dkegel-fastly]

I misspoke, just meant "it works with the wasi target but not with no target specified".

[dkegel-fastly]

Build instructions are in main branch's README but seem somewhat garbled when they try to specify build tags.
I couldn't quite build it with tinygo. Here's what I tried:

$ cd ~/go/src
$ git clone https://github.com/refaktor/rye
$ cd rye
$ git checkout tinygo-test-purged
$ export GO111MODULE=auto
$ go get github.com/refaktor/go-peg  # PEG parser (rye loader)
$ go get github.com/refaktor/liner   # library for REPL
$ go get golang.org/x/net/html       # for html parsin - will probably remove for b_tiny
$ go get github.com/pkg/profile      # for runtime profiling - will probably remove for b_tiny
$ go get github.com/pkg/term         # 
$ time go build -tags b_tiny
real	0m6.001s
$ ls -l rye
-rwxr-xr-x  1 dkegel  staff  8563872 Feb 20 08:40 rye
$ time tinygo build -tags b_tiny -o rye.tiny
../github.com/pkg/profile/profile.go:13:2: cannot find package "runtime/trace" in any of:
	/Users/dkegel/Library/Caches/tinygo/goroot-3f8a1098b62c88362a4f44d4819b37125b9458ef1ad4a13260086e9ce1cdbae3/src/runtime/trace (from $GOROOT)
	/Users/dkegel/go/src/runtime/trace (from $GOPATH)

so it's running into #1997. I suppose we could provide a stub runtime/trace for tinygo. But how did you get around that?

[refaktor]

Sorry, this is ad-hoc program removed of all dependencies. I didn't write any build instructions, I will add them. Basically, program doesn't yet use modules (longer story), so you need to do:

export GO111MODULE=auto

You don't need to run any of the go get commands since I removed all deps as far as I can see. And you don't need tags, so it should just be.

tinygo build -target wasm -size full -o main.wasm main.go
# or 
GOARCH=wasm GOOS=js go build 

I compiled with -size full and got:

------------------------------- | --------------- | -------
1856015       0   44041   74287 | 1900056  118328 | (unknown)
      8       0       0       0 |       8       0 | /home/runner/work/tinygo/tinygo/lib/wasi-libc/libc-top-half/musl/src/exit
   2929       0       0       0 |    2929       0 | /home/runner/work/tinygo/tinygo/lib/wasi-libc/libc-top-half/musl/src/string
    406       0       0       0 |     406       0 | encoding/base64
    126       0       0       0 |     126       0 | errors
  28341       0       0       0 |   28341       0 | fmt
    606       0       0       0 |     606       0 | internal/bytealg
    462       0       0       0 |     462       0 | internal/fmtsort
   1148       0       0       4 |    1148       4 | internal/task
    363       0       0       8 |     363       8 | main
    423       0       0       0 |     423       0 | math
    363       0      64       0 |     427      64 | math/bits
    409       0       0       0 |     409       0 | net/mail
   1330       0       0      12 |    1330      12 | os
  12133       0       0       0 |   12133       0 | reflect
  15269       0      12     156 |   15281     168 | runtime
2368961       0       0       0 | 2368961       0 | rye/env
1148186       0       0       0 | 1148186       0 | rye/evaldo
   1226       0       0       0 |    1226       0 | rye/util
   2854       0       0       0 |    2854       0 | sort
  22773       0   11320       0 |   34093   11320 | strconv
   8181       0     256       0 |    8437     256 | strings
    794       0       0       0 |     794       0 | sync
  14787       0     112      40 |   14899     152 | syscall/js
  28635       0     140      76 |   28775     216 | time
   1436       0     256       0 |    1692     256 | unicode
   2068       0     288       0 |    2356     288 | unicode/utf8
------------------------------- | --------------- | -------
5520232       0   56489   74583 | 5576721  131072 | total ```

[dkegel-fastly]

Ha, I was building main by accident. Works better if I actually build your branch :-)

Building it not-for-wasm for the moment, just to see how big the binary is. Corrected instructions:

$ cd ~/go/src
$ git clone https://github.com/refaktor/rye
$ cd rye
$ git checkout tinygo-test-purged
$ export GO111MODULE=auto
$ go build -o main
$ GOARCH=wasm GOOS=js go build  -o main.wasm
$ tinygo build -o main.tinygo
$ tinygo build -target wasm -size full -o main.tinygo.wasm main.go

Go takes about 6 seconds to build; tinygo takes about 32 minutes.
Interestingly, the tinygo wasm build took only 2 minutes. Hrm.

Result (roughly; actual commands maybe not quite as shown above):

-rwxr-xr-x  1 dkegel  staff  3259360 Feb 20 09:41 main
-rwxr-xr-x  1 dkegel  staff  3755963 Feb 20 09:47 main.wasm
-rwxr-xr-x  1 dkegel  staff  4184872 Feb 20 09:39 main.tinygo
-rwxr-xr-x  1 dkegel  staff  7231795 Feb 20 09:44 main.tinygo.wasm

and the size full output is like yours.

rye/env alone is 2.3 MB, which is pretty impressive bloat for 60KB of source code.

Does seem like something's fishy here.

[refaktor]

hm ... env/ is mostly mostly type definitions for various language objects and environment. The code is pretty usual I think. evaldo/ is the evaluator, and a bunch of builtin functions which could be more convoluted.

It is not urgent to me to get this solved right now. I would love to use smaller WASM binary and have an example of tinygo option of course, but it's not a pressing issue. I could try even further remove code and see if I hit a big difference in output size at some specific point ... other than that I can't really help with the tinygo compiler :P

[aykevl]

hm ... env/ is mostly mostly type definitions for various language objects and environment. The code is pretty usual I think.

I quickly scanned through it and also didn't see anything strange. Looks like pretty regular Go code to me. It certainly shouldn't blow up as it does at the moment.

[dkegel-fastly]

Someone could maybe look at the generated code.

Alternately, removing part of the source of env and measuring the resulting reduction in generated code size might give us a hint which part of the code is getting bloated.

Or removing code until the bloat goes away to find the smallest source tree that exhibits the awful bloat.

[dkegel-fastly]

$ time tinygo build  -o main.tinygo2 -opt 2 -cpuprofile foo2 main.go
real	33m19.591s
user	33m16.282s
sys	0m5.554s
$ go tool pprof foo2
Type: cpu
Time: Feb 20, 2022 at 2:05pm (PST)
Duration: 1999.50s, Total samples = 1522.82s (76.16%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1519.17s, 99.76% of 1522.82s total
Dropped 275 nodes (cum <= 7.61s)
Showing top 10 nodes out of 18
      flat  flat%   sum%        cum   cum%
  1504.29s 98.78% 98.78%   1504.32s 98.79%  runtime.cgocall
    14.87s  0.98% 99.76%     14.87s  0.98%  runtime.nanotime1
     0.01s 0.00066% 99.76%     16.52s  1.08%  runtime.findrunnable
         0     0% 99.76%     18.66s  1.23%  github.com/tinygo-org/tinygo/builder.Build.func2
         0     0% 99.76%   1485.79s 97.57%  github.com/tinygo-org/tinygo/builder.Build.func3
         0     0% 99.76%        18s  1.18%  github.com/tinygo-org/tinygo/builder.optimizeProgram
         0     0% 99.76%   1504.47s 98.79%  github.com/tinygo-org/tinygo/builder.runJob
         0     0% 99.76%     17.72s  1.16%  github.com/tinygo-org/tinygo/transform.Optimize
         0     0% 99.76%     16.64s  1.09%  runtime.mcall
         0     0% 99.76%     14.87s  0.98%  runtime.nanotime (inline)

So... I guess the slow bit is in llvm...

[refaktor]

I am trying to remove more code from env/ and recompile. I will let you know if I see any bigger change.

[aykevl]

@dkegel-fastly you could also try top -cum or web, they may give a better idea.

[dkegel-fastly]

(pprof) top -cum
Showing nodes accounting for 1504.29s, 98.78% of 1522.82s total
Dropped 275 nodes (cum <= 7.61s)
Showing top 10 nodes out of 18
      flat  flat%   sum%        cum   cum%
         0     0%     0%   1504.47s 98.79%  github.com/tinygo-org/tinygo/builder.runJob
  1504.29s 98.78% 98.78%   1504.32s 98.79%  runtime.cgocall
         0     0% 98.78%   1485.79s 97.57%  github.com/tinygo-org/tinygo/builder.Build.func3
         0     0% 98.78%   1485.79s 97.57%  tinygo.org/x/go-llvm.TargetMachine.EmitToMemoryBuffer
         0     0% 98.78%   1485.79s 97.57%  tinygo.org/x/go-llvm.TargetMachine.EmitToMemoryBuffer.func1
         0     0% 98.78%   1485.79s 97.57%  tinygo.org/x/go-llvm._Cfunc_LLVMTargetMachineEmitToMemoryBuffer
         0     0% 98.78%     18.66s  1.23%  github.com/tinygo-org/tinygo/builder.Build.func2
         0     0% 98.78%        18s  1.18%  github.com/tinygo-org/tinygo/builder.optimizeProgram
         0     0% 98.78%     17.72s  1.16%  github.com/tinygo-org/tinygo/transform.Optimize
         0     0% 98.78%     17.10s  1.12%  tinygo.org/x/go-llvm.PassManager.Run

[refaktor]

I removed some of the Objects in Rye, but it was linear reduction ... When I then removed some builtin functions (evaldo/builtins.go) which are defined as a map of objects, the reduction was larger. I now removed all builtin functions, to see if this huge map of objects/functions is problematic.

Before removing all builtin functions in evaldo/ (tinygo.wasm file 6.4MB vs. go.wasm 3.7MB)

   code  rodata    data     bss |   flash     ram | package
------------------------------- | --------------- | -------
1801993       0   43240   75088 | 1845233  118328 | (unknown)
      8       0       0       0 |       8       0 | /home/runner/work/tinygo/tinygo/lib/wasi-libc/libc-top-half/musl/src/exit
   2929       0       0       0 |    2929       0 | /home/runner/work/tinygo/tinygo/lib/wasi-libc/libc-top-half/musl/src/string
    406       0       0       0 |     406       0 | encoding/base64
    126       0       0       0 |     126       0 | errors
  28341       0       0       0 |   28341       0 | fmt
    606       0       0       0 |     606       0 | internal/bytealg
    462       0       0       0 |     462       0 | internal/fmtsort
   1148       0       0       4 |    1148       4 | internal/task
    363       0       0       8 |     363       8 | main
    423       0       0       0 |     423       0 | math
    363       0      64       0 |     427      64 | math/bits
    409       0       0       0 |     409       0 | net/mail
   1330       0       0      12 |    1330      12 | os
  12133       0       0       0 |   12133       0 | reflect
  15269       0      12     156 |   15281     168 | runtime
2226880       0       0       0 | 2226880       0 | rye/env
1141810       0       0       0 | 1141810       0 | rye/evaldo
   1226       0       0       0 |    1226       0 | rye/util
   2854       0       0       0 |    2854       0 | sort
  22773       0   11320       0 |   34093   11320 | strconv
   8181       0     256       0 |    8437     256 | strings
    794       0       0       0 |     794       0 | sync
  14787       0     112      40 |   14899     152 | syscall/js
  28635       0     140      76 |   28775     216 | time
   1436       0     256       0 |    1692     256 | unicode
   2068       0     288       0 |    2356     288 | unicode/utf8
------------------------------- | --------------- | -------
5317753       0   55688   75384 | 5373441  131072 | total

After removing all map[string]*env.Builtin from evaldo/ ... now sizes are comparable (tinygo.wasm file 2.1MB vs. go.wasm 2.4MB)

   code  rodata    data     bss |   flash     ram | package
------------------------------- | --------------- | -------
 506360       0   15408  103584 |  521768  118992 | (unknown)
      8       0       0       0 |       8       0 | /home/runner/work/tinygo/tinygo/lib/wasi-libc/libc-top-half/musl/src/exit
   2929       0       0       0 |    2929       0 | /home/runner/work/tinygo/tinygo/lib/wasi-libc/libc-top-half/musl/src/string
     35       0       0       0 |      35       0 | errors
  18489       0       0       0 |   18489       0 | fmt
    462       0       0       0 |     462       0 | internal/fmtsort
   1186       0       0       4 |    1186       4 | internal/task
    341       0       0       8 |     341       8 | main
    130       0       0       0 |     130       0 | math
    347       0      64       0 |     411      64 | math/bits
   1160       0       0      12 |    1160      12 | os
  10875       0       0       0 |   10875       0 | reflect
  12556       0      12     156 |   12568     168 | runtime
 855854       0       0       0 |  855854       0 | rye/env
 161699       0       0       0 |  161699       0 | rye/evaldo
  16601       0   11320       0 |   27921   11320 | strconv
   3666       0       0       0 |    3666       0 | strings
    177       0       0       0 |     177       0 | sync
  11541       0     112      40 |   11653     152 | syscall/js
     60       0       0      64 |      60      64 | time
    295       0       0       0 |     295       0 | unicode
   1577       0     288       0 |    1865     288 | unicode/utf8
------------------------------- | --------------- | -------
1606348       0   27204  103868 | 1633552  131072 | total

So this map could be a problem?

Is this around the reduction I should expect / hope for, or should it be smaller still given that now program doesn't really import anything big and also doesn't do much practically :) ... I have no clue but that 800k at rye/env still seems a lot to me for a bunch of type definitions. I could further remove the code there and see.

// edit: latest code is branch: tinygo-test-purged-2 https://github.com/refaktor/rye/tree/tinygo-test-purged-2

[dkegel-fastly]

The line

var builtins = map[string]*env.Builtin{

may be key to the slow compile times, though I still don't understand what's going on exactly.
When you run things at init time, a different and slower system (interp) takes over.

Try using lazy initialization of that instead of initializing it at init time.
(If that makes things better but leads to slow startup, you could then consider using wizer to work around the slow startup, at least on wasm.)

[refaktor]

In the latest test (link to brach is up there) there were no more var builtins = map[string]*env.Builtin{ ... which did produce similar even a little smaller size of .wasm as regular Go compiler, but I think it still took a long time. I didn't measure it exactly.

Otherwise, I wanted to not resort to code-generation too soon, so builtin functions are dynamically built now, hence this map[], but in future it might show better to have smart amount of codegen for builtin functions and they can then be statically defined, which might be better for the compiler?

[dkegel-fastly]

If you bring back the map, but only initialize it after main() starts, does that help the bloat problem?

(I hear rumblings about fixing the speed problem with interp, but there's nothing solid to point to yet, and it may be a while.)

[dgryski]

I wonder if the env issue is similar to what's happening in #2525 with all the time being spent in llvm ?