cmd/link: add a flag to the linker to do not write function names to runtime.pclntab #36555

xaionaro · 2020-01-14T19:03:06Z

What version of Go are you using (`go version`)?

go version
go version devel +71154e061f Tue Jan 14 17:13:34 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/experiment0/.cache/go-build"
GOENV="/home/experiment0/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/experiment0/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/experiment0/.gimme/versions/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/experiment0/.gimme/versions/go/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build891839909=/tmp/go-build -gno-record-gcc-switches"

What did you do?

go get github.com/u-root/u-root
u-root -o /dev/null -build bb core boot
go get github.com/u-root/u-root/bb
go tool nm -size -sort size "$(go env GOPATH)"/bin/bb | head -1

What did you expect to see?

  e02ba0    3836272 r runtime.pclntab

What did you see instead?

  e39500    4808314 r runtime.pclntab

Proposal

I found here https://groups.google.com/d/msg/golang-nuts/hEdGYnqokZc/zQojaoWlAgAJ that a binary size could be reduced by avoiding of wasting space on function names within runtime.pclntab. So I propose to add a linker flag (for example -stripfnnames) to do that. We need such feature for applying Golang into embedded environments (with very limited total space).

I would like to prepare a PR if it will be approved.

The text was updated successfully, but these errors were encountered:

networkimprov · 2020-01-14T19:26:07Z

See #36313

xaionaro · 2020-01-14T20:06:19Z

@networkimprov : thank you. However I saw that ticket. It contains discussion how to reduce runtime.pclntab for a generic case. While my proposal is about a special case (embedded systems), so I created a new ticket :)

UPD: Wrote a comment to #36313 with a proposal to add an option to select a simple GC which does not require so complex runtime.pclntab. If the proposal will be approved then this ticket is indeed not actual.

ianlancetaylor · 2020-01-14T21:02:20Z

Removing function names from runtime.pclntab would break programs that use runtime.Callers, which is a lot of programs. For example, the testing and log packages would partially break.

xaionaro · 2020-01-14T21:10:41Z

Removing function names from runtime.pclntab would break programs that use runtime.Callers, which is a lot of programs. For example, the testing and log packages would partially break.

@ianlancetaylor :

It's only for embedded (where it's extremely important to reduce the size, and people are ready to sacrifice with log/testing for that). Moreover we can add a build-tag for -stripfnnames and use it to remove functions which uses runtime.Callers or to change their behavior. In my opinion it would good enough to just change the behavior to print <unknown> instead of function names.
If I understand correctly testing and log will actually continue to work even without modifications (but I did not look into to this part of the code, yet). They will just print emptiness instead of function names.

ianlancetaylor · 2020-01-14T21:28:30Z

I believe the testing.(*T).Helper function, for example, will break.

This will also likely break profiling information.

I understand that you would find it useful but I'm reluctant to endorse a change that will break code in ways that people will not know how to expect. I would rather find other ways to reduce binary size.

xaionaro · 2020-01-14T21:54:46Z

that will break code in ways that people will not know how to expect.

We can forbid using broken functions via the build-tag (added automatically by -stripfnnames).

networkimprov · 2020-01-14T21:57:13Z

Ian may be concerned that you'll eventually realize that Go isn't the best choice for an embedded app, and switch to Rust or http://ziglang.org ;-)

beoran · 2020-01-15T07:28:38Z

For embedded Go, where size is really important, for the time being, https://tinygo.org/ is the better solution.

Rather than having a specific build flag for stripping runtime.pclntab, I think that in the long run, what we need is new GOOS, such as "none" for running on raw iron, and "linux-embedded" for running an on embedded linux OS should be added. The "none" variants then allow building an OS kernel, and the -embedded variants then do a few things differently to save on space.

xaionaro · 2020-01-15T10:39:18Z

For embedded Go, where size is really important, for the time being, https://tinygo.org/ is the better solution.

I tried tinygo few times for u-root. And tinygo is a nice and interesting project, but it appears it has invalid CGo parser (which has problems with #define and other stuff), (if I remember correctly) does not support Golang's assembly language, it does not implement too much stuff from standard packages (which it overrides) and it even does not support something related to func() variables so it panics while compilation of u-root/bb. I've spent some time and realized that it's would require too much time to port the project to tinygo. Also tinygo's README.md says:

Non-goals: ...; Be able to compile every Go program out there.

Ian may be concerned that you'll eventually realize that Go isn't the best choice for an embedded app, and switch to Rust or http://ziglang.org ;-)

I believe the community of Golang is wise and will be able to find a solution for cases like mine :)
And I will be glad to somehow help to implement an approved solution when there will be one.

Anyway it would take much more time to port project to Rust than even to port it to tinygo :)

embeddedgo · 2020-12-06T16:15:40Z

@xaionaro I think we can try to introduce such -stripfnnames flag to the embeddedgo fork. Can you help?

I'd love to test such flag because I've just reached the limit of 1 MiB Flash in STM32L476RG:

$ size shell.elf
text data bss dec hex filename
1043876 3736 14004 1061616 1032f0 shell.elf

where pclntab is 345 KiB (34%):

$ nm --size-sort -S shell.elf|tail -5
08050104 00000fd4 T time.Time.AppendFormat
08059f98 00001444 T fmt.(*pp).printValue
200024e8 00001a74 B runtime.mheap_
08040ce0 00003728 T unicode.init
080a89c0 000565ac r runtime.pclntab

The embeddedgo fork adds support for noos/thumb, noos/riscv64 and linux/thumb.

I use linux/thumb only to run tests but it is usable and you can give it a try (no cgo support, sorry). It generates compressed Thumb2 instructions instead of ARM instructions so it produces smaller binaries that can be run on almost any ARMv7-A machine.

The noos ones are for bare metal programming.

xaionaro · 2020-12-07T11:30:56Z

I think we can try to introduce such -stripfnnames flag to the embeddedgo fork.

Sounds very interesting. It could become very useful for https://github.com/u-root/u-root.

But I do not understand: why do you do a fork instead of working with the upstream?

Can you help?

I would love to try. How can I help? :)

embeddedgo · 2020-12-07T12:27:02Z

why do you do a fork instead of working with the upstream?

You can much easily introduce new things in a fork. Even Go teem itself creates ~~forks~~ branches for for a bigger changes.

There is clearly no place for rarely used architectures or operating systems in the upstream but linux/thumb can be probably upstreamed as the 32-bit ARM seems to be now mainly used in embedded niche.

I would love to try. How can I help? :)

Simply send a pull request that implements -stripfnnames flag to the embeddedgo fork.

The embeddedgo compiler is released as patches to the original Go source so you can relatively easy see what was added/changed.

embeddedgo · 2020-12-12T12:22:42Z

@xaionaro I've just implemented stripfnn flag:

-stripfn int
    strip function names in pclntab, 1: remove package path, 2: blank names

Test results:

stripfn       text    data     bss
       0   1044754    3744   14004
       1   1026745    3744   14004
       2    973563    3744   14004

Sample panic outputs:

stripfn=1

panic: stripfn test

goroutine 1 [running]:
ramfs.(*dir).Close(0x2000b440, 0x20008f33, 0x0)
        dir.go:76 +0xaa
os.(*File).Close(0x2000a7c8, 0xffffffff, 0x20008070)
        file_noos.go:171 +0x1e
main.ls(0x20008050, 0x2, 0x2)
        ls.go:28 +0x70
main.runCmd(0x20008050, 0x2, 0x2)
        main.go:62 +0x80
main.main()
        main.go:25 +0x9e

stripfn=2

panic: stripfn test

goroutine 1 [running]:
(0x806d008, 0x8096f58)
        :1064 +0x3c6
(0x2000b600, 0x20008f33, 0x0)
        :76 +0xaa
(0x2000a7d8, 0xffffffff, 0x200080a0)
        :171 +0x1e
(0x20008080, 0x2, 0x2)
        :28 +0x70
(0x20008080, 0x2, 0x2)
        :62 +0x80
()
        :25 +0x9e
()
        :206 +0x11c
()
        :525 +0x2

In case of stripfn=2 panics are almost unusable. But in case of stripfn=1 there is enough information to find the problem.

Clone https://github.com/embeddedgo/go and try it.

dedo1911 · 2020-12-28T15:37:22Z

Would be nice to have stripfn flag in official Go too 🤔

eripi · 2021-08-24T03:39:29Z

@embeddedgo I clone https://github.com/embeddedgo/go and run all.bash to build go tools, when I do compile with stripfn=2 go build -ldflags '-s -w -stripfn 2' -o test main.go), when I run my test program, I get a stack of

panic: http new request fail

goroutine 1 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001f80c0, 0x0, 0x0, 0x0)
	:232 +0x532
go.uber.org/zap.(*Logger).Panic(0xc000196420, 0x5c5cba, 0x15, 0x0, 0x0, 0x0)
	:230 +0x85
main.main()
	:17 +0xc5

there are function names , my system is
Ubuntu 18.04.5 LTS \n \l

my test code is

package main

import "go.uber.org/zap"

var Olog *zap.Logger

func main() {
	opts := []zap.Option{zap.AddStacktrace(zap.PanicLevel)}

  logger, err := zap.NewProduction(opts...)
	if err != nil {
		panic("new logger error")
	}

	Olog = logger

   Olog.Panic("http new request fail")
}

I don't know why is there any configuration that needs to do?

embeddedgo · 2021-08-30T10:54:33Z

I didn't tested the stripfn flag much with go1.16.x but it seems with the last pcln changes it only strips the file names:

func A(s string) {
 B(s)
}

func B(s string) {
 panic(s)
}

func main() {
 A("hello!")
}

stripfn=0

panic: hello!

goroutine 1 [running]:
main.B(...)
 /home/michal/P/go/src/testy/hello_world/main.go:9
main.A(...)
 /home/michal/P/go/src/testy/hello_world/main.go:4
main.main()
 /home/michal/P/go/src/testy/hello_world/main.go:13 +0x51

stripfn=1

panic: hello!

goroutine 1 [running]:
main.B(...)
 main.go:9
main.A(...)
 main.go:4
main.main()
 main.go:13 +0x51

stripfn=2

panic: hello!

goroutine 1 [running]:
main.B(...)
 :9
main.A(...)
 :4
main.main()
 :13 +0x51

It seems the stripfn=1 shrinks the binary enough to meet my current requirements therefore i didn't investigated it further.

Bellow are the numbers for the above code (linux/amd64):

stripfn=0, text=814635 bytes
stripfn=1, text=810635 bytes
stripfn=2, text=809483 bytes

cagedmantis added this to the Backlog milestone Jan 15, 2020

cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 15, 2020

xaionaro mentioned this issue Dec 14, 2020

Remove pclntab u-root/u-root#1886

Open

ALTree mentioned this issue Dec 27, 2020

[Question] Opt out of GOPCLNTAB generation to combat reverse engineering binaries #43396

Closed

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022

mknyszek added this to Go Compiler / Runtime Jul 13, 2022

mknyszek moved this to Triage Backlog in Go Compiler / Runtime Jul 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/link: add a flag to the linker to do not write function names to runtime.pclntab #36555

cmd/link: add a flag to the linker to do not write function names to runtime.pclntab #36555

xaionaro commented Jan 14, 2020 •

edited

Loading

networkimprov commented Jan 14, 2020

xaionaro commented Jan 14, 2020 •

edited

Loading

ianlancetaylor commented Jan 14, 2020 •

edited by cespare

Loading

xaionaro commented Jan 14, 2020 •

edited

Loading

ianlancetaylor commented Jan 14, 2020

xaionaro commented Jan 14, 2020

networkimprov commented Jan 14, 2020

beoran commented Jan 15, 2020 •

edited

Loading

xaionaro commented Jan 15, 2020 •

edited

Loading

embeddedgo commented Dec 6, 2020 •

edited

Loading

xaionaro commented Dec 7, 2020 •

edited

Loading

embeddedgo commented Dec 7, 2020 •

edited

Loading

embeddedgo commented Dec 12, 2020 •

edited

Loading

dedo1911 commented Dec 28, 2020

eripi commented Aug 24, 2021

embeddedgo commented Aug 30, 2021

cmd/link: add a flag to the linker to do not write function names to runtime.pclntab #36555

cmd/link: add a flag to the linker to do not write function names to runtime.pclntab #36555

Comments

xaionaro commented Jan 14, 2020 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Proposal

networkimprov commented Jan 14, 2020

xaionaro commented Jan 14, 2020 • edited Loading

ianlancetaylor commented Jan 14, 2020 • edited by cespare Loading

xaionaro commented Jan 14, 2020 • edited Loading

ianlancetaylor commented Jan 14, 2020

xaionaro commented Jan 14, 2020

networkimprov commented Jan 14, 2020

beoran commented Jan 15, 2020 • edited Loading

xaionaro commented Jan 15, 2020 • edited Loading

embeddedgo commented Dec 6, 2020 • edited Loading

xaionaro commented Dec 7, 2020 • edited Loading

embeddedgo commented Dec 7, 2020 • edited Loading

embeddedgo commented Dec 12, 2020 • edited Loading

dedo1911 commented Dec 28, 2020

eripi commented Aug 24, 2021

embeddedgo commented Aug 30, 2021

xaionaro commented Jan 14, 2020 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

xaionaro commented Jan 14, 2020 •

edited

Loading

ianlancetaylor commented Jan 14, 2020 •

edited by cespare

Loading

xaionaro commented Jan 14, 2020 •

edited

Loading

beoran commented Jan 15, 2020 •

edited

Loading

xaionaro commented Jan 15, 2020 •

edited

Loading

embeddedgo commented Dec 6, 2020 •

edited

Loading

xaionaro commented Dec 7, 2020 •

edited

Loading

embeddedgo commented Dec 7, 2020 •

edited

Loading

embeddedgo commented Dec 12, 2020 •

edited

Loading