Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/cgo: document how to cross-compile from darwin/amd64 to darwin/arm64 with cgo #44112

Closed
briandealwis opened this issue Feb 5, 2021 · 9 comments
Closed

cmd/cgo: document how to cross-compile from darwin/amd64 to darwin/arm64 with cgo #44112

briandealwis opened this issue Feb 5, 2021 · 9 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Documentation FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@briandealwis
Copy link

Attempting to use cgo for cross-compiling a Go app for darwin/arm64 from darwin/amd64 is not entirely straightforward for those not familiar with Apple's toolchain when compiling from macOS 10.15 or earlier.

What version of Go are you using (go version)?

go 1.16rc1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go1.16rc1 env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/bdealwis/Library/Caches/go-build"
GOENV="/Users/bdealwis/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/bdealwis/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/bdealwis/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/bdealwis/sdk/go1.16rc1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/bdealwis/sdk/go1.16rc1/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16rc1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/bdealwis/Projects/Skaffold/repo-skaffold/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/jp/fyjblw1x6k71gyjnz3smzbjr00kl65/T/go-build1743423011=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I was trying to build Skaffold for darwin/arm64. We normally build Skaffold with cgo to take advantage of github.com/rjeczalik/notify's bindings to native file-watching APIs like macOS's FSEvents.

I understood that XCode 12.2 and later support cross-compilation between amd64 and arm64, I thought this would be straightforward. But my attempts failed with odd errors:

$ CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 \
    go1.16rc1 build ./cmd/skaffold 
# runtime/cgo
In file included from _cgo_export.c:3:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdlib.h:62:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
In file included from _cgo_export.c:3:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdlib.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_types.h:27:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:33:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/machine/_types.h:34:2: error: architecture not supported
...

It turns out that cross-compiling for arm64 from amd64 requires using a macOS SDK 11.x. As I was building from a machine running macOS 10.15, the default SDK used for compilation was 10.15. It was not clear how to specify an 11.x SDK. From what I found online, it seemed like I should be able to build by specifying CC='clang --target arm64-apple-macos11, or CC='clang -isysroot to point to a macOS 11.x SDK, but these failed in the similar fashion:

$ CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 \
    CC='clang -isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk` \
    go1.16rc1 build -o out/skaffold-darwin-arm64 ./cmd/skaffold
# runtime/cgo
In file included from _cgo_export.c:3:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdlib.h:62:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
...

After much more digging, it turns out the best way is to set the SDKROOT environment variable to point to the SDK. And to use xcrun to find the newest SDK installed:

$ CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 \
    SDKROOT=$(xcrun --sdk macosx --show-sdk-path) \
    go1.16rc1 build -o out/skaffold-darwin-arm64 ./cmd/skaffold

The --sdk macosx chooses the MacOSX.sdk, which is normally linked to the latest SDK.
@seankhliao
Copy link
Member

feel free to add a page to the wiki

@seankhliao seankhliao changed the title Document how to cross-compile from darwin/amd64 to darwin/arm64 with cgo cmd/cgo: document how to cross-compile from darwin/amd64 to darwin/arm64 with cgo Feb 5, 2021
@bcmills
Copy link
Contributor

bcmills commented Feb 5, 2021

CC @cherrymui @FiloSottile @stevetraut

@cherrymui
Copy link
Member

I think the general question is how to cross compile with cgo, which boils down to how to cross compile C code. It is not specific to darwin. Basically, it needs a C cross compiler, and set CC and/or CGO_CFLAGS/CGO_LDFLAGS accordingly. Sometimes it is relatively easy, and sometimes it requires a nontrivial amount of work.

I don't think we want do document about how to cross compile C. Are there existing documentation about that? Maybe we can link to that, and add the part of setting CC/CGO_CFLAGS/etc.

@briandealwis
Copy link
Author

I thought it was worth calling out darwin/arm64 as most build systems do not yet support Apple M1 and are on macOS 10.15 or earlier. One of the wiki pages seems fine? Perhaps https://github.com/golang/go/wiki/Darwin?

@dmitshur dmitshur added this to the Backlog milestone Feb 8, 2021
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 8, 2021
bazbremner added a commit to bazbremner/aws-vault that referenced this issue Mar 26, 2021
@bazbremner
Set flags to cross-compile darwin/arm64 on darwin/amd64 and vice versa
845039b 
This is an attempt to fix a lack of keychain support in darwin/arm64
binaries that have been cross-compiled on other platforms as described
in 99designs#758 and hinted at in
the linked
99designs/keyring@756c48d

Given the keychain support from keyring[1] is provided by cgo, and CGO
is disabled by default in cross-compilation, we need to enable that,
and deal with dev tooling/libraries.

I dug this solution from the Go issues, specifically
golang/go#44112

Be warned, I am not familiar with the ins and outs of Go compilation,
especially when it comes to cross-compilation of CGO code, but at
least in this case, this change allows for a functional cross-compiled
binary.

I fully expect that attempting to cross-compile darwin/arm64 on
anything other than darwin/amd64 (or the opposite way around) is going
to end badly.

[1] https://github.com/99designs/keyring
bazbremner added a commit to bazbremner/aws-vault that referenced this issue Mar 26, 2021
@bazbremner
Set flags to cross-compile darwin/arm64 on darwin/amd64 and vice versa
e141e71 
This is an attempt to fix a lack of keychain support in darwin/arm64
binaries that have been cross-compiled on other platforms as described
in 99designs#758 and hinted at in
the linked
99designs/keyring@756c48d

Given the keychain support from keyring[1] is provided by cgo, and CGO
is disabled by default in cross-compilation, we need to enable that,
and deal with dev tooling/libraries.

I dug this solution from the Go issues, specifically
golang/go#44112

Be warned, I am not familiar with the ins and outs of Go compilation,
especially when it comes to cross-compilation of CGO code, but at
least in this case, this change allows for a functional cross-compiled
binary.

I fully expect that attempting to cross-compile darwin/arm64 on
anything other than darwin/amd64 (or the opposite way around) is going
to end badly.

[1] https://github.com/99designs/keyring
@bazbremner bazbremner mentioned this issue Mar 26, 2021
Merged
@geoah geoah mentioned this issue Mar 30, 2021
Merged
@mvdan mvdan mentioned this issue Sep 22, 2021
Open
@MuweiHe
Copy link

MuweiHe commented Oct 8, 2021

I chose MacOSX.sdk but still got unsupported arch errors:

In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdlib.h:62:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture

@epk epk mentioned this issue Oct 24, 2021
Merged
@rk0cc
Copy link

rk0cc commented Nov 1, 2021 •
edited
Loading

Any detailed information that do the same thing in GitHub Action? Since I don't sure the SDK path is the same form the action one.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
@mknyszek mknyszek moved this to Triage Backlog in Go Compiler / Runtime Jul 15, 2022
@WilliamDugdale WilliamDugdale mentioned this issue Nov 10, 2022
Open
@cytown
Copy link

cytown commented Dec 26, 2023

Is it still working on or just won't fix???

@cherrymui
Copy link
Member

cherrymui commented Dec 26, 2023 •
edited
Loading

Now in the usual case GOARCH=arm64 CGO_ENABLED=1 go build just works. Not sure if there is anything else need to be done.

In the usual case, with a standard Xcode installation, there is no need to set SDK path or anything (unless you're targeting iOS, not macOS).

@cherrymui cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Dec 26, 2023
@gopherbot
Copy link
Contributor

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 26, 2024
@github-project-automation github-project-automation bot moved this from Triage Backlog to Done in Go Compiler / Runtime Jan 26, 2024
@jmorganca jmorganca mentioned this issue Oct 13, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Documentation FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
9 participants
@cytown @briandealwis @dmitshur @bcmills @gopherbot @seankhliao @cherrymui @rk0cc @MuweiHe