Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] Statically built Nix #56281

Closed
wants to merge 12 commits into from
Closed

[wip] Statically built Nix #56281

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b5c6080
stdenvAdapters: don’t separate debug info with static binaries
matthewbauer Feb 13, 2019
9b92e6e
busybox: fix musl builds
matthewbauer Feb 13, 2019
a44937b
krb5: remove static handling
matthewbauer Feb 13, 2019
d932883
boehm-gc: support static + musl
matthewbauer Feb 13, 2019
3cb7e9f
static: support some more libraries
matthewbauer Feb 13, 2019
634b049
brotli: add staticOnly option
matthewbauer Feb 13, 2019
f1a5fb8
nix: fix static building
matthewbauer Feb 13, 2019
350f3d0
top-level/impure.nix: expose crossOverlays
matthewbauer Mar 2, 2019
78bd45b
don’t use stackprotector for x86_32
matthewbauer Mar 28, 2019
f2290fb
systems: correct qemu architectures
matthewbauer Apr 17, 2019
dccd9e3
systems: fix emulator identity
matthewbauer Apr 17, 2019
3afa626
fix shell quoting
matthewbauer Apr 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions lib/systems/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,9 @@ rec {
else if final.isx86 then "i386"
else {
"powerpc" = "ppc";
"powerpcle" = "ppc";
"powerpc64" = "ppc64";
"powerpc64le" = "ppc64";
"mips64" = "mips";
"mipsel64" = "mipsel";
"powerpc64le" = "ppc64le";
}.${final.parsed.cpu.name} or final.parsed.cpu.name;

emulator = pkgs: let
Expand All @@ -103,7 +102,7 @@ rec {
in
if final.parsed.kernel.name == pkgs.stdenv.hostPlatform.parsed.kernel.name &&
pkgs.stdenv.hostPlatform.isCompatible final
then "${pkgs.runtimeShell} -c"
then "${pkgs.runtimeShell} -c '$@' --"
Ericson2314 marked this conversation as resolved.
Show resolved Hide resolved
else if final.isWindows
then "${wine}/bin/${wine-name}"
else if final.isLinux && pkgs.stdenv.hostPlatform.isLinux
Expand Down
4 changes: 4 additions & 0 deletions pkgs/build-support/cc-wrapper/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,10 @@ stdenv.mkDerivation {
hardening_unsupported_flags+=" stackprotector fortify"
''

+ optionalString targetPlatform.isx86_32 ''
hardening_unsupported_flags+=" stackprotector"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is temporary I assume?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe... binutils and musl seemed to get stuck. I don't think anyone supports i686 very well any more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that combination or in isolation? with targetPlatform.isx86_32 && targetPlatform.isMusl I'd be more comfortable with that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see #62817

''

+ optionalString (targetPlatform.libc == "newlib") ''
hardening_unsupported_flags+=" stackprotector fortify pie pic"
''
Expand Down
7 changes: 5 additions & 2 deletions pkgs/development/libraries/boehm-gc/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,17 @@ stdenv.mkDerivation rec {
export NIX_CFLAGS_COMPILE+=" -D_GNU_SOURCE -DUSE_MMAP -DHAVE_DL_ITERATE_PHDR"
'';

patches =
patches = [ (fetchpatch {
name = "boehm-gc-7.6.0-sys_select.patch";
url = "https://gitweb.gentoo.org/proj/musl.git/plain/dev-libs/boehm-gc/files/boehm-gc-7.6.0-sys_select.patch?id=85b6a600996bdd71162b357e9ba93d8559342432";
sha256 = "1gydwlklvci30f5dpp5ccw2p2qpph5y41r55wx9idamjlq66fbb3";
}) ] ++
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still needed? It was explicitly removed in a recent-ish PR with the idea that this didn't hurt but didn't help, IIRC. Might be good to check.

# https://github.com/ivmai/bdwgc/pull/208
lib.optional stdenv.hostPlatform.isRiscV ./riscv.patch;

configureFlags =
[ "--enable-cplusplus" ]
++ lib.optional enableLargeConfig "--enable-large-config"
++ lib.optional (stdenv.hostPlatform.libc == "musl") "--disable-static"
# Configure script can't detect whether C11 atomic intrinsics are available
# when cross-compiling, so it links to libatomic_ops, which has to be
# propagated to all dependencies. To avoid this, assume that the intrinsics
Expand Down
6 changes: 0 additions & 6 deletions pkgs/development/libraries/kerberos/krb5.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,6 @@

# Extra Arguments
, type ? ""
# This is called "staticOnly" because krb5 does not support
# builting both static and shared, see below.
, staticOnly ? false
}:

let
Expand All @@ -25,9 +22,6 @@ stdenv.mkDerivation rec {
outputs = [ "out" "dev" ];

configureFlags = [ "--with-tcl=no" "--localstatedir=/var/lib"]
# krb5's ./configure does not allow passing --enable-shared and --enable-static at the same time.
# See https://bbs.archlinux.org/viewtopic.php?pid=1576737#p1576737
++ optional staticOnly [ "--enable-static" "--disable-shared" ]
++ optional stdenv.isFreeBSD ''WARN_CFLAGS=""''
++ optionals (stdenv.buildPlatform != stdenv.hostPlatform)
[ "krb5_cv_attr_constructor_destructor=yes,yes"
Expand Down
6 changes: 1 addition & 5 deletions pkgs/os-specific/linux/busybox/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{ stdenv, lib, buildPackages, fetchurl
, enableStatic ? false
, enableMinimal ? false
, useMusl ? stdenv.hostPlatform.libc == "musl", musl
, useMusl ? stdenv.hostPlatform.libc == "musl"
, extraConfig ? ""
}:

Expand Down Expand Up @@ -88,10 +88,6 @@ stdenv.mkDerivation rec {
runHook postConfigure
'';

postConfigure = lib.optionalString useMusl ''
makeFlagsArray+=("CC=${stdenv.cc.targetPrefix}cc -isystem ${musl.dev}/include -B${musl}/lib -L${musl}/lib")
'';

depsBuildBuild = [ buildPackages.stdenv.cc ];

buildInputs = lib.optionals (enableStatic && !useMusl) [ stdenv.cc.libc stdenv.cc.libc.static ];
Expand Down
1 change: 1 addition & 0 deletions pkgs/stdenv/adapters.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ rec {
then throw "Cannot build fully static binaries on Darwin/macOS"
else stdenv'.mkDerivation (args // {
NIX_CFLAGS_LINK = toString (args.NIX_CFLAGS_LINK or "") + " -static";
separateDebugInfo = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem right, why must this be disabled for static adapter?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah maybe not... I was getting an issue that I thought was due to this but it might have been unrelated.

configureFlags = (args.configureFlags or []) ++ [
"--disable-shared" # brrr...
];
Expand Down
10 changes: 8 additions & 2 deletions pkgs/tools/compression/brotli/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{ stdenv, fetchFromGitHub, cmake }:
{ stdenv, fetchFromGitHub, cmake, fetchpatch, staticOnly ? false }:

# ?TODO: there's also python lib in there

Expand All @@ -15,7 +15,13 @@ stdenv.mkDerivation rec {

nativeBuildInputs = [ cmake ];

cmakeFlags = [];
patches = stdenv.lib.optional staticOnly (fetchpatch {
url = "https://github.com/google/brotli/pull/655/commits/7289e5a378ba13801996a84d89d8fe95c3fc4c11.patch";
sha256 = "1bghbdvj24jrvb0sqfdif9vwg7wx6pn8dvl6flkrcjkhpj0gi0jg";
});

cmakeFlags = []
++ stdenv.lib.optional staticOnly "-DBUILD_SHARED_LIBS=OFF";

outputs = [ "out" "dev" "lib" ];

Expand Down
10 changes: 6 additions & 4 deletions pkgs/tools/package-management/nix/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{ lib, fetchurl, fetchFromGitHub, callPackage
{ lib, fetchurl, fetchFromGitHub, fetchpatch, callPackage
, storeDir ? "/nix/store"
, stateDir ? "/nix/var"
, confDir ? "/etc"
Expand All @@ -9,7 +9,7 @@ let

common =
{ lib, stdenv, fetchurl, fetchpatch, perl, curl, bzip2, sqlite, openssl ? null, xz
, pkgconfig, boehmgc, perlPackages, libsodium, brotli, boost, editline
, pkgconfig, boehmgc, perlPackages, libsodium, brotli, boost, editline, libatomic_ops
, autoreconfHook, autoconf-archive, bison, flex, libxml2, libxslt, docbook5, docbook_xsl_ns
, busybox-sandbox-shell
, storeDir
Expand All @@ -31,14 +31,16 @@ common =

VERSION_SUFFIX = lib.optionalString fromGit suffix;

patches = [ ./static.patch ];

outputs = [ "out" "dev" "man" "doc" ];

nativeBuildInputs =
[ pkgconfig ]
++ lib.optionals (!is20) [ curl perl ]
++ lib.optionals fromGit [ autoreconfHook autoconf-archive bison flex libxml2 libxslt docbook5 docbook_xsl_ns ];
++ [ autoreconfHook autoconf-archive bison flex libxml2 libxslt docbook5 docbook_xsl_ns ];

buildInputs = [ curl openssl sqlite xz bzip2 ]
buildInputs = [ curl openssl sqlite xz bzip2 libatomic_ops ]
++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
++ lib.optionals is20 [ brotli boost editline ]
++ lib.optional withLibseccomp libseccomp
Expand Down
55 changes: 55 additions & 0 deletions pkgs/tools/package-management/nix/static.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
diff --git a/Makefile.config.in b/Makefile.config.in
index b01a4afb..59730b64 100644
--- a/Makefile.config.in
+++ b/Makefile.config.in
@@ -1,4 +1,6 @@
+AR = @AR@
BDW_GC_LIBS = @BDW_GC_LIBS@
+BUILD_SHARED_LIBS = @BUILD_SHARED_LIBS@
CC = @CC@
CFLAGS = @CFLAGS@
CXX = @CXX@
diff --git a/configure.ac b/configure.ac
index 5a252667..410b2097 100644
--- a/configure.ac
+++ b/configure.ac
@@ -64,6 +64,7 @@ AC_PROG_CXX
AC_PROG_CPP
AX_CXX_COMPILE_STDCXX_14

+AC_CHECK_TOOL([AR], [ar])

# Use 64-bit file system calls so that we can support files > 2 GiB.
AC_SYS_LARGEFILE
@@ -267,6 +268,15 @@ AC_ARG_WITH(sandbox-shell, AC_HELP_STRING([--with-sandbox-shell=PATH],
sandbox_shell=$withval)
AC_SUBST(sandbox_shell)

+AC_ARG_ENABLE(shared, AC_HELP_STRING([--enable-shared],
+ [Build shared libraries for Nix [default=yes]]),
+ shared=$enableval, shared=yes)
+if test "$shared" = yes; then
+ AC_SUBST(BUILD_SHARED_LIBS, 1, [Whether to build shared libraries.])
+else
+ AC_SUBST(BUILD_SHARED_LIBS, 0, [Whether to build shared libraries.])
+fi
+

# Expand all variables in config.status.
test "$prefix" = NONE && prefix=$ac_default_prefix
diff --git a/mk/libraries.mk b/mk/libraries.mk
index 14c95fa9..28173629 100644
--- a/mk/libraries.mk
+++ b/mk/libraries.mk
@@ -125,9 +125,9 @@ define build-library
$(1)_PATH := $$(_d)/$$($(1)_NAME).a

$$($(1)_PATH): $$($(1)_OBJS) | $$(_d)/
- $(trace-ar) ar crs $$@ $$?
+ $(trace-ar) $(AR) crs $$@ $$?

- $(1)_LDFLAGS_USE += $$($(1)_PATH) $$($(1)_LDFLAGS)
+ $(1)_LDFLAGS_USE += -Wl,--whole-archive $$($(1)_PATH) -Wl,--no-whole-archive $$($(1)_LDFLAGS)

$(1)_INSTALL_PATH := $$(libdir)/$$($(1)_NAME).a

4 changes: 3 additions & 1 deletion pkgs/top-level/impure.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ in
else overlays homeOverlaysDir
else []

, crossOverlays ? []

, ...
} @ args:

Expand All @@ -80,7 +82,7 @@ in
assert args ? localSystem -> !(args ? system || args ? platform);

import ./. (builtins.removeAttrs args [ "system" "platform" ] // {
inherit config overlays crossSystem;
inherit config overlays crossSystem crossOverlays;
# Fallback: Assume we are building packages on the current (build, in GNU
# Autotools parlance) system.
localSystem = (if args ? localSystem then {}
Expand Down
15 changes: 15 additions & 0 deletions pkgs/top-level/static.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,10 @@ in {
boost = super.boost.override {
enableStatic = true;
enableShared = false;

# Don’t use new stdenv for boost because it doesn’t like the
# --disable-shared flag
stdenv = super.stdenv;
};
gmp = super.gmp.override {
withStatic = true;
Expand Down Expand Up @@ -148,4 +152,15 @@ in {
};
};

brotli = super.brotli.override {
staticOnly = true;
};

curl = super.curl.override {
gssSupport = false;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would imagine your target audience for this is users who are running and deploying on rootless "enterprise" environments such a RHEL or CentOS. It's worth noting that in many of these setups, kerberos is used extensively for all HTTP and git authentication, so if you could get it working in this static variant it'd probably be worthwhile.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is an issue. libkrb5 doesn't have good way to link statically, however:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=439039

kerberos heimdal might support it though.


nix = (super.nix.override { withAWS = false; }).overrideAttrs (o: {
NIX_LDFLAGS = "-lssl -lbrotlicommon -lssh2 -lz -lnghttp2 -lcrypto -latomic";
});
}