Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crystal: 1.2 -> 1.7 #195606

Merged
merged 21 commits into from
Jan 20, 2023
Merged

crystal: 1.2 -> 1.7 #195606

merged 21 commits into from
Jan 20, 2023

Conversation

GeopJr
Copy link
Member

@GeopJr GeopJr commented Oct 12, 2022

Description of changes

Changelog

This is a copy of #173928 but for 1.2 -> 1.6 with some additional changes.

Changes:

Not sure if this should have been a review instead to the original draft but it targets different versions.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.11 Release Notes (or backporting 22.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot added 8.has: clean-up 8.has: package (new) This PR adds a new package labels Oct 12, 2022
@ofborg ofborg bot requested review from david50407, peterhoeg and manveru October 12, 2022 20:28
@GeopJr GeopJr marked this pull request as ready for review October 13, 2022 12:52
Copy link
Member

@peterhoeg peterhoeg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for taking this on. There are a few things related to the version we keep around as well as comments.

pkgs/development/compilers/crystal/default.nix Outdated Show resolved Hide resolved
pkgs/development/compilers/crystal/default.nix Outdated Show resolved Hide resolved
@peterhoeg peterhoeg mentioned this pull request Oct 22, 2022
13 tasks
feat: bump 1.6.x to 1.6.1
fix: fetch 12601 patch for 1.3.x-1.6.0
fix: darwin url for versions < 1.2.0
or > 1.5.x and aarch64-linux
@GeopJr
Copy link
Member Author

GeopJr commented Oct 22, 2022

I'm not really sure how to deal with the current error.

aarch64-darwin is broken for < 1.2.0
i686-linux builds are no longer available since 1.3.0
aarch64-linux are not yet available for >= 1.5.0

@GeopJr
Copy link
Member Author

GeopJr commented Oct 22, 2022

Also I'm still not super-convinced about compiling it in release mode, as straight-shoota mentioned:

And I don't think the compiler itself needs to be built in release mode. (though it might be relevant which LLVM version it uses)

It does take awfully long to compile the compiler AND the compiler spec AND run them when release mode is on.

];

LLVM_CONFIG = "${llvmPackages.llvm.dev}/bin/llvm-config";

FLAGS = [
"--release"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we no longer need to do this as you're patching the versions that do not support it.

Otherwise good to. Once this is reinstated I'll squash and merge.

Thanks a ton @GeopJr !!!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The release flag moved to makeFlags (which has the same effect as far as I am aware) from the #173928 PR. Is there another reason to include it in FLAGS?

@GeopJr
Copy link
Member Author

GeopJr commented Nov 14, 2022

ofborg still has an issue with the platforms that I am too unfamiliar with Nix to solve

@miangraham
Copy link
Contributor

miangraham commented Nov 15, 2022

Thanks for pushing this.

Looks like some failed depending pkgs need updates or pinning to 1.2.

Also I'm pretty sure invidious from current master will fail at runtime if crystal >= v1.5. I was out of date on this, looks like invidious official releases are on 1.5 and CI testing against 1.6 so they might be okay as-is or need to use 1.5 at worst. If I can free the time I'll try testing them against this in the next day or so.

Update: Invidious Tested.

  • Invidious from current nixpkgs/master configured via nixos module
  • Crystal 1.6.2 (default) from this PR added via overlay

Result: Invidious works great with this as-is. 🎉 Sign-up, login, subscriptions, trending, video viewing all work without error, and upgrading to this version from 1.2 finally cleans up some nasty log noise about unhandled SSL ctrl codes that's been a problem for ages. Ship it! ...at least when the other stuff is fixed! 😃


Result of nixpkgs-review pr 195606 run on x86_64-linux 1

3 packages failed to build:
  • ameba
  • mint
  • scry
13 packages built:
  • crystal (crystal_1_6)
  • crystal2nix
  • crystal_1_0
  • crystal_1_1
  • crystal_1_2
  • crystal_1_5
  • icr
  • invidious
  • kakoune-cr
  • lucky-cli
  • oq
  • shards (shards_0_17)
  • thicket

error: builder for '/nix/store/ycj6wir2c4gnbr0ia7n9j135lnc9apqs-scry-0.9.1.drv' failed with exit code 1;
       >  45 | exit(status, ex)
       >       ^---
       > Error: no overload matches 'Crystal.exit' with types Int32, (Exception | Nil)

error: builder for '/nix/store/wcvj565wm22lc9fz8xi4jg2ynn0a1p7w-mint-0.15.1.drv' failed with exit code 1;
       > In lib/kemal/src/kemal/static_file_handler.cr:55:25
       >  55 | last_modified = modification_time(file_path)
       >                       ^----------------
       > Error: undefined method 'modification_time' for Kemal::StaticFileHandler
 
error: builder for '/nix/store/v166ypiz7xh5hig1p0v7aqzw8h2annfz-ameba-1.0.1.drv' failed with exit code 2;
       > /nix/store/039g378vc3pc3dvi9dzdlrd0i4q93qwf-binutils-2.39/bin/ld: _main.o: in function `finalize':
       > /nix/store/5hpxpml8pjwizcixbdjzkrzkhcgmzban-crystal-1.6.2-lib/crystal/openssl/digest.cr:38: undefined reference to `EVP_MD_CTX_destroy'

@miangraham
Copy link
Contributor

miangraham commented Nov 16, 2022

Suggested fixes for downstream build errors:

scry and mint: Quickfix by using crystal 1.2 for now.

diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 49d1abbf9d05f..ed61424497ce7 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -13455,7 +13455,7 @@ with pkgs;
 
   icr = callPackage ../development/tools/icr { };
 
-  scry = callPackage ../development/tools/scry { };
+  scry = callPackage ../development/tools/scry { crystal = crystal_1_2; };
 
   dasm = callPackage ../development/compilers/dasm { };
 
@@ -14500,7 +14500,7 @@ with pkgs;
 
   millet = callPackage ../development/tools/millet {};
 
-  mint = callPackage ../development/compilers/mint { };
+  mint = callPackage ../development/compilers/mint { crystal = crystal_1_2; };
 
   mitscheme = callPackage ../development/compilers/mit-scheme
     { stdenv = gcc10StdenvCompat; texLive = texlive.combine { inherit (texlive) scheme-small epsf texinfo; }; };

ameba: 1.0.1 -> 1.3.1. Couldn't immediately get 1.0.1 to build on old crystal. 1.3.1 builds fine on new crystal.

diff --git a/pkgs/development/tools/ameba/default.nix b/pkgs/development/tools/ameba/default.nix
index 4239f5c0056a9..27068cbc4c49f 100644
--- a/pkgs/development/tools/ameba/default.nix
+++ b/pkgs/development/tools/ameba/default.nix
@@ -2,13 +2,13 @@
 
 crystal.buildCrystalPackage rec {
   pname = "ameba";
-  version = "1.0.1";
+  version = "1.3.1";
 
   src = fetchFromGitHub {
     owner = "crystal-ameba";
     repo = "ameba";
     rev = "v${version}";
-    hash = "sha256-dvhGk6IbSV3pxtoIV7+0+qf47hz2TooPhsSwFd2+xkw=";
+    hash = "sha256-SZ2sBQeZgtPOYioH9eK5MveFtWVGPvgKMrqsCfjoRGM=";
   };
 
   format = "make";

0 packages fail to build with the above.

@GeopJr
Copy link
Member Author

GeopJr commented Dec 18, 2022

I have nix installed on an arm mac right now. I'm not sure how to test things in a nixpkgs pull request though, but if you tell me what to run I can try it out if that helps.

I believe you can use nixpkgs-review pr 195606 (or nix-shell -p nixpkgs-review --run "nixpkgs-review pr 195606"). However, ofborg (ci) will test on darwin anyway (it will just take some time).

@miangraham
Copy link
Contributor

miangraham commented Dec 19, 2022

Apologies if you know the below already. Throwing in an x86 darwin build of the latest here. Looks like a similar ld error.

ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv.dylib) was built for newer macOS version (10.12) than being linked (10.11)
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/ii163yvg5qxqqmfczyadb05z51wjnqpn-source
source root is source
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
build flags: -j12 SHELL=/nix/store/2198gb5ws3cyma9cxrx3clq6p83781kc-bash-5.1-p16/bin/bash CRYSTAL_CONFIG_VERSION=1.2.2 progress=1 all docs release=1
Using /nix/store/5axyk93lvvxwzjpsgwrihzmzw4lxm6cm-llvm-13.0.1-dev/bin/llvm-config [version=13.0.1]
clang++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/nix/store/5axyk93lvvxwzjpsgwrihzmzw4lxm6cm-llvm-13.0.1-dev/include -std=c++14 -stdlib=libc++  -fno-exceptions -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS ->CRYSTAL_CONFIG_BUILD_COMMIT="" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1671412426" CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build --single-module --release --progress --t>[1/13] Parse                             ^M[1/13] Parse                             ^M[2/13] Semantic (top level)              ^M[2/13] Semantic (top level)              ^M[3/13] Semantic (new)                    ^M[3/13] Sema>ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv.dylib) was built for newer macOS version (10.12) than being linked (10.11)
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv-nocharset.dylib) was built for newer macOS version (10.12) than being linked (10.11)
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libcharset.1.dylib) was built for newer macOS version (10.12) than being linked (10.11)
Undefined symbols for architecture x86_64:
  "___darwin_check_fd_set_overflow", referenced from:
      _select_add in libevent.a(select.o)
      _select_del in libevent.a(select.o)
      _select_dispatch in libevent.a(select.o)
ld: symbol(s) not found for architecture x86_64
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `clang "${@}" -o /private/tmp/nix-build-crystal-1.2.2.drv-0/tmp.COQvbMvS2e/private-tmp-nix-build-crystal-1.2.2.drv-0-source-src-ecr-process.cr/macro_run  -rdynamic -L/nix/store/>make: *** [Makefile:168: .build/crystal] Error 1

Result of nixpkgs-review pr 195606 run on x86_64-darwin 1

13 packages failed to build:
  • ameba
  • crystal (crystal_1_6)
  • crystal2nix
  • crystal_1_2
  • icr
  • invidious
  • kakoune-cr
  • lucky-cli
  • mint
  • oq
  • scry
  • shards (shards_0_17)
  • thicket

@@ -13889,18 +13889,17 @@ with pkgs;
};

inherit (callPackages ../development/compilers/crystal {
llvmPackages = if stdenv.system == "aarch64-darwin" then llvmPackages_11 else llvmPackages_10;
llvmPackages = llvmPackages_13;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something I don't get from the x86-darwing error is that it refers to clang-11, but shouldn't it refer to clang-13?

Maybe we could try sticking with llvmPackages_11 here? I'm not sure for the underlying reason, but the llvm package in nixpkgs points to 11.

Or we are missing some path to use llvmPackages_13.clang

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:/ Closest discussion I could find: https://discourse.nixos.org/t/nixpkgs-aarch64-darwin-build-m1-uses-wrong-old-clang-how-to-fix/ (in our case x86_64 instead of aarch64)

the suggested change adds clang-unwrapped to PATH

${if stdenv.isDarwin && stdenv.isAarch64 then ''
# We need the unwrapped binutils and clang:
# We also want to build a fat library with x86_64, arm64, arm64e in there.
# Because we use the unwrapped tools, we need to provide -isystem for headers
# and the library search directory for libdl.
# We can't build this on x86_64, because the libSystem we point to doesn't
# like arm64(e).
PATH=${bintools-unwrapped}/bin:${llvmPackages_13.clang-unwrapped}/bin:$PATH \
clang -arch x86_64 -arch arm64 -arch arm64e \
-isystem ${llvmPackages_13.clang.libc}/include \
-isystem ${llvmPackages_13.libclang.lib}/lib/clang/*/include \
-L${llvmPackages_13.clang.libc}/lib \
-Wl,-install_name,$libName \
-Wall -std=c99 -O3 -fPIC libredirect.c \
-ldl -shared -o "$libName"

Someone on darwin needs to give it a try otherwise I'll just revert to llvmPackages_11 for x86_64

The nix build definitely downloaded clang-13:

...
/nix/store/5b4s502i0ifbivjfb0pwxjlf7vrwgr2v-clang-13.0.1-lib
...
/nix/store/dx1q2d5zh3f45y8amkisy5lbq9b48zsx-clang-13.0.1
...
/nix/store/myqkicxg2zg8wmc84k048hn6fznilqm9-clang-wrapper-13.0.1
...

https://logs.nix.ci/?attempt_id=65e680ba-2ce4-4aae-8375-2b4107302113&key=nixos%2Fnixpkgs.195606

@bcardiff
Copy link
Contributor

On macOS 12.6.1 (Monterrey)

nixpkgs % uname -a
Darwin galera.local 21.6.0 Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64 x86_64 i386 MacBookPro16,1 Darwin

Building a979bd9214a622ef444a20e2023d37c268c43ea4 I got

% nix-build -A crystal .
these derivations will be built:
  /nix/store/pwlhra7iy35xljqgrb9sqa7i2i4z91m4-crystal-1.6.2.drv
building '/nix/store/pwlhra7iy35xljqgrb9sqa7i2i4z91m4-crystal-1.6.2.drv'...
unpacking sources
unpacking source archive /nix/store/n5zijddfgjn4vrl0nwl7b7h9190wr5ff-source
source root is source
patching sources
applying patch /nix/store/jf31m11h0h5xpmgzpa2a4j0dm7kjsqyh-tzdata.patch
patching file src/crystal/system/unix/time.cr
configuring
no configure script, doing nothing
building
build flags: -j16 SHELL=/nix/store/2198gb5ws3cyma9cxrx3clq6p83781kc-bash-5.1-p16/bin/bash CRYSTAL_CONFIG_VERSION=1.6.2 progress=1 all docs release=1
Using /nix/store/5axyk93lvvxwzjpsgwrihzmzw4lxm6cm-llvm-13.0.1-dev/bin/llvm-config [version= 13.0.1]
clang++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/nix/store/5axyk93lvvxwzjpsgwrihzmzw4lxm6cm-llvm-13.0.1-dev/include -std=c++14 -stdlib=libc++  -fno-exceptions -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1672089671"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build --single-module -D strict_multi_assign --release --progress --threads 16 -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
ld: warning: directory not found for option '-L/nix/store/63z68smi2s24nyzhl9l5iv1qpp3rsgn4-crystal-1.6.2-lib/crystal'
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv.dylib) was built for newer macOS version (10.12) than being linked (10.11)
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv-nocharset.dylib) was built for newer macOS version (10.12) than being linked (10.11)
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libcharset.1.dylib) was built for newer macOS version (10.12) than being linked (10.11)
Undefined symbols for architecture x86_64:
  "___darwin_check_fd_set_overflow", referenced from:
      _select_add in libevent.a(select.o)
      _select_del in libevent.a(select.o)
      _select_dispatch in libevent.a(select.o)
ld: symbol(s) not found for architecture x86_64
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `clang "${@}" -o /private/tmp/nix-build-crystal-1.6.2.drv-0/tmp.QcGXtyzn7v/private-tmp-nix-build-crystal-1.6.2.drv-0-source-src-ecr-process.cr/macro_run  -rdynamic -L/nix/store/63z68smi2s24nyzhl9l5iv1qpp3rsgn4-crystal-1.6.2-lib/crystal -L/nix/store/6yny71ym1icg45rvfwgn6ip5l4bwksq7-crystal-binary-1.2.2/embedded/lib -lgc -L/nix/store/mykvqcspihpk61a8pw0dfnkm615kmksl-libevent-2.1.12/lib -levent -liconv`
make: *** [Makefile:189: .build/crystal] Error 1
builder for '/nix/store/pwlhra7iy35xljqgrb9sqa7i2i4z91m4-crystal-1.6.2.drv' failed with exit code 2
error: build of '/nix/store/pwlhra7iy35xljqgrb9sqa7i2i4z91m4-crystal-1.6.2.drv' failed

We can see that the missing ___darwin_check_fd_set_overflow is there and that clang-11 is used despite the llvmPackages = llvmPackages_13 in all-packages.nix

On top of this branch I attempted to use llvm 13 clang by setting the CC env variable:

nixpkgs % git diff
diff --git a/pkgs/development/compilers/crystal/default.nix b/pkgs/development/compilers/crystal/default.nix
index e642e8ee7bc..f8593a2dccf 100644
--- a/pkgs/development/compilers/crystal/default.nix
+++ b/pkgs/development/compilers/crystal/default.nix
@@ -165,6 +165,7 @@ let
         export threads=$NIX_BUILD_CORES
         export CRYSTAL_CACHE_DIR=$TMP
         export MACOSX_DEPLOYMENT_TARGET=10.11
+        export CC=${llvmPackages.clang-unwrapped}/bin/clang
       '';
nixpkgs % nix-build -A crystal .                      
these derivations will be built:
  /nix/store/d63pfq147brgjdydzp3y0i4x2mf6c89n-crystal-1.6.2.drv
building '/nix/store/d63pfq147brgjdydzp3y0i4x2mf6c89n-crystal-1.6.2.drv'...
unpacking sources
unpacking source archive /nix/store/n5zijddfgjn4vrl0nwl7b7h9190wr5ff-source
source root is source
patching sources
applying patch /nix/store/jf31m11h0h5xpmgzpa2a4j0dm7kjsqyh-tzdata.patch
patching file src/crystal/system/unix/time.cr
configuring
no configure script, doing nothing
building
build flags: -j16 SHELL=/nix/store/2198gb5ws3cyma9cxrx3clq6p83781kc-bash-5.1-p16/bin/bash CRYSTAL_CONFIG_VERSION=1.6.2 progress=1 all docs release=1
Using /nix/store/5axyk93lvvxwzjpsgwrihzmzw4lxm6cm-llvm-13.0.1-dev/bin/llvm-config [version= 13.0.1]
clang++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/nix/store/5axyk93lvvxwzjpsgwrihzmzw4lxm6cm-llvm-13.0.1-dev/include -std=c++14 -stdlib=libc++  -fno-exceptions -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1672064457"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build --single-module -D strict_multi_assign --release --progress --threads 16 -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
ld: warning: directory not found for option '-L/nix/store/6ci2jz3hm2q498x5yq2zfd4jih1jhr54-crystal-1.6.2-lib/crystal'
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv.dylib) was built for newer macOS version (10.12) than being linked (10.11)
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv-nocharset.dylib) was built for newer macOS version (10.12) than being linked (10.11)
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libcharset.1.dylib) was built for newer macOS version (10.12) than being linked (10.11)
Undefined symbols for architecture x86_64:
  "___darwin_check_fd_set_overflow", referenced from:
      _select_add in libevent.a(select.o)
      _select_del in libevent.a(select.o)
      _select_dispatch in libevent.a(select.o)
ld: symbol(s) not found for architecture x86_64
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `/nix/store/dx1q2d5zh3f45y8amkisy5lbq9b48zsx-clang-13.0.1/bin/clang "${@}" -o /private/tmp/nix-build-crystal-1.6.2.drv-0/tmp.rOLzPfAfOq/private-tmp-nix-build-crystal-1.6.2.drv-0-source-src-ecr-process.cr/macro_run  -rdynamic -L/nix/store/6ci2jz3hm2q498x5yq2zfd4jih1jhr54-crystal-1.6.2-lib/crystal -L/nix/store/6yny71ym1icg45rvfwgn6ip5l4bwksq7-crystal-binary-1.2.2/embedded/lib -lgc -L/nix/store/mykvqcspihpk61a8pw0dfnkm615kmksl-libevent-2.1.12/lib -levent -liconv`
make: *** [Makefile:189: .build/crystal] Error 1
builder for '/nix/store/d63pfq147brgjdydzp3y0i4x2mf6c89n-crystal-1.6.2.drv' failed with exit code 2
error: build of '/nix/store/d63pfq147brgjdydzp3y0i4x2mf6c89n-crystal-1.6.2.drv' failed

We can see that clang-13 was used instead of clang-11, yet the missing ___darwin_check_fd_set_overflow is still there.

If I downgrade llvmPackages to llvmPackages_10 as follows (to mimic how it was before this PR for x86_64-darwin),

% git diff
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 3b22223300c..22b4a8ae828 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -13889,7 +13889,7 @@ with pkgs;
   };
 
   inherit (callPackages ../development/compilers/crystal {
-    llvmPackages = llvmPackages_13;
+    llvmPackages = llvmPackages_10;
   })

I still get the ___darwin_check_fd_set_overflow error :-(

% nix-build -A crystal .
these derivations will be built:
  /nix/store/dafwli8r6akrf6n8hlprazmnlwl9k5a7-crystal-1.6.2.drv
building '/nix/store/dafwli8r6akrf6n8hlprazmnlwl9k5a7-crystal-1.6.2.drv'...
unpacking sources
unpacking source archive /nix/store/n5zijddfgjn4vrl0nwl7b7h9190wr5ff-source
source root is source
patching sources
applying patch /nix/store/jf31m11h0h5xpmgzpa2a4j0dm7kjsqyh-tzdata.patch
patching file src/crystal/system/unix/time.cr
configuring
no configure script, doing nothing
building
build flags: -j16 SHELL=/nix/store/2198gb5ws3cyma9cxrx3clq6p83781kc-bash-5.1-p16/bin/bash CRYSTAL_CONFIG_VERSION=1.6.2 progress=1 all docs release=1
Using /nix/store/081pffb2aa5ksmav3302cx25l160kyiy-llvm-10.0.1-dev/bin/llvm-config [version= 10.0.1]
clang++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/nix/store/081pffb2aa5ksmav3302cx25l160kyiy-llvm-10.0.1-dev/include -std=c++14 -stdlib=libc++  -fno-exceptions -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1672093091"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build --single-module -D strict_multi_assign --release --progress --threads 16 -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
ld: warning: directory not found for option '-L/nix/store/9va99kkpm2k5lryf10p1rsbi699q715r-crystal-1.6.2-lib/crystal'
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv.dylib) was built for newer macOS version (10.12) than being linked (10.11)
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libiconv-nocharset.dylib) was built for newer macOS version (10.12) than being linked (10.11)
ld: warning: dylib (/nix/store/z1d2x7ybr06vi5czzr27vsd5j77sc6qw-libiconv-50/lib/libcharset.1.dylib) was built for newer macOS version (10.12) than being linked (10.11)
Undefined symbols for architecture x86_64:
  "___darwin_check_fd_set_overflow", referenced from:
      _select_add in libevent.a(select.o)
      _select_del in libevent.a(select.o)
      _select_dispatch in libevent.a(select.o)
ld: symbol(s) not found for architecture x86_64
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `clang "${@}" -o /private/tmp/nix-build-crystal-1.6.2.drv-0/tmp.N6TcXL8fwc/private-tmp-nix-build-crystal-1.6.2.drv-0-source-src-ecr-process.cr/macro_run  -rdynamic -L/nix/store/9va99kkpm2k5lryf10p1rsbi699q715r-crystal-1.6.2-lib/crystal -L/nix/store/6yny71ym1icg45rvfwgn6ip5l4bwksq7-crystal-binary-1.2.2/embedded/lib -lgc -L/nix/store/mykvqcspihpk61a8pw0dfnkm615kmksl-libevent-2.1.12/lib -levent -liconv`
make: *** [Makefile:189: .build/crystal] Error 1
builder for '/nix/store/dafwli8r6akrf6n8hlprazmnlwl9k5a7-crystal-1.6.2.drv' failed with exit code 2
error: build of '/nix/store/dafwli8r6akrf6n8hlprazmnlwl9k5a7-crystal-1.6.2.drv' failed

I think the problem might be on libevent package actually. The lib is already built and references ___darwin_check_fd_set_overflow in x86-64_darwin .

Given that curl/curl#5210 is similar and it was resolved eventually with an xcode upgrade maybe the issue here is how libevent was built on x86-64_darwin initially.

I attempted to change something in the libevent formula to trigger a rebuilt locally

% git diff
diff --git a/pkgs/development/libraries/libevent/default.nix b/pkgs/development/libraries/libevent/default.nix
index bd5edec68a0..9f2cc1a2fea 100644
--- a/pkgs/development/libraries/libevent/default.nix
+++ b/pkgs/development/libraries/libevent/default.nix
@@ -20,8 +20,8 @@ stdenv.mkDerivation rec {
     })
   ];
 
-  preConfigure = lib.optionalString (lib.versionAtLeast stdenv.hostPlatform.darwinMinVersion "11") ''
-    MACOSX_DEPLOYMENT_TARGET=10.16
+  preConfigure = ''
+    MACOSX_DEPLOYMENT_TARGET=10.11
   '';

Unfortunately, the ___darwin_check_fd_set_overflow error is still present after that.

I am not sure how to maybe tweak which xcode version should be used.

@BenediktBroich BenediktBroich mentioned this pull request Dec 28, 2022
31 tasks
@auroraanna auroraanna mentioned this pull request Dec 28, 2022
13 tasks
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/unable-to-build-crystal-on-x86-64-darwin/24409/1

@GeopJr GeopJr changed the title crystal: 1.2 -> 1.6 crystal: 1.2 -> 1.7 Jan 9, 2023
@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jan 9, 2023
@GeopJr
Copy link
Member Author

GeopJr commented Jan 17, 2023

(At the time of writing this)

While I made the Crystal package use sdk 11 in an attempt to get rid of was built for newer macOS version (10.12) than being linked (10.11), it seems to have actually fixed the x86_64 darwin issue and is currently running the compiler specs for it!

Unfortunately the was built for newer macOS version (10.12) than being linked (10.11) did not get fixed.

@GeopJr
Copy link
Member Author

GeopJr commented Jan 17, 2023

It finished! So our only issue now is was built for newer macOS version (10.12) than being linked (10.11) (though, not critical)

Copy link
Contributor

@bcardiff bcardiff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! @GeopJr it's great you found about stdenv override. How did you manage?

I think is not critical and it would be great if we have a non-broken crystal in nixpkgs 😊

@GeopJr
Copy link
Member Author

GeopJr commented Jan 18, 2023

How did you manage?

Quite the rabbit hole - I was trying to find more info on was built for newer macOS version (10.12) than being linked (10.11) on this repo, searching 10.12 brought me to this issue: #101229 and after reading the comments and linked PRs I saw that zig and go used it so I gave it try (didn't expect it to fix the x86_64 darwin issue)!

I think is not critical and it would be great if we have a non-broken crystal in nixpkgs

Awesome! I'll run a review to see if anything is broken on 1.7.1 and then try to move this forward!

@miangraham
Copy link
Contributor

miangraham commented Jan 18, 2023

x86 darwin: Crystal itself looks like it's out of the woods! Nice!

Result of nixpkgs-review pr 195606 run on x86_64-darwin 1

5 packages failed to build:
  • ameba
  • crystal2nix
  • mint
  • oq
  • thicket
8 packages built:
  • crystal (crystal_1_7)
  • crystal_1_2
  • icr
  • invidious
  • kakoune-cr
  • lucky-cli
  • scry
  • shards (shards_0_17)

@GeopJr
Copy link
Member Author

GeopJr commented Jan 19, 2023

Thanks for the review @miangraham!

Unless someone on darwin wants to debug the failing packages, I'm ok with leaving them as is (if allowed), considering they haven't been built in darwin for ages and might need some workarounds from their maintainers.

This doesn't seem to be a Crystal version issue: ameba, oq, thicket, crystal2nix should work on 1.7 and mint is locked to 1.2.

@domenkozar domenkozar merged commit 15188f9 into NixOS:master Jan 20, 2023
@GeopJr GeopJr deleted the update/crystal branch April 15, 2023 23:17
@Janik-Haag Janik-Haag added the 12. first-time contribution This PR is the author's first one; please be gentle! label Jun 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.