Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Haskell binaries with excessive closure sizes on aarch64-darwin #318013

Open
1 of 11 tasks
sternenseemann opened this issue Jun 7, 2024 · 1 comment
Open
1 of 11 tasks

Haskell binaries with excessive closure sizes on aarch64-darwin #318013

sternenseemann opened this issue Jun 7, 2024 · 1 comment
Labels
0.kind: enhancement Add something new 3.skill: sprintable A larger issue which is split into distinct actionable tasks 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems 6.topic: darwin Running or building packages on Darwin 6.topic: haskell

Comments

@sternenseemann
Copy link
Member

sternenseemann commented Jun 7, 2024 •
edited
Loading

As you may remember from #140774, GHC on aarch64-darwin can't properly eliminate dead code, leading to a lot of references if Paths_ modules are involved that may not be necessary.

A common case is that binaries that depend on other libraries that use a Paths_ module will incur a reference on that library's out path despite being statically linked against the library (even though the reference itself is dead code). On other platforms (even x86_64-darwin) GHC is able to eliminate such references, but not on aarch64-darwin. For such packages, justStaticExecutables is practically useless since GHC is pulled in transitively.

@9999years has contributed an extra check to justStaticExecutables which prevents this scenario (#304352). As a consequence, a few packages are broken on aarch64-darwin. In the interest of landing the bundled changes from haskell-updates soon, I'll disable justStaticExecutables for the following packages on aarch64-darwin since it wasn't doing much anyways:

This will take effect when #315167 is merged.

Maintainers are encouraged to submit a workaround for the issue on aarch64-darwin via remove-references-to (which seems to be the only solution at the moment) and to reenable justStaticExecutables. Examples for such fixes are:

Please make sure these fixes are conditional to aarch64-darwin.
@sternenseemann sternenseemann added 0.kind: enhancement Add something new 6.topic: darwin Running or building packages on Darwin 6.topic: haskell 3.skill: sprintable A larger issue which is split into distinct actionable tasks 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems labels Jun 7, 2024
@sternenseemann sternenseemann mentioned this issue Jun 7, 2024
Merged
6 tasks
sternenseemann added a commit that referenced this issue Jun 8, 2024
@sternenseemann
treewide: no justStaticExecutables on aarch64-darwin for bogus refs
d785f11 
This commit disables justStaticExecutables for packages on
aarch64-darwin that incur a (usually incorrect) reference on GHC.
justStaticExecutables now fails when a GHC reference remains in the
output (#304352). Due to this reference justStaticExecutables (before
these changes) would only marginally reduce closure size.

In the future, we'll be able to re-introduce justStaticExecutables
after (manually) removing the incorrect references stemming from the use
of Paths_* modules. Tracking Issue: #318013
@roberth
Copy link
Member

roberth commented Jun 9, 2024 •
edited
Loading

EDIT: If you're just trying to get it done, these instructions are easier and more helpful: #304352 (comment)

I've opened NixOS/nix#10877 to make this easier to analyze.
Worth a try it, especially if you have no idea what to even look for, but if you're running a nix-daemon you'll have to run e.g. sudo nix run github:hercules-ci/nix/report-forbidden-path-as-chain build <your actual thing>, with sudo so that the freshly built Nix will build on its own instead of letting the old daemon do it.

This was referenced Jun 9, 2024
Open
Open
wolfgangwalther added a commit to wolfgangwalther/postgrest that referenced this issue Jun 17, 2024
@wolfgangwalther
refactor: Remove dependency on Paths_ module
57f2bf7 
The cabal-provided Paths_ module allows us to use the version number
from postgrest.cabal. This can be done equally well with the GHC-defined
CPP macro "VERSION_postgrest".

By making this change we avoid the inclusion of the Paths_ module, which
also stores some paths related to the cabal configuration. Those are
problematic to go into the final executable, because for nix-based
builds those are paths to the /nix/store/... - which means that our
static executable then depends on those paths.. and we can't build a
minimal docker image anymore.

To counter this, we have been using dead code elimination when building
the static executable. This has been working well, but there is a
problem on aarch64-darwin, which we will hit once can finally make our
way there: GHC on aarch64-darwin (or darwin in general?) can't do dead
code elimination - and thus it'd be impossible to create those minimal
docker images for those platforms. More information upstream in nixpkgs:
NixOS/nixpkgs#318013
wolfgangwalther added a commit to PostgREST/postgrest that referenced this issue Jun 18, 2024
@wolfgangwalther
refactor: Remove dependency on Paths_ module
c045b26 
The cabal-provided Paths_ module allows us to use the version number
from postgrest.cabal. This can be done equally well with the GHC-defined
CPP macro "VERSION_postgrest".

By making this change we avoid the inclusion of the Paths_ module, which
also stores some paths related to the cabal configuration. Those are
problematic to go into the final executable, because for nix-based
builds those are paths to the /nix/store/... - which means that our
static executable then depends on those paths.. and we can't build a
minimal docker image anymore.

To counter this, we have been using dead code elimination when building
the static executable. This has been working well, but there is a
problem on aarch64-darwin, which we will hit once can finally make our
way there: GHC on aarch64-darwin (or darwin in general?) can't do dead
code elimination - and thus it'd be impossible to create those minimal
docker images for those platforms. More information upstream in nixpkgs:
NixOS/nixpkgs#318013
@maralorn maralorn mentioned this issue Jul 14, 2024
Closed
sternenseemann added a commit that referenced this issue Aug 30, 2024
@sternenseemann
oama: ignore erroneous references on aarch64-darwin
6cc8c10 
The use of the Paths_ module causes erroneous references on
aarch64-darwin (due to a lack of dead code elimination), so we can't use
justStaticExecutables.

See #318013.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0.kind: enhancement Add something new 3.skill: sprintable A larger issue which is split into distinct actionable tasks 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems 6.topic: darwin Running or building packages on Darwin 6.topic: haskell
Projects
Haskell infrastructure in nixpkgs enhancements
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
2 participants
@roberth @sternenseemann