WIP: splitting packages into multiple outputs #7701

vcunat · 2015-05-05T10:07:57Z

(descriptions will be updated in-place)

Current status

Most packages build fine: KDE4, some GNOME3 packages, Xfce. Hydra jobsets: http://hydra.nixos.org/jobset/nixos/closure-size http://hydra.nixos.org/jobset/nixpkgs/closure-size
I'm using NixOS off this (almost) without any problems since October, and there seem to be a few other users.
Preliminary introduction: http://lipa.ms.mff.cuni.cz/~cunav5am/nix/closure-size-notes.pdf (slightly outdated ATM).

Current merge blocker:

See WIP: splitting packages into multiple outputs #7701 (comment).

Minor stuff (not blocking merge to master)

linking problems in rare cases stdenv: principial linking problems when putting static and dynamic libraries into different directories #12085
a few icons stopped appearing.
Darwin seems untested, and many other platforms might have problems.
Handling of outputs installed by default. Merging PRs: nixos systemPackages: rework default outputs #12653, nix-env: respect meta.outputsToInstall nix#815, and perhaps further improvements.

vcunat · 2015-05-05T10:11:19Z

Opened a PR so people can browse the differences more easily, and we get a place to discuss the topic. Some hoped to have this ready for the upcoming release, but I don't seem to be able to stabilize it enough all by myself.

vcunat · 2015-05-05T10:17:52Z

I hope the key to making NixOS boot is in understanding these messages:

systemd[722]: Failed at step CHDIR spawning /nix/store/i1k1qs79ib9674mgdh9vw6xjbzxlhyb5-coreutils-8.23/bin/rm: No such file or directory

I don't understand that – the executable does exist on that path, and that path pkgs.coreutils is (still) in the default list of packages to be visible by systemd services. Moreover, only pkgs.coreutils.info output was added in this branch, as the rest are just the executables and locales (and man pages, but we typically want them and they are relatively small).

copumpkin · 2015-05-05T15:06:50Z

👍 really exciting to see this happening, thanks!

copumpkin · 2015-05-05T15:07:40Z

Is there a single VM test or simple command that someone can run to reproduce/iterate on the failures you're encountering?

vcunat · 2015-05-05T15:48:07Z

I think any attempt to boot NixOS would fail, even in a VM, but I haven't tried. Some of my filesystems won't mount IIRC (it's just plain btrfs).

copumpkin · 2015-05-05T18:23:21Z

This might be silly, but would it ease the transition if we took the existing mkDerivation and made it so that multiple-output derivations had a "synthetic" (lazily generated, obviously) outPath that acted like a buildEnv of the multiple outputs?

vcunat · 2015-05-05T18:28:36Z

Not much AFAIK. The point is getting rid of dependencies on the other files. If I knew what particular packages are causing the problems, I could e.g. just make them single-output again, for the time being. I've done a few such (partial) steps back for now, when there were too many references that would need changing and the space benefit was low.

lucabrunox · 2015-05-06T16:57:57Z

@vcunat perhaps about your mounting problems, you could try not splitting the mount.* related packages, or btrfs util related packages etc..

copumpkin · 2015-05-27T19:28:56Z

Is there any progress? Is this going to make it into 15.06?

vcunat · 2015-05-27T19:57:20Z

No, we would need several weeks at least to stabilize that in any case (it's a rather disturbing change), so I postponed it past the release (focusing more on that ATM).

copumpkin · 2015-05-27T20:02:31Z

Fair enough. Another thing I was wondering is whether there's a more incremental approach to merging this in, where we can slowly make the harmless packages multi-outputs in small bites without trying to do so much at once. In my experience, monolithic changes tend to languish because they're too overwhelming to fix/review/work on.

Basically, is there a sensible way to break this PR into smaller less-risky units that we could start merging in sooner?

vcunat · 2015-05-27T20:46:17Z

Well, maybe. I might somehow drop lots of the old commits and focus just e.g. on stdenv stuff first, or whatever packages have best savings/risk ratio. There were lots of packages split years ago, but the changes seem untested properly (and/or rotten w.r.t. master changes in the meantime).

domenkozar · 2015-06-01T09:03:19Z

I've cherry-picked a5fb184 into staging as it makes Python closure for ~100MB lighter

vcunat · 2015-07-30T10:23:29Z

TODO note:

vcunat: so .gir are only useful during build, right?
Lethalman: vcunat, yes
vcunat: (I wondered about .gir when working on multiple-output splitting)
Lethalman: .gir are needed to create .typelib
Lethalman: and because a .gir include another .gir, you need them in XDG_DATA_DIRS to be found
Lethalman: but at runtime, only the .typelib is needed

copumpkin · 2015-08-27T17:24:40Z

Perhaps we should cherry-pick more of this into master, updating the PR branch as we go along to show the diff getting smaller? It seems dauntingly large right now, which is a maintenance nightmare given an evolving nixpkgs, and also scares people from contributing. Are the commits mostly self-contained, or do they actually depend on one another? I'd be willing to pick a few low-hanging fruit in the name of helping this along.

vcunat · 2015-08-27T17:45:15Z

They often aren't self-contained by themselves, and none of the "commit states" in the branch yields a fully functioning system, I believe.

I expect to go back to it after the release and start by porting the shared infrastructure to staging (mostly autotools-focused ATM), which by itself should be relatively easy (for me) and breaking very few packages (with fixes somewhere in the branch already). And perhaps write some description/documentation/tutorial.

After that, anyone will be able to use it to split packages they understand. When I wanted to stabilize this, I was plagued by the scope of breakage in packages that were split and broken by it (e.g. systemd+udev proved to be painful).

dezgeg · 2015-08-27T19:24:55Z

So I took a look at this last week (branch at https://github.com/dezgeg/nixpkgs/commits/multiple-outputs-v2) and while the merge conflicts were indeed pretty bad, this isn't a totally lost cause (it does boot in Virtualbox for me, and at least closures.{smallContainer,xfce,ec2} can be built). Off the top of my head, these are the breakages I noticed:

udev can't find it's rules provided by the package (see commit msg of dezgeg@04bd57b for details). This probably explains the problems mounting filesystems.
some KDE/Qt packages can't locate PCRE
gcj fails to build due to cycles between outputs
libsodium doesn't compile on ARM (linking fails due to bad relocation)
cross compilation is totally broken; the cross GCC fails to build

Close #9935. I don't think it's good to split thunarx into a separate derivation in this way. We would have the library and associated stuff twice and two expressions, etc. Distributions do these things by an analogy to multiple-output derivation (Debian does, for example), so we could use that, but it doesn't seem important in this case and would be better after #7701 anyway.

Close NixOS#9935. I don't think it's good to split thunarx into a separate derivation in this way. We would have the library and associated stuff twice and two expressions, etc. Distributions do these things by an analogy to multiple-output derivation (Debian does, for example), so we could use that, but it doesn't seem important in this case and would be better after NixOS#7701 anyway.

vcunat · 2015-10-02T13:25:10Z

@dezgeg: thank you! I'm wading through the branch again, and I do find your commits useful. I'll be integrating them.

You haven't resolved the udev/systemd issues, right? I'm thinking of unsplitting it and perhaps using a separately built libudev in dependencies or something.

dezgeg · 2015-10-02T13:40:30Z

Nope, I didn't even try fixing udev/systemd.

vcunat · 2015-10-04T17:24:29Z

So, now I'm running my NixOS on this branch!

The boot problems were probably fixed by a changed approach to systemd/udev b44d846.
As noted above, some KDE packages were failing to locate pcre or automoc, but I just added those.
libsodium had problems even on x86_64-linux, but not anymore c83b591.
Many other small build problems got fixed.

The switch to gcc-5 (and others) are queued for evaluation on staging now, so I don't expect staging this PR before we stabilize that a bit. Also, I merged latest systemd branch into this one (due to possible interactions with that udev rework).

…tation ccacheStdenv: provide working example config in docs

khard: 0.8.1 -> 0.9.0

libinput: 1.2.1 -> 1.2.2, propagate udev dependency

wayland: 1.9.0 -> 1.10.0, weston: 1.9.0 -> 1.10.0, wayland-protocols: init at 1.3

minidlna: logging via journalctl, use systemd runtimedirectory, install manpages

Is not used according to source code

vcunat · 2016-04-10T07:32:48Z

I think this branch is stable enough to go to master, so I'll resolve conflicts and merge it to staging now. That's because there's a security mass rebuild in staging that hasn't started rebuilding yet, and I don't want to build both mass rebuilds separately. People wanting to avoid breakages can switch to 16.03...

Comparison to master evaluations on Hydra: - 1255515 for nixos - 1255502 for nixpkgs

deepfire · 2016-04-10T09:42:39Z

: -)

domenkozar · 2016-04-10T10:17:45Z

I cancelled the closure-size jobset on hydra. Thanks!

On Sun, 10 Apr 2016, 10:42 Kosyrev Serge, [email protected] wrote:

: -)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#7701 (comment)

lucabrunox · 2016-04-10T10:21:11Z

It's happening........... thanks a lot for the hard work.

abbradar · 2016-04-10T10:38:38Z

I don't want to spam this thread with congratulations but I can't keep myself from doing this ~_^. Thank you very much -- and now we can share the burden of fixing what broke and also enjoy the benefits of this wonderful work!

jagajaga · 2016-04-10T10:53:33Z

Great job!!

abbradar · 2016-04-10T11:04:59Z

On Hydra closure-size branch is not yet cancelled (or at least looks so) -- should it be like this?

vcunat · 2016-04-10T17:23:30Z

The closure-size branch contains the same tree as staging, so it won't really matter. I had cancelled all scheduled builds when I merged (on staging and closure-size).

... needed after closure-size merge (#7701)

jtojnar · 2018-03-29T11:00:17Z

How does libxml2Python work?

{
libxml2Python = pkgs.buildEnv { # slightly hacky
    name = "libxml2+py-${self.libxml2.version}";
    paths = with libxml2; [ dev bin py ];
    inherit (libxml2) passthru;
    # the hook to find catalogs is hidden by buildEnv
    postBuild = ''
      mkdir "$out/nix-support"
      cp '${libxml2.dev}/nix-support/propagated-build-inputs' "$out/nix-support/"
    '';
  };
}

Is not it enough to add libxml2.py to propagatedBuildInputs of python package?

Why does not the postBuild fail? dev output is not listed in propagatedBuildOutputs.

vcunat · 2018-03-30T21:12:24Z

@jtojnar: I think that buildEnv hack was needed at that time, but I could have been wrong. I was really swamped by all the changes (splitting something like a hundred packages). It will certainly be better if it works without the hack.

dev is the default output for build inputs. And explicitly referenced paths are put into build-time closure anyway...

vcunat added 0.kind: enhancement Add something new 9.needs: reporter feedback This issue needs the person who filed it to respond 1.severity: mass-rebuild This PR causes a large number of packages to rebuild 2.status: work-in-progress This PR isn't done 9.needs: changelog labels May 5, 2015

vcunat mentioned this pull request Jun 25, 2015

Make mariadb conform to multiple-outputs #8494

Closed

vcunat mentioned this pull request Sep 3, 2015

imagemagick: add pkgconfig as a runtime dependency #9604

Closed

vcunat mentioned this pull request Oct 3, 2015

systemd: update to 227 #6671

Closed

joachifm and others added 11 commits April 9, 2016 19:38

Merge pull request #14382 from aneeshusa/update-ccache-stdenv-documen…

1d36e6e

…tation ccacheStdenv: provide working example config in docs

duc: init at 1.3.3

73bf336

khard: 0.8.1 -> 0.9.0

2601b32

duc: fix meta.license

aec8dae

manual multiple-output: clarify a note

60fa61c

Merge pull request #14560 from DamienCassou/khard-0.9.0

44af20f

khard: 0.8.1 -> 0.9.0

Merge pull request #14544 from groxxda/bump/libinput

870ce8f

libinput: 1.2.1 -> 1.2.2, propagate udev dependency

Merge pull request #14547 from groxxda/bump/wayland

2579128

wayland: 1.9.0 -> 1.10.0, weston: 1.9.0 -> 1.10.0, wayland-protocols: init at 1.3

Merge pull request #14543 from groxxda/fix/minidlna

b667040

minidlna: logging via journalctl, use systemd runtimedirectory, install manpages

virt-manager: remove sqlalchemy from (transitive) dependencies

fe97489

Is not used according to source code

virtinst: Fix cherry-pick merge

2463e09

vcunat added 2 commits April 10, 2016 09:33

Merge branch 'staging' into closure-size

50fccad

Merge branch 'master' into closure-size

30f1424

Comparison to master evaluations on Hydra: - 1255515 for nixos - 1255502 for nixpkgs

vcunat merged commit 30f1424 into staging Apr 10, 2016

vcunat added a commit that referenced this pull request Apr 10, 2016

Merge branch 'closure-size' (#7701) into staging

4fa9983

vcunat added a commit that referenced this pull request Apr 13, 2016

Merge branch 'staging', containing closure-size #7701

39ebb01

vcunat deleted the closure-size branch May 12, 2016 01:45

vcunat added a commit that referenced this pull request Jun 10, 2016

Merge #15867: glibc, gcc: fixes for ARM targets

46f22d8

... needed after closure-size merge (#7701)

FRidh mentioned this pull request Jan 11, 2020

libxml2: don't propagate Python bindings #77474

Merged

10 tasks

wolfgangwalther mentioned this pull request May 26, 2024

gettext, texinfo: Remove spurious xz #314670

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: splitting packages into multiple outputs #7701

WIP: splitting packages into multiple outputs #7701

vcunat commented May 5, 2015

vcunat commented May 5, 2015

vcunat commented May 5, 2015

copumpkin commented May 5, 2015

copumpkin commented May 5, 2015

vcunat commented May 5, 2015

copumpkin commented May 5, 2015

vcunat commented May 5, 2015

lucabrunox commented May 6, 2015

copumpkin commented May 27, 2015

vcunat commented May 27, 2015

copumpkin commented May 27, 2015

vcunat commented May 27, 2015

domenkozar commented Jun 1, 2015

vcunat commented Jul 30, 2015

copumpkin commented Aug 27, 2015

vcunat commented Aug 27, 2015

dezgeg commented Aug 27, 2015

vcunat commented Oct 2, 2015

dezgeg commented Oct 2, 2015

vcunat commented Oct 4, 2015

vcunat commented Apr 10, 2016

deepfire commented Apr 10, 2016

domenkozar commented Apr 10, 2016

lucabrunox commented Apr 10, 2016

abbradar commented Apr 10, 2016

jagajaga commented Apr 10, 2016

abbradar commented Apr 10, 2016

vcunat commented Apr 10, 2016

jtojnar commented Mar 29, 2018

vcunat commented Mar 30, 2018

WIP: splitting packages into multiple outputs #7701

WIP: splitting packages into multiple outputs #7701

Conversation

vcunat commented May 5, 2015

Current status

Current merge blocker:

Minor stuff (not blocking merge to master)

vcunat commented May 5, 2015

vcunat commented May 5, 2015

copumpkin commented May 5, 2015

copumpkin commented May 5, 2015

vcunat commented May 5, 2015

copumpkin commented May 5, 2015

vcunat commented May 5, 2015

lucabrunox commented May 6, 2015

copumpkin commented May 27, 2015

vcunat commented May 27, 2015

copumpkin commented May 27, 2015

vcunat commented May 27, 2015

domenkozar commented Jun 1, 2015

vcunat commented Jul 30, 2015

copumpkin commented Aug 27, 2015

vcunat commented Aug 27, 2015

dezgeg commented Aug 27, 2015

vcunat commented Oct 2, 2015

dezgeg commented Oct 2, 2015

vcunat commented Oct 4, 2015

vcunat commented Apr 10, 2016

deepfire commented Apr 10, 2016

domenkozar commented Apr 10, 2016

lucabrunox commented Apr 10, 2016

abbradar commented Apr 10, 2016

jagajaga commented Apr 10, 2016

abbradar commented Apr 10, 2016

vcunat commented Apr 10, 2016

jtojnar commented Mar 29, 2018

vcunat commented Mar 30, 2018