[RFC 0092] Computed derivations #92

Ericson2314 · 2021-04-26T05:23:47Z

This is the spiritual successor of #40.

Rendered merged

Tracking issue

Also: - Allow fixed output builds (in that temp store) - Clean up drawbacks and alternatives.

Thanks @siddharthist! Co-Authored-By: Ericson2314 <[email protected]>

@Mic92

Thanks @Mic92 Co-Authored-By: Ericson2314 <[email protected]>

@globin

Thanks @globin Co-Authored-By: Ericson2314 <[email protected]>

Thanks @siddharthist! Co-Authored-By: Ericson2314 <[email protected]>

Break off some previously inline observations into their own subsection.

@Mic92

Thanks @Mic92! Co-Authored-By: Ericson2314 <[email protected]>

@ocharles

The later examples with `nix-instantiate` automatically get all the outputs of the final rewritten drv, so there's no output iteration needed. `"$o"` was mistakenly copied over from the earlier examples. Thanks to @ocharles for asking me the question that led me to this. Hopefully this change answers that?

Co-Authored-By: Domen Kožar <[email protected]>

about -> out

No `let`, so don't need `in`.

This was in the "wrapper" derivation example.

@globin

We already cache builds just fine, thanks! Thanks @globin for catching

asymmetric

Left a couple of comments to make this more accessible.

rfcs/0000-plan-dynanism.md

asymmetric · 2021-04-26T12:30:50Z

rfcs/0000-plan-dynanism.md

+   e ⇓ e'
+   e' is derivation
+   ------
+   builtins.assumeDerivation e ⇓ e'
+   ```


Could you add a section explaining this syntax? I assume it's familiar for PLT folks, but it's very cryptic to me :)

Those are inference rules expressed in operational semantics.

You can read that as if e evaluates to e' and e' is a derivation, then builtins.assumeDerivation e evaluates to e'.

I agree with asymmetric, we're not a PLT-focused symposium, prose is fine here.

I think a more accessible presentation would be better. This is pseudo-formal, because (AFAIK) the ⇓ and is derivation judgements aren't actually defined anywhere.

Agreed. Unfortunately not everyone has that background.

A "translation" would suffice. Those compaxt bits of formalisms are great, once one has learned to parse them.

Well.....I'll put it this way:

I think CA derivations' various notions of store coherence (by far the subtlest part), GUIX-style "grafting", mixing native- and cross-compiled packages, and probably other things I can't anticipate are going to repeatedly keep us thinking about these sorts of inductive relations. Some sort of sequent-ish syntax, or prolog's horn clause syntax, or many other such things are far and away the clearest and most concise way to convey this information once the reader is familiar --- it's supposed to be nice and "pictoral" not inscrutable ivory tower. I think it be best if we start moving in that direction.

Even if I right some prose now, It's going to a pain to update. VS with this sort of syntax it can be easier to just stare at and have all the important bits come out, so it's easier to revise.

So I am fine writing some sort of prose explanation, but I rather do it once the RFC is further along.

P.S. whether you love our hate this stuff, https://www.youtube.com/watch?v=dCuZkaaou0Q is a fun talk.

You could include a link to a short introduction to natural deduction.

Since this is gone we can remove this.

rfcs/0000-plan-dynanism.md

sternenseemann · 2021-04-26T12:32:21Z

rfcs/0000-plan-dynanism.md

+                       |  <single-installable> ! *
+   ```
+
+   Now that we have `path` vs `path!*`, we also don't need `---derivation` as a disambiguator, and so that should be removed along with all the complexity that goes with it.


This should probably be clarified. I took me three readings to figure out that /nix/store/…foo.drv!* is to build all outputs (which is the current default behavior unless --derivation is specified, I assume?) and /nix/store/…foo.drv would just build the derivation.

Oh sheesh, yes I never said what the semantics were. Did that in addition to the examples @asymmetric asked for.

Does it make sense now? (before I mark this resolved.)

The example clarifies things yeah! I am unsure though if the mention of --derivation is detremental at this point since it is a nixUnstable only thing and I could not find any documentation of the flag that even remotely explained to me what it does.

I guess we have a bit of a problem that this RFC somewhat overlaps with problems related to the Installables abstraction and the unstable CLI?

rfcs/0000-plan-dynanism.md

edolstra · 2022-01-12T14:41:58Z

Yes indeed, so I've merged the RFC. Thanks all!

nixos-discourse · 2022-04-26T16:50:52Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/73

nixos-discourse · 2022-05-03T21:20:33Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/recent-nix-change-worthy-of-greater-attention/18972/1

Ericson2314 · 2022-10-23T03:34:07Z

L-as · 2022-10-23T10:21:04Z

We should post the pseudoshepherd meeting notes somewhere

das-g · 2022-10-23T10:34:49Z

We should post the pseudoshepherd meeting notes somewhere

I know our RFC process has the concept of shepherd teams, but what's a "pseudoshepherd meeting"?

L-as · 2022-10-23T12:42:07Z

@das-g We talked about this RFC at NixCon.

tomberek · 2022-10-24T18:55:00Z

Notes from NixCon opportunistic meeting in the courtyard:

Summary

derivations that produce derivations
IFD is normally used for bulk import of language-specifics
ninja/make export for fine-grained caching
performance characteristics
equivalent ergonomics possible to full recursive-nix

History

2019 initial implementation
Niel Mitchell, Applicative build systems are Monadic under the hood link
on the fly lazy handling of the build graph instead of static build plan
changes to the heart of Nix
need more code review
keep the momentum instead of exacerbating merge conflicts
get John off of "single point of failure"
add builtPath case of derivedPath
treat the output of a derivation as a derivation

Summary of internals

nix evaluation creates a derivation and it builds into another derivation.
Uses the top-level goal to guide forward progress.
Self-modifying graph, substitute new graphs in to the old.
nix expression, with import nixpkgs hello, inside derivation, evaluate, build
useful to avoid IFD
parallelize evaluation
Crates, extract names to do auto-completion
repl supports IFD while live
"^" for user feedback, interally the daemon protocol uses "!"
RFC92 does not impact repl
dervations produced do not have Nix values associated with them, store-layer concept
override of dependency. with aeson we know static dependencies, with dynamic derivations
might be possible to override, but might fail at build time

next step brainstorm

create use cases (required for stabilization, not for experimental)
make lang2nix examples
needs library to make it ergonomic, or an extra primitive
primary advantage is it is a limited IFD
ninja2nix
GNUMake 2 nix
nixpkgs is limited because it cannot import nix code from elsewhere
does this solve incremental builds
- build Linux, it calculates the build graph
- the evaluation inside the derivation can be cached
- purs-nix
IFD can be made correct, very hard to make it fast
92 is easy to make performant
distcc, ccache
"optimize IFD to do eval in parallel"
Hercules-CI agent does some eval
caching Nix evaluation, builtins.cache
nix build .#x can make the derivation for you...
this might also implement cached Nix evaluation (see Nix PR for caching primop)
well suited for big builds
recursive Nix is easier to adapt, speak the daemon protocol inside the sandbox
The test case is the existing yarn2nix, old version of snack,
cross system is an idiom question, how to express this?
what conventions, when to use, when not to
clearly describe when
makePackage function, eval-time attributes added

there are simple advantages, but unclear if we can get the benefits for large projects

makes the sizes of necessary locks much smaller, making previously unpalatable appoaches better

Steps

review and merge PR
Nixpkgs? or branch with Hydra to test this
staging hydra
Project/example to prove it helps and measure

dependencies

determine if we need dynamic number of outputs? or recursive nix?
review existing PR to debug cycles of multi-output derivation (not a blocker, but helpful)
determine what we can do without the recursive-nix constraint

recursive-nix followup

"can we limit recursive-nix to only build from dynamic-derivation and use only nix-instantiate?"

question: dynamic derivation

only limit to nix-instantiate
"userspace" nix-instantiate
"kernel" add store paths

roberth · 2023-03-30T16:01:40Z

Computed derivations could provide a means to access old packages in case of a major change to the language.

I'm not suggesting we should change the language in backwards incompatible ways, but if we were to do it, this feature would let us preserve Nix's unique selling point of being able to use very old software alongside very up to date software.

It will not generalize to any expression though; mostly just to derivations, so things like NixOS modules would still have to be ported, but this is much less effort than porting the entire toolchain and dependencies that produced the legacy package. (And you should probably review an old module anyway, if you even dare to run it as part of your OS)

Interestingly, this could be one way in which we'd have a need for nested computed derivations: if the legacy package before the language change was itself a computed derivation, because it uses a derivation-supported build system. Though for now it's a bit hard to imagine such a fancy package to be old.

More likely we'll first see a derivation-supported build system to rely on a computed derivation (to build the build system), though that seems to be a different kind of nesting, as such a build system is parented by Nixpkgs and could be injected into the final dynamic derivation by means of a store path, rather than having to do a double outputOf. We might not even call that nesting.

nixos-discourse · 2023-09-09T23:28:05Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixcon-governance-workshop/32705/9

nixos-discourse · 2023-12-20T12:48:37Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/community-calendar/18589/102

@Mic92

* Initial draft "ret-cont" recursive Nix * Fix typos and finish trailing sentance * Switch to advocating temp store rather than daemon socket Also: - Allow fixed output builds (in that temp store) - Clean up drawbacks and alternatives. * ret-cont-recursive-nix: Fix typo Thanks @siddharthist! Co-Authored-By: Ericson2314 <[email protected]> * ret-cont-recursive-nix: Fix typo Thanks @Mic92 Co-Authored-By: Ericson2314 <[email protected]> * ret-cont-recursive-nix: Fix typo Thanks @globin Co-Authored-By: Ericson2314 <[email protected]> * ret-cont-recursive-nix: Fix typo Thanks @siddharthist! Co-Authored-By: Ericson2314 <[email protected]> * ret-cont-recursive-nix: Clean up motivation, adding examples * ret-cont-recursive-nix: Improve syntax highlighting * Do a lousy job formalizing the detailed design Break off some previously inline observations into their own subsection. * ret-cont-recursive-nix: Mention `builtins.exec` in alternatives * ret-cont-recursive-nix: Fix typo Thanks @Mic92! Co-Authored-By: Ericson2314 <[email protected]> * ret-cont-recursive-nix: Remove dangling "$o" The later examples with `nix-instantiate` automatically get all the outputs of the final rewritten drv, so there's no output iteration needed. `"$o"` was mistakenly copied over from the earlier examples. Thanks to @ocharles for asking me the question that led me to this. Hopefully this change answers that? * Update rfcs/0000-ret-cont-recursive-nix.md Co-Authored-By: Domen Kožar <[email protected]> * ret-cont-recursive: Fix typo about -> out * ret-cont: Add examples and expand future work * ret-cont: Fix syntax error No `let`, so don't need `in`. * ret-cont: Mention Ninja's upcomming `dyndep` and C++ oppertunity * ret-cont: Fix missing explicit `outputs` and `__recursive` This was in the "wrapper" derivation example. * ret-cont: "caching builds" -> "caching evaluation" We already cache builds just fine, thanks! Thanks @globin for catching * ret-cont: Improve formalism and reference NixOS#62 * drv-build-drv: Start drafting from old ret-cont-recursive-nix RFC * drv-buiild-drv: WIP rewrite * plan-dynamism: Rewrite RFC yet again * plan-dynamism: Rename file accordingly * plan-dynanism: Fix typo Thanks @mweinelt. * plan-dynanism: Fix formalism slightly * Apply suggestions from code review Thanks! Co-authored-by: Rosario Pulella <[email protected]> Co-authored-by: Ninjatrappeur <[email protected]> * plan-dynamism: `Buildables` -> `DerivedPathsWithHints` Thanks @sternenseemann for catching. * plan-dynamism: Add semantics and examples for `!` syntax * plan-dynamism: Too many dashes in `--derivation` * plan-dynanism: Put pupose of text hashing before name * Apply suggestions from code review Co-authored-by: Ollie Charles <[email protected]> * Apply suggestions from code review Co-authored-by: Ollie Charles <[email protected]> * Apply suggestions from code review * Update rfcs/0000-plan-dynanism.md * plan-dynanism: Fix bad sentence Thanks @roberth! * plan-dynamism: Number the two parts * plan-dynamism: Rip out part 2 There is more to do I'm sure, but I wanted to get the ball rolling. * plan-dynamism: New motivation * plan-dynamism: Fix typo Thanks @L-as * TEMP PLES AMEND * [RFC 0092] Rename file * [RFC 0092] Fix YAML header * [RFC 0092] Rewrite summary * [RFC 0092] Add link to documentation * [RFC 0092] Rewrite example section * [RFC 0092] Small fix * [RFC 0092] Rewrite drawbacks and alternatives * [RFC 0092] Improve alternatives section * [RFC 0092] Fix syntax error * [RFC 0092] Small change * [RFC 0092] Remove unnecessary file * [RFC 0092] Add comment about IFD * [RFC 0092] Fix typo * Update rfcs/0092-plan-dynamism.md Co-authored-by: Jörg Thalheim <[email protected]> * plan-dynamism-experiment: Make clear is experimental * plan-dynamism-experiment: Fix typo Thanks @L-as Co-authored-by: Langston Barrett <[email protected]> Co-authored-by: Jörg Thalheim <[email protected]> Co-authored-by: Robin Gloster <[email protected]> Co-authored-by: Domen Kožar <[email protected]> Co-authored-by: Rosario Pulella <[email protected]> Co-authored-by: Ninjatrappeur <[email protected]> Co-authored-by: Ollie Charles <[email protected]> Co-authored-by: Las Safin <[email protected]> Co-authored-by: Eelco Dolstra <[email protected]>

nixos-discourse · 2024-10-01T15:03:22Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/state-of-haskell-nix-ecosystem-2024/53740/8

Ericson2314 and others added 26 commits February 1, 2019 01:08

Initial draft "ret-cont" recursive Nix

3a8338a

Fix typos and finish trailing sentance

3b8422a

Switch to advocating temp store rather than daemon socket

4da9193

Also: - Allow fixed output builds (in that temp store) - Clean up drawbacks and alternatives.

ret-cont-recursive-nix: Fix typo

800b5f3

Thanks @siddharthist! Co-Authored-By: Ericson2314 <[email protected]>

ret-cont-recursive-nix: Fix typo

f708983

Thanks @Mic92 Co-Authored-By: Ericson2314 <[email protected]>

ret-cont-recursive-nix: Fix typo

6a87c1b

Thanks @globin Co-Authored-By: Ericson2314 <[email protected]>

ret-cont-recursive-nix: Fix typo

36193e5

Thanks @siddharthist! Co-Authored-By: Ericson2314 <[email protected]>

ret-cont-recursive-nix: Clean up motivation, adding examples

ffb9203

ret-cont-recursive-nix: Improve syntax highlighting

5564fdb

Do a lousy job formalizing the detailed design

22f8322

Break off some previously inline observations into their own subsection.

ret-cont-recursive-nix: Mention builtins.exec in alternatives

7f5f854

ret-cont-recursive-nix: Fix typo

5c9f1fb

Thanks @Mic92! Co-Authored-By: Ericson2314 <[email protected]>

Update rfcs/0000-ret-cont-recursive-nix.md

ba7dcce

Co-Authored-By: Domen Kožar <[email protected]>

ret-cont-recursive: Fix typo

8bcb4e6

about -> out

ret-cont: Add examples and expand future work

baae1e6

ret-cont: Fix syntax error

9448a2a

No `let`, so don't need `in`.

ret-cont: Mention Ninja's upcomming dyndep and C++ oppertunity

37a643e

ret-cont: Fix missing explicit outputs and __recursive

14b134d

This was in the "wrapper" derivation example.

ret-cont: "caching builds" -> "caching evaluation"

1b0a6a1

We already cache builds just fine, thanks! Thanks @globin for catching

ret-cont: Improve formalism and reference NixOS#62

3fe1c3d

drv-build-drv: Start drafting from old ret-cont-recursive-nix RFC

a38f245

Merge remote-tracking branch 'upstream/master' into ret-cont

4022058

drv-buiild-drv: WIP rewrite

6c01e4d

plan-dynamism: Rewrite RFC yet again

6d97de1

plan-dynamism: Rename file accordingly

2079528

asymmetric reviewed Apr 26, 2021

View reviewed changes

sternenseemann reviewed Apr 26, 2021

View reviewed changes

mweinelt reviewed Apr 26, 2021

View reviewed changes

rfcs/0000-plan-dynanism.md Outdated Show resolved Hide resolved

Ericson2314 marked this pull request as draft April 26, 2021 14:23

edolstra added status: accepted and removed status: FCP in Final Comment Period labels Jan 12, 2022

Ericson2314 deleted the plan-dynamism branch January 13, 2022 05:00

This was referenced Feb 9, 2022

stdenv.mkDerivation: overlay style overridable recursive attributes NixOS/nixpkgs#119942

Merged

lib: Add encapsulate, attrsets that have overlay-based private attrs in their closure NixOS/nixpkgs#158781

Draft

This was referenced Feb 11, 2022

[RFC] Build Drivers for cabal haskell/cabal#7906

Open

C++20 modules are in: discussing a sane (experimental) design for Meson mesonbuild/meson#5024

Open

roberth mentioned this pull request Apr 24, 2022

[RFC 0119] Formalize testing for nixpkgs packages #119

Merged

roberth mentioned this pull request Jul 18, 2022

Legacy stdenv.mkDerivation NixOS/nixpkgs#295044

Open

roberth mentioned this pull request Dec 9, 2022

Split out a package interface for non-single-drv packages DavHau/drv-parts#4

Closed

Ericson2314 mentioned this pull request Jan 6, 2023

Tracking issue for RFC 92: Dynamic derivations NixOS/nix#6316

Open

4 tasks

roberth mentioned this pull request Feb 3, 2023

Experimental python bindings NixOS/nix#7735

Draft

14 tasks

Ericson2314 mentioned this pull request Feb 13, 2023

[RFC 0134] Carve out a store-only Nix #134

Merged

roberth mentioned this pull request Mar 6, 2023

name should be package name, not store path name NixOS/nixpkgs#295034

Open

ghost mentioned this pull request Jan 18, 2024

Opt-in for running installCheckPhase on cross builds NixOS/nixpkgs#273110

Closed

Mic92 mentioned this pull request Jul 15, 2024

Cargo.lock considered harmful NixOS/nixpkgs#327063

Closed

roberth mentioned this pull request Nov 14, 2024

Dynamic derivations-based lock-based fetching NixOS/nixpkgs#355909

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC 0092] Computed derivations #92

[RFC 0092] Computed derivations #92

Ericson2314 commented Apr 26, 2021 •

edited

Loading

asymmetric left a comment

asymmetric Apr 26, 2021

picnoir Apr 26, 2021

ocharles Apr 30, 2021

andir May 31, 2021

blaggacao Jun 2, 2021

Ericson2314 Jun 5, 2021

L-as Aug 26, 2021

L-as Oct 20, 2021

sternenseemann Apr 26, 2021

Ericson2314 Apr 27, 2021 •

edited

Loading

Ericson2314 Jun 5, 2021

sternenseemann Jun 6, 2021

edolstra commented Jan 12, 2022

nixos-discourse commented Apr 26, 2022

nixos-discourse commented May 3, 2022

Ericson2314 commented Oct 23, 2022

L-as commented Oct 23, 2022

das-g commented Oct 23, 2022 •

edited

Loading

L-as commented Oct 23, 2022

tomberek commented Oct 24, 2022

roberth commented Mar 30, 2023

nixos-discourse commented Sep 9, 2023

nixos-discourse commented Dec 20, 2023

nixos-discourse commented Oct 1, 2024

[RFC 0092] Computed derivations #92

[RFC 0092] Computed derivations #92

Conversation

Ericson2314 commented Apr 26, 2021 • edited Loading

asymmetric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Apr 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edolstra commented Jan 12, 2022

nixos-discourse commented Apr 26, 2022

nixos-discourse commented May 3, 2022

Ericson2314 commented Oct 23, 2022

L-as commented Oct 23, 2022

das-g commented Oct 23, 2022 • edited Loading

L-as commented Oct 23, 2022

tomberek commented Oct 24, 2022

Summary

History

Summary of internals

next step brainstorm

Steps

dependencies

recursive-nix followup

roberth commented Mar 30, 2023

nixos-discourse commented Sep 9, 2023

nixos-discourse commented Dec 20, 2023

nixos-discourse commented Oct 1, 2024

Ericson2314 commented Apr 26, 2021 •

edited

Loading

Ericson2314 Apr 27, 2021 •

edited

Loading

das-g commented Oct 23, 2022 •

edited

Loading