[Bug]: Including jupyenv in flake causes nix evaluation to OOM #525

turion · 2023-10-02T13:14:40Z

Current Behavior

In tweag/monad-bayes@5d08e7c I bumped our jupyenv version to revision 97f76f9. This causes nix develop to go into a memory-leaking infinite loop.

Expected Behavior

Just evaluating nix develop

Steps To Reproduce

Check out tweag/monad-bayes@5d08e7c
nix develop

OS

- system: `"x86_64-linux"`
 - host os: `Linux 6.1.51, NixOS, 23.11 (Tapir), 23.11.20230904.f53ec4f`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - nixpkgs: `/nix/store/mwy5r2z2c1v4sgya3v5rszs7fmk89hzc-source`

Version

97f76f9

Additional Context

Running nix develop -vvv gives this ever repeating loop:

Relevant log output

instantiated 'source' -> '/nix/store/6xf538aynk7v7g573rhdd973vskip143-source.drv'
instantiated 'python3.10-bootstrap-flit-core-3.9.0' -> '/nix/store/ywzpzlkmp0rbmz21ziwwjbwgynqfakg2-python3.10-bootstrap-flit-core-3.9.0.drv'
instantiated 'source' -> '/nix/store/kzp5mfwd5p759mkgwia1dgcgsddbr9my-source.drv'
instantiated 'python3.10-bootstrap-installer-0.7.0' -> '/nix/store/garcn982ah65akbry7wzxpasb13amrqk-python3.10-bootstrap-installer-0.7.0.drv'
instantiated 'pyproject_hooks-1.0.0.tar.gz' -> '/nix/store/7lg3d41h9gslk4k47ydcgvaizki69axy-pyproject_hooks-1.0.0.tar.gz.drv'
instantiated 'python3.10-bootstrap-pyproject-hooks-1.0.0' -> '/nix/store/0j4ksf6yhmasl45wig4ifvys2y218pcy-python3.10-bootstrap-pyproject-hooks-1.0.0.drv'
instantiated 'packaging-23.0.tar.gz' -> '/nix/store/0khi1cnwxi1840lxlbmx91gil1zd1jvc-packaging-23.0.tar.gz.drv'
instantiated 'python3.10-bootstrap-packaging-23.0' -> '/nix/store/0lpni15gfp8acx7a2dga3fhz21qkp2v6-python3.10-bootstrap-packaging-23.0.drv'
instantiated 'tomli-2.0.1.tar.gz' -> '/nix/store/i08s0mz38gj8ixll8qphxg3l4digvxgh-tomli-2.0.1.tar.gz.drv'
instantiated 'python3.10-bootstrap-tomli-2.0.1' -> '/nix/store/b68wnrpfvsgcpirrf74b9rz44wgljbmz-python3.10-bootstrap-tomli-2.0.1.drv'
instantiated 'source' -> '/nix/store/w6fl1dqjvc7q74ris8m6kprmajnqsd4y-source.drv'
instantiated 'python3.10-bootstrap-build-0.10.0' -> '/nix/store/ss8bagnad1p9gvkj48yhx077jkihm87x-python3.10-bootstrap-build-0.10.0.drv'
instantiated 'pypa-build-hook.sh' -> '/nix/store/syv9j21gnivl1yvbqbl1cgl55kmky8j4-pypa-build-hook.sh.drv'
instantiated 'python-setup-hook.sh' -> '/nix/store/1ggi1d0idq4ikpxqv886sd7f9467psfc-python-setup-hook.sh.drv'
instantiated 'Python-3.10.12.tar.xz' -> '/nix/store/a5hx93i2r3x2g00kd5ywb2phj91hfq59-Python-3.10.12.tar.xz.drv'
instantiated 'python3-3.10.12' -> '/nix/store/fbyjcpdqv92z8fc85plz3m6j4f0m2jv2-python3-3.10.12.drv'
instantiated 'python-setup-hook.sh' -> '/nix/store/1ggi1d0idq4ikpxqv886sd7f9467psfc-python-setup-hook.sh.drv'
instantiated 'Python-3.10.12.tar.xz' -> '/nix/store/a5hx93i2r3x2g00kd5ywb2phj91hfq59-Python-3.10.12.tar.xz.drv'
instantiated 'python3-3.10.12' -> '/nix/store/fbyjcpdqv92z8fc85plz3m6j4f0m2jv2-python3-3.10.12.drv'
instantiated 'wrap-python-hook' -> '/nix/store/ipxz7lkjp6jjr6xkcvz33jal354154gh-wrap-python-hook.drv'
instantiated 'python-remove-tests-dir-hook' -> '/nix/store/13cyxa5v1syj13xmzsccs0s9bq98gxcg-python-remove-tests-dir-hook.drv'
instantiated 'python-remove-bin-bytecode-hook' -> '/nix/store/9ch8hyg0zjccvav10pmrs92a7lkswmrl-python-remove-bin-bytecode-hook.drv'

The text was updated successfully, but these errors were encountered:

turion · 2023-10-02T13:16:47Z

I can observe a memory leak with v0.1.0 (37c9075) as well. The last commit that worked was: 3ad2c95

turion · 2023-10-02T13:31:20Z

Sorry, it seems there is some interaction with a particular nixpkgs version, maybe it's not jupyenv's fault. Let me close until I understand it better.

turion · 2023-10-02T13:37:33Z

This problem seems to be solved after removing nixpkgs.follows = "nixpkgs"; in my code.

turion · 2023-10-02T14:03:54Z

I'm afraid the problem does indeed persist. After updating with nix flake update, the problem is back because for some reason I don't understand, it does update the jupyenv/nixpkgs input.

turion · 2023-10-02T14:05:02Z

It seems that jupyenv doesn't work with nixpkgs revision f5892ddac112a1e9b3612c39af1b72987ee5783a.

turion · 2023-10-02T14:41:09Z

A working combination is jupyenv 97f76f9 with nixpkgs revision 18036c0be90f4e308ae3ebcab0e14aae0336fe42.

A few haskell packages can't be built, so some notebooks will not compile. Still, we're in a better place than before. I also had to remove jupyterEnvironment from the mkShell, otherwise there would be an infinite loop: (tweag/jupyenv#525)

turion · 2023-10-02T16:02:00Z

My minimal reproducer:

{
  description = "A very basic flake";

  inputs = {
    jupyenv = {
      inputs.nixpkgs.follows = "nixpkgs";
      url = "github:tweag/jupyenv";
    };
  };

  outputs = { self, nixpkgs, jupyenv }:
  let system = "x86_64-linux";
  in {
    devShells.x86_64-linux.default = with import nixpkgs { inherit system; }; mkShell {
      packages = [
        (jupyenv.lib.${system}.mkJupyterlabNew {})
      ];
    };

    packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;

    packages.x86_64-linux.default = self.packages.x86_64-linux.hello;

  };
}

Running nix develop in a folder with this file triggers the bug.

The two input locks:

jupyenv: github:tweag/jupyenv/97f76f92fa14814e955a4f6847eaa4198525a19e
nixpkgs: github:NixOS/nixpkgs/f902cb49892d300ff15cb237e48aa1cad79d68c3

A few haskell packages can't be built, so some notebooks will not compile. Still, we're in a better place than before. I also had to remove jupyterEnvironment from the mkShell, otherwise there would be an infinite loop: (tweag/jupyenv#525) Further, I had to remove haskell-flake-utils, it seems not to be supported sufficiently. This dropped cabal-docspec support.

A few haskell packages can't be built, so some notebooks will not compile. Still, we're in a better place than before. Further, I had to remove haskell-flake-utils, it seems not to be supported sufficiently. This dropped cabal-docspec support. I also had to remove the nixpkgs follow input, otherwise there would be an infinite loop: (tweag/jupyenv#525)

GTrunSec · 2023-10-02T19:41:59Z

  # FIXME: pkgs.poetry is segfaulting on nixpkgs-unstable & poetry2nix
https://github.com/nix-community/poetry2nix/issues/1291
https://github.com/nix-community/poetry2nix/issues/1316

see #524

GTrunSec · 2023-10-02T20:01:16Z

please try: nix-community/poetry2nix#1329

turion · 2023-10-04T07:38:37Z

Indeed, adding inputs.jupyenv.inputs.poetry2nix.url = "github:nix-community/poetry2nix/?ref=refs/pull/1329/head"; fixes the issue.

turion · 2023-10-04T10:21:01Z

To illustrate the bug, I created https://github.com/turion/jupyenv-bug/. Reproduce like this:

$ git clone [email protected]:turion/jupyenv-bug.git

$  git branch --show-current
main

$ nix develop # This will work fine, feel free to abort

$ git checkout follow

$  git diff main -- flake.nix
diff --git a/flake.nix b/flake.nix
index 123f264..946ad46 100644
--- a/flake.nix
+++ b/flake.nix
@@ -3,6 +3,7 @@
 
   inputs = {
     jupyenv = {
+      inputs.nixpkgs.follows = "nixpkgs";
       url = "github:tweag/jupyenv";
     };
   };

$ nix develop # This will OOM, be sure to cancel before your machine becomes unresponsive!

$ git checkout resolution

$  git diff main -- flake.nix
diff --git a/flake.nix b/flake.nix
index 123f264..7b13f75 100644
--- a/flake.nix
+++ b/flake.nix
@@ -3,6 +3,8 @@
 
   inputs = {
     jupyenv = {
+      inputs.nixpkgs.follows = "nixpkgs";
+      inputs.poetry2nix.url = "github:nix-community/poetry2nix/?ref=refs/pull/1329/head";
       url = "github:tweag/jupyenv";
     };
   };

$ nix develop # This will work again

turion added the bug Something isn't working label Oct 2, 2023

turion assigned djacu Oct 2, 2023

turion closed this as completed Oct 2, 2023

turion reopened this Oct 2, 2023

GTrunSec mentioned this issue Oct 2, 2023

Refactor jupyenv #524

Open

teto mentioned this issue Nov 1, 2023

nix run .#jupyter-all generates errors that make it unusable NixOS/nixpkgs#264840

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Including jupyenv in flake causes nix evaluation to OOM #525

[Bug]: Including jupyenv in flake causes nix evaluation to OOM #525

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023 •

edited

Loading

GTrunSec commented Oct 2, 2023 •

edited

Loading

GTrunSec commented Oct 2, 2023

turion commented Oct 4, 2023

turion commented Oct 4, 2023

[Bug]: Including jupyenv in flake causes nix evaluation to OOM #525

[Bug]: Including jupyenv in flake causes nix evaluation to OOM #525

Comments

turion commented Oct 2, 2023

Current Behavior

Expected Behavior

Steps To Reproduce

OS

Version

Additional Context

Relevant log output

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023

turion commented Oct 2, 2023 • edited Loading

GTrunSec commented Oct 2, 2023 • edited Loading

GTrunSec commented Oct 2, 2023

turion commented Oct 4, 2023

turion commented Oct 4, 2023

turion commented Oct 2, 2023 •

edited

Loading

GTrunSec commented Oct 2, 2023 •

edited

Loading