Update to work with nixpkgs unstable

[bcdarwin]

Just a bunch of updates, trying to get things simplified a bit by using nixpkgs expressions when possible before doing any refactoring. Currently one big monolithic commit -- can be split out into more logical pieces before final review. Also note I have not built the full GPU-ized version here.

A few issues:

• sometimes it's not clear why a package has been overridden -- in future this should probably be indicated with a comment when it's not obvious.
• not all Python packages existing in the overlay are actually built by nix develop
• it's not clear which packages are actually needed by us and which are there for demonstration purposes (the list of dependencies seems long enough I suspect the former)
• many of the included packages have no tests (also a problem for nixpkgs python+js packages tbh)
• as a result there's no clear acceptance criteria for the repo.

Nonetheless, initial comments welcome.

[bcdarwin]

So, currently we cannot build this package set due to NixOS/nixpkgs#217878, preventing (at least) torchvision from building. It's not trivial to resolve this until NVIDIA releases a CUDNN compatible with CUDA 12.

Currently, options include:

1. build torchvision without CUDA support (unsure how speed/memory of our models will be affected)
2. revert part or all of the gcc11Stdenv -> gcc12Stdenv bump (mass rebuild, unclear extent and whether it will work)
3. keep the old package set for now.

At the moment we're leaning toward keeping the old package set, so marking this draft for now.

[bcdarwin]

@cfhammill this should be ready to go -- once we think it's ready I'll split apart the commit before merging.

[bcdarwin]

So, currently we cannot build this package set due to NixOS/nixpkgs#217878, preventing (at least) torchvision from building. It's not trivial to resolve this until NVIDIA releases a CUDNN compatible with CUDA 12.

Currently, options include:

1. build torchvision without CUDA support (unsure how speed/memory of our models will be affected)

2. revert part or all of the gcc11Stdenv -> gcc12Stdenv bump (mass rebuild, unclear extent and whether it will work)

3. keep the old package set for now.

At the moment we're leaning toward keeping the old package set, so marking this draft for now.

This has been mostly addressed by some recent CUDA-related fixes in Nixpkgs. The main outstanding issue for us is NixOS/nixpkgs#220341, which stops us from building grad-cam or from enabling cudaSupport globally (since this also breaks python310Packages.jax which is somehow in our dependency chain).

[cfhammill]

So, currently we cannot build this package set due to NixOS/nixpkgs#217878, preventing (at least) torchvision from building. It's not trivial to resolve this until NVIDIA releases a CUDNN compatible with CUDA 12.
Currently, options include:

1. build torchvision without CUDA support (unsure how speed/memory of our models will be affected)

2. revert part or all of the gcc11Stdenv -> gcc12Stdenv bump (mass rebuild, unclear extent and whether it will work)

3. keep the old package set for now.

At the moment we're leaning toward keeping the old package set, so marking this draft for now.

This has been mostly addressed by some recent CUDA-related fixes in Nixpkgs. The main outstanding issue for us is NixOS/nixpkgs#220341, which stops us from building grad-cam or from enabling cudaSupport globally (since this also breaks python310Packages.jax which is somehow in our dependency chain).

I think jax is sneaking in via optuna, not bad to have around though. Does this mean jax can't be built with CUDA support at all (like grad-cam) or just doesn't work with cuda enabled globally?

[bcdarwin]

So, currently we cannot build this package set due to NixOS/nixpkgs#217878, preventing (at least) torchvision from building. It's not trivial to resolve this until NVIDIA releases a CUDNN compatible with CUDA 12.
Currently, options include:

1. build torchvision without CUDA support (unsure how speed/memory of our models will be affected)

2. revert part or all of the gcc11Stdenv -> gcc12Stdenv bump (mass rebuild, unclear extent and whether it will work)

3. keep the old package set for now.

At the moment we're leaning toward keeping the old package set, so marking this draft for now.

This has been mostly addressed by some recent CUDA-related fixes in Nixpkgs. The main outstanding issue for us is NixOS/nixpkgs#220341, which stops us from building grad-cam or from enabling cudaSupport globally (since this also breaks python310Packages.jax which is somehow in our dependency chain).

I think jax is sneaking in via optuna, not bad to have around though. Does this mean jax can't be built with CUDA support at all (like grad-cam) or just doesn't work with cuda enabled globally?

The latter; I've added jaxlibWithCuda to the dependencies for now (until config.enableCuda works) to verify that we have the correct build.

[cfhammill]

do you know why the build results are different for jaxWithCuda and with cuda enabled globally?

[bcdarwin]

do you know why the build results are different for jaxWithCuda and with cuda enabled globally?

Hits nixpkgs/#220341 when config.enableCuda. Hopefully fixed soon though.