Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add remote modules support for OTB nixpkgs #15

Merged
merged 12 commits into from
Aug 31, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -9,6 +9,9 @@ on:
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
strategy:
fail-fast: false
matrix:
@@ -28,6 +31,7 @@ jobs:
http-connections = 128
max-substitution-jobs = 128
extra-platforms = aarch64-linux
- uses: DeterminateSystems/magic-nix-cache-action@main
- name: Build system
run: |
nix build --system ${{ matrix.machine.platform }} .
nix build --system ${{ matrix.machine.platform }} .#otb-dev
7 changes: 5 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
build_docker_x86_64:
nix run .\#otb-docker-x86_64.copyToDockerDaemon
build_docker:
nix run .\#otb-docker.copyToDockerDaemon

build_dev_docker:
nix run .\#otb-dev-docker.copyToDockerDaemon

#build_docker_arch64:
# nix run .\#otb-docker-aarch64.copyToDockerDaemon
144 changes: 130 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -70,11 +70,13 @@ compressor for `Zarr` dataset. [Related Issue](https://github.com/remicres/otbtf
Nix pacakge solved those issues as `GDAL` on `Nixpkgs` already has those.
- One can combine OTB with different geospatial python libraries such as `rasterio`, `geopandas` etc.
which as already available with [Nixpkgs](https://search.nixos.org/packages) no need to create a separate environment.
- As of today July 2024, there isn't any conda package for OTB, although it is under plan;
- There isn't any conda package for OTB, although it is under plan;
this Nixpkgs should help people who want to use `OTB` with a different python version or packages
such as (gdal,rasterio, geopandas etc.), which should solve most of those use cases for OTB to be
imported in an custom python environment.

imported in an custom python environment [As of today August 2024,].
- One can also build multi-arch version of OTB package apart from `AMD64 (x86_64-linux)` i.e.
`ARM64 (aarch64-linux)` as currently OTB has no `aarch64-linux` version.
- One can easily build docker image as well.

## Advanced Configuration

@@ -156,19 +158,26 @@ Here is an example of how to create an `flake.nix` with all the above python pac
## Docker
- Docker Image for OTB:
- Linux AMD64
- Linux ARM64: To build OTB for `linux-aarach-64` there are 3 options:
- Linux ARM64: To build OTB for `linux-aarch-64` there are 3 options:
- The easiest way would be to build natively with ARM64 (Still needs to be tested)
- Complie using an emulator (Tested with GitHub Action and it builds fine)
- Compile using an emulator (Tested with [GitHub Action](https://github.com/daspk04/otb-nix/actions/runs/10191427474) and it builds fine)
- We can also via nix cross compiler (Still needs to be Tested)

One can build a docker image for OTB with python support (default).
1) One can build a docker image for OTB with python support and no remote modules (default).
```
1) Clone this repo
2) make build_docker_x86_64
2) make build_docker
2) docker run -it --rm otb
```
Example: How build a docker image based on `OTB`, `pyotb`, `gdal` and `rasterio`
without cloning this repo.
2) One can build a docker image for OTB with python support and all the remote modules.
```
1) Clone this repo
2) make build_dev_docker
2) docker run -it --rm otb-dev
```

### Example 1:
##### Build a docker image based on `OTB`, `pyotb`, `gdal` and `rasterio` without cloning this repo.

1) Create a local directory
2) Create a `flake.nix` as shown below which already has the required python packages and as
@@ -184,10 +193,9 @@ docker.nix
4) Run command as mentioned below:

```
1) nix run .\#otb-docker-x86_64.copyToDockerDaemon
1) nix run .\#otb-docker.copyToDockerDaemon
2) docker run -it --rm otb
```

Example of `flake.nix` with `OTB`, `Gdal`, `Pyotb` and `Rasterio`:
```nix
{
@@ -245,7 +253,7 @@ Example of `flake.nix` with `OTB`, `Gdal`, `Pyotb` and `Rasterio`:
};
in rec {
packages = {
otb-docker-x86_64 = pkgs.callPackage ./docker.nix {
otb-docker = pkgs.callPackage ./docker.nix {
inherit pkgs otb nix2containerPkgs;
img-name = "otb";
img-tag = "latest";
@@ -272,11 +280,112 @@ Example of `flake.nix` with `OTB`, `Gdal`, `Pyotb` and `Rasterio`:
}
);
}
```

### Example2:
Build a docker image based on `OTB`, `pyotb`, `gdal` and `rasterio` without cloning this repo and some specific remote modules

1) The steps and the command are same as `Example1` except we just change the `flake.nix` file
and the `otb` section for only remote modules.


Example of `flake.nix` with only activated remote modules such as `prefetch` and `otbtf` [note on OTBTF](#note)
```nix
{
description = "A flake for Orfeo Toolbox";

inputs = {
nixpkgs.url = "github:nixos/nixpkgs/24.05";
nixpkgs-unstable.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
flake-utils.url = "github:numtide/flake-utils";
otbpkgs.url = "github:daspk04/otb-nix";
nix2container.url = "github:nlewo/nix2container";
};

outputs = {
self,
nixpkgs,
nixpkgs-unstable,
nix2container,
flake-utils,
otbpkgs,
}:
flake-utils.lib.eachDefaultSystem (
system: let
pkgs = import nixpkgs {inherit system;};
nix2containerPkgs = nix2container.packages.${system};
python = pkgs.python312;
pyPkgs = python.pkgs;

## we only enable Prefetch and Otbtf remote modules
otb = pkgs.callPackage ./pkgs/otb/. {
inherit system;
shark = packages.shark;
itk_4_13 = packages.itk_4_13;
gdal = gdal;
python3 = python; # build otb with fixed python version
enablePython = true;
enablePrefetch = true;
enableOtbtf = true;
};

otbPath = with pkgs; pkgs.lib.makeLibraryPath [otb];

pyotb = pyPkgs.buildPythonPackage rec {
pname = "pyotb";
version = "2.0.3.dev2";
format = "pyproject";
docheck = false;

src = builtins.fetchGit {
name = pname;
url = "https://github.com/orfeotoolbox/pyotb.git";
ref = "refs/tags/${version}";
rev = "de801eae7e2bd80706801df4a48b23998136a5cd";
};

nativeBuildInputs = with pyPkgs; [
setuptools
];

propagatedBuildInputs = with pyPkgs; [
numpy
];
};
in rec {
packages = {
otb-docker = pkgs.callPackage ./docker.nix {
inherit pkgs otb nix2containerPkgs;
img-name = "otb";
img-tag = "latest";
python3 = python;
extra-python-packages = with pyPkgs; [pyotb gdal rasterio];
};
};
devShells.default = pkgs.mkShell rec {
packages = with pkgs; [
bashInteractive
pyPkgs.gdal
pypkgs.rasterio
pyPkgs.python
pyPkgs.venvShellHook
pyotb
otb
];
venvDir = "./.venv";

postShellHook = ''
export PYTHONPATH="$PYTHONPATH:${otbPath}/otb/python"
'';
};
}
);
}
```




## Develop
- In case one needs to develop or experiment then
- Clone this repo and make changes and push to your repository
@@ -289,7 +398,8 @@ otbpkgs.url = "github:{userName}/{repoName}?ref={branchName}";
### How to build OTB locally
```markdown
1) clone this repo
2) nix build #.otb
2) nix build #.otb
or # nix build #.otb-all --> this will build OTB with all the remote modules
3) ./result/bin/otbcli_BandMathX -help
```

@@ -312,7 +422,13 @@ Nix should be installed and flakes should be enabled.

## TODO
- [X] Build a Nix Docker for the OTB package [Linux AMD64 and Linux ARM64]
- [ ] Build OTB with [remote modules]((https://www.orfeo-toolbox.org/CookBook/RemoteModules.html)) (OTBTF, TimeSeriesGapFilling, etc)
- [X] Build OTB with [remote modules]((https://www.orfeo-toolbox.org/CookBook/RemoteModules.html)) (OTBTF, TimeSeriesGapFilling, etc)
- [ ] Build OTB with Tensorflow for OTBTF (OTBTF remote modules doesn't have Tensorflow)

## NOTE:
- The `OTBTF` build with `Nix` still doesn't have the `TF` applications such as `TensorflowModelServe`
`ImageClassifierFromDeepFeatures` `TensorflowModelTrain` `TrainClassifierFromDeepFeatures` `ImageClassifierFromDeepFeatures`
as the `TF` building from source via Nix is yet to be done by me.


#### **NOTE: This repo is currently experimental, please open an issue in case you encounter any.**
29 changes: 27 additions & 2 deletions docker.nix
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2024 Pratyush Das
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# https://github.com/efabless/openlane2/blob/main/nix/docker.nix
{
nix2containerPkgs,
python3,
@@ -9,6 +23,8 @@
pkgs,
...
}: let
user = "otbuser";
home-dir = "/home/${user}";
py-env = (
python3.withPackages (pp:
with pp;
@@ -24,7 +40,7 @@ in
nix2containerPkgs.nix2container.buildImage rec {
name = img-name;
tag = img-tag;

initializeNixDatabase = true;
copyToRoot = pkgs.buildEnv {
name = "root";
paths = with pkgs.dockerTools;
@@ -50,6 +66,7 @@ in
git
neovim-unwrapped
zsh
nix

# Library
otb
@@ -59,6 +76,12 @@ in

pathsToLink = ["/bin" "/etc" "/var" "/run" "/tmp" "/lib"];
postBuild = ''
if -e $HOME; then
echo "home directory exists"
exit 1
fi
mkdir -p $out/${home-dir}
mkdir -p ~/.config/nix/ && printf "experimental-features = nix-command flakes\n" > ~/.config/nix/nix.conf
mkdir -p $out/tmp
mkdir -p $out/etc
cat <<HEREDOC > $out/etc/zshrc
@@ -79,8 +102,10 @@ in
"EDITOR=nvim"
"PYTHONPATH=${py-env-sitepackages}:${otbLibPath}/otb/python"
"PATH=${py-env-bin}:${otbBinPath}:/bin"
"OTB_APPLICATION_PATH=${otbLibPath}/otb/applications"
"TMPDIR=/tmp"
"USER=${user}"
"HOME=${home-dir}"
];
};
}

54 changes: 48 additions & 6 deletions flake.nix
Original file line number Diff line number Diff line change
@@ -1,3 +1,16 @@
# Copyright 2024 Pratyush Das
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{
description = "A flake for Orfeo Toolbox";

@@ -25,11 +38,11 @@
# The minimal version of gdal and doesn't have arrow support but helps in leaner build
# we can mix as per requiremnt which will compile gdal from source
# https://github.com/NixOS/nixpkgs/blob/8c50662509100d53229d4be607f1a3a31157fa12/pkgs/development/libraries/gdal/default.nix#L7
# gdal = pkgs.gdalMinimal.override {python3 = python;};
# gdal = pkgs.gdalMinimal.override {python3 = python;
# useArrow = true;
# useHDF = true;
# useNetCDF = true;};
# gdal = pkgs.gdalMinimal.override {python3 = python;};
# gdal = pkgs.gdalMinimal.override {python3 = python;
# useArrow = true;
# useHDF = true;
# useNetCDF = true;};
gdal = pkgs.gdal; # gdal full version
pyPkgs = python.pkgs;
in rec {
@@ -45,7 +58,27 @@
enablePython = true;
};

otb-docker-x86_64 = pkgs.callPackage ./docker.nix {
otb-dev = pkgs.callPackage ./pkgs/otb/. {
inherit system;
shark = packages.shark;
itk_4_13 = packages.itk_4_13;
gdal = gdal;
python3 = python; # build otb with fixed python version
enablePython = true;
enablePrefetch = true;
enableOtbtf = true;
enableMLUtils = true;
enablePhenology = true;
enableBioVars = true;
enableGRM = true;
enableLSGRM = true;
enableSimpleExtraction = true;
enableTemporalGapfilling = true;
enableTimeSeriesUtils = true;
enableTemporalSmoothing = true;
};

otb-docker = pkgs.callPackage ./docker.nix {
inherit pkgs nix2containerPkgs;
img-name = "otb";
img-tag = "latest";
@@ -54,6 +87,15 @@
extra-python-packages = with pyPkgs; [packages.otb.propagatedBuildInputs];
};

otb-dev-docker = pkgs.callPackage ./docker.nix {
inherit pkgs nix2containerPkgs;
img-name = "otb-dev";
img-tag = "latest";
otb = packages.otb-dev;
python3 = python;
extra-python-packages = with pyPkgs; [packages.otb.propagatedBuildInputs];
};

# otb-aarch64 = pkgs.pkgsCross.aarch64-multiplatform.callPackage ./pkgs/otb/. {
# inherit system;
# shark = packages.shark;
Loading