Development (#29)

* cmake update

* submodule update

* Fix pip build

* Update license

* Update license

* license update

* fix license issue

* Test AVX

* Ignore .vscode folder

* Cleanup pip build

* Cleanup pip build

* Try building with cibuildwheel

* Build wheel

* wheel build

* build wheel

* Improve imports

* Fix compilation

* Update author list

* Fix pip build

* Add option to build library

* Add license

* Add license

* Gitlab build wheel

* Add missing dependency

* Add reference

* Add wheel build

* Update .gitlab-ci.yml file

* Gitlab build

* Update .gitlab-ci.yml file

* Update submodule in wheel build

* Update submodule

* twine upload

* add build tags

* Twine

* Fix pyi classifier

* upgrade pip

* Upload linux wheels

* Upload non master to testpypi

* Add windows wheel

* windows wheel

* windows wheel

* Windows wheel

* Skip existing wheels

* Use prebuild docker img for windows

* Windows wheels

* skip win32 and pypy

* FFTW dir in windows build

* Compiler flags for HDF5 and fftw

* Use fftw only with ICS

* Add swig

* Mask hdf5 dependency

* Mask hdf5

* Fix if close

* Hdf5 issue

* Static boost linking

* Windows boost linking

* win boost

* remove boost system

* static boost

* boost exception issue

* Use vcpkg on win

* Static boost builds

* avoid boost

* Fix typo

* auto link

* linking

* Build photon hdf by default

* Remove openmp dependency

* remove boost::filesystem and locale depency

* remove boost:locale

* Enable language

* remove vcpkg

* only boost header

* remove pip swig cmake boost

* add cmake

* Use cmake findpython

* switch findpzthon3 for findpython

* Set  DPython_ROOT_DIR in setup.py

* Add swig

* remove cmake swig

* remove cmake submodule

* move cmake files from submodule to folder

* remove cmake submodule

* Add pocketfft

* Reduce boost dependency

* Remove unused import

* Move other peoples code to external

* Move other peoples code to external

* Move to external

* Refactoring

* refactor

* Add itlib

* Restore

* Add aglib

* Add aglib

* Add optimization wrapper

* Add optimization wrapper

* Add msstl

* use itlib flat_set

* Cleanup

* Cleanup

* bump version

* separate external include files

* cleanup

* Add documentation

* win issue

* Use BUILD_PYTHON_DOCS

* Win issue

* win build issue

* Move external thirdparty code

to highlight other projects code
external third party code is moved
to thirdparty folder.

* Remove boost dependency

* Static linking of boost

remove uneeded boost dependency and
link (just in case) boost statically

* switch fftw for pocketfftw

* switch fftw for pocketfftw

* fftw -> pocketfft

* Fix typos

* remove boost fftw

* update license info

* remote fftw

* cleanup namespace

* Add windows wheel Dockerfile

* Add wheel:win build

* Remove unused config

* fix indentation

* Fix typo

* use cmd.exe for wheel:win

* use cmd for whee:win

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Simplify test

* fix skip on linux

* fix wheel:win pypi upload

* Add pwsh to docker

* Update .gitlab-ci.yml file

* fix

* xx

* ff

* hfd

* Update file .gitlab-ci.yml

* Update file pocketfft_hdronly.h

* Add example writing CLSM data

* Windows

* new c++ on osx

* Dynamic hdf5 lib

* fix call

* wheel build

* add header setter

* doc fixes

* add windows wheel

* docs

* docs

* add osx:wheel

* macos wheel

* osx fixes

* macos wheel

* use modern c++ on osx

* wheel osx fix

* doc fix

* Fix badges and license

* Removed TODO

* doc fixes

* DOC FIX

* Also upload development docs

* doc build fix

* Docs buil

* docs building

* fix doc

* fix doc

* Redirect RTD page to docs

* Support more py versions

* docs deposit

* pin boost

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Compile wheel for py311

* Set min macos version

* Update .gitlab-ci.yml

* Update .gitlab-ci.yml

* Specify mmacosx version

* Update .gitlab-ci.yml

* Cleanup

* Cleanup

* Add read clsm settings method

* Add mask flip

* Fix typo

* Add option for mask

* Change default masking.

* Bump version

* Pypi 2fa requires tokens

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Replace old np.float types

* Replace old np.float types

* Fix TTTRMask issue

need to flip

* Improved documentation

* Separate doc from doc-tttrlib env

* Fix typo

* Fix typo

* Fix bixtex issue

* Fix doxygen issue

* Bump version

* Improve docstrings

* Additional sphinx modules

* Fix typo

* Fix typo

* Fix issue of deprecated std

* Remove missing sphinx-extension

* Fix windows build

* Fix doc deploy

* Fix doc deploy

* Fix doc deploy

* Fix doc deploy

* Fix Pda histogram_function

* fix wrong path

* CI gitlab github sync

* Fix doc issues

* Add stubs for doc

* Add stubs for doc

* Fix wrong path

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml file

* Update .gitlab-ci.yml

* Update pyproject.toml

* Add github action

* Remove gitlab webide config

* Remove gitlab webide config

* Remove github action demo

---------

Co-authored-by: tpeulen <[email protected]>
Co-authored-by: Thomas-Otavio Peulen <[email protected]>
Co-authored-by: Administrator <[email protected]>

.github/workflows/gitlab-sync.yml

@@ -0,0 +1,22 @@
+name: GitlabSync
+
+on:
+  - push
+  - delete
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    name: Git Repo Sync
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+      - uses: wangchucheng/[email protected]
+        with:
+          # Such as https://github.com/wangchucheng/git-repo-sync.git
+          target-url: ${{ secrets.TARGET_URL }}
+          # Such as wangchucheng
+          target-username: ${{ secrets.TARGET_USERNAME }}
+          # You can store token in your project's 'Setting > Secrets' and reference the name here. Such as ${{ secrets.ACCESS\_TOKEN }}
+          target-token: ${{ secrets.TARGET_TOKEN }}

.gitignore

@@ -1,8 +1,10 @@
 build/
 bld-dir/
 dist/
-doc/examples
+doc/auto_examples
 tmp/
+.vscode
+.vs
 .ipynb_checkpoints
 cmake-build-*/
 temp/
@@ -11,8 +13,8 @@ __pycache__
 .vscode/
 tttr-data
 .idea/
-cmake/
 tools/
+leftovers/
 
 # Documentation and tools
 doc/_*

.gitlab-ci.yml

@@ -1,115 +1,205 @@
+variables:
+  GIT_SSL_NO_VERIFY: "1"
+
 stages:
-- build
-- test
-- deploy
+#  - sync
+  - build
+  - test
+  - deploy
+
+# github:push:
+#   stage: sync
+#   image: condaforge/mambaforge
+#   tags:
+#     - linux
+#   before_script:
+#     - git submodule update --init --recursive --remote
+#   script:
+#     - git switch -c ${CI_COMMIT_REF_NAME}
+#     - git push --force https://${GITHUB_TOKEN}@github.com/fluorescence-tools/imp.bff
+
 
+# general definitions
 .build: &build
   stage: build
   artifacts:
     paths:
       - bld-dir/
-.build_posix: &build_posix
-  <<: *build
+.conda:build:posix: &conda_build_posix
+  <<: *build  
   script:
     - ./tools/build.sh
-.build:lnx: &build_lnx
-  <<: *build_posix
+.conda:build:lnx: &conda_build_lnx
+  <<: *conda_build_posix
   before_script:
     - source "/opt/conda/etc/profile.d/conda.sh"
     - conda activate /opt/conda/
+    - git config --global safe.directory '*'
     - git submodule update --init --recursive --remote
-build:lnx_x86:
-  <<: *build_lnx
+
+# linux conda builds
+conda:build:lnx:x86:
+  <<: *conda_build_lnx
   image:
-    name: condaforge/linux-anvil-cos7-x86_64
     entrypoint: [ "/bin/bash", "-i", "-c" ]
+    name: condaforge/linux-anvil-cos7-x86_64
   tags:
+    - linux
     - x86_64
-build:lnx_ppc64le:
-  <<: *build_lnx
+conda:build:lnx:ppc64le:
+  <<: *conda_build_lnx
   image:
-    name: condaforge/linux-anvil-ppc64le
     entrypoint: [ "/bin/bash", "-i", "-c" ]
+    name: condaforge/linux-anvil-ppc64le
   tags:
+    - linux
     - ppc64le
-build:lnx_aarch64:
-  <<: *build_lnx
+conda:build:lnx:aarch64:
+  <<: *conda_build_lnx
   image:
-    name: condaforge/linux-anvil-aarch64
     entrypoint: [ "/bin/bash", "-i", "-c" ]
+    name: condaforge/linux-anvil-aarch64
   tags:
+    - linux
     - aarch64
-build:osx:
-  <<: *build
-  <<: *build_posix
+
+conda:build:osx:
+  <<: *conda_build_posix
   tags:
     - osx
   before_script:
     - git submodule update --init --recursive --remote
-build:windows:
+
+conda:build:windows:
   <<: *build
-  image: mambaforge:vs16
   tags:
     - win
   script:
-    - cmd.exe
     - conda activate base
-    - cd tools && git pull --force && cd..
     - .\tools\build.bat
 
-test:linux:
+conda:test:linux:
   stage: test
   tags:
-    - local
+    - local # go for a server marked as local
     - linux
   image: condaforge/mambaforge
-  needs: ["build:lnx_x86"]
+  needs: ["conda:build:lnx:x86"]
   before_script:
     - |
         export DEBIAN_FRONTEND=interactive
         export TZ=Europe/Berlin
         apt-get update -qy && apt-get install -qy git nfs-common # sshfs
   script:
     - |
-        # requires runner in privileged mode see (https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4122)
         source activate base
         conda config --add channels "file://`pwd`/bld-dir"
-        git clone --depth 1 https://gitlab.peulen.xyz/skf/tttrlib && cd tttrlib && git fetch && git switch $CI_COMMIT_REF_NAME
+        git clone https://gitlab.peulen.xyz/skf/tttrlib && cd tttrlib && git fetch && git switch $CI_COMMIT_REF_NAME
         mamba install python tttrlib nose scipy
-        mount.nfs -o nolock 192.168.124.254:tank/www /srv
+        # requires runner in privileged mode see
+        mount.nfs -o nolock 192.168.124.254:/mnt/user/www /srv
         # sshfs -o password_stdin -oStrictHostKeyChecking=no -o Ciphers=aes128-ctr -o Compression=yes -p 2222 [email protected]:/www www <<< $FTP_PASSWORD
         ln -s /srv/tttr-data tttr-data
         nosetests test
 
-deploy:conda:
+conda:deploy:
   stage: deploy
   image: condaforge/mambaforge
   tags:
-    - x86_64
     - linux
   before_script:
     - git submodule update --init --recursive --remote
   script:
     - ./tools/deploy.sh
 
 deploy:doc:
-# requires runner in privileged mode (setup in config.toml)
+   # requires runner in privileged mode (setup in config.toml)
    stage: deploy
    image: condaforge/mambaforge
    tags:
      - local
      - linux
-   needs: ["build:lnx_x86"]
+   needs: ["conda:build:lnx:x86"]
    before_script:
     - |
         export DEBIAN_FRONTEND=interactive
         export TZ=Europe/Berlin
         apt-get update -qy && apt-get install -qy git nfs-common make
    script:
-    - |
-        source activate base
-        git clone --depth 1 https://gitlab.peulen.xyz/skf/tttrlib && cd tttrlib && git fetch && git switch $CI_COMMIT_REF_NAME
-        mount.nfs -o nolock 192.168.124.254:tank/www /srv && ln -s /srv/tttr-data tttr-data
-        cd doc && mamba env update --file environment.yml && conda activate doc
-        make html
-        if [[ "$CI_COMMIT_REF_NAME" == "master" ]]; then rm -rf /srv/docs.peulen.xyz/tttrlib && mv -v _build/html/stable /srv/docs.peulen.xyz/tttrlib; fi
+     - conda config --add channels "file://`pwd`/bld-dir"
+     - mount.nfs -o nolock 192.168.124.254:/mnt/user/www /srv
+     - ln -s /srv/tttr-data tttr-data
+     - cd doc
+     - source activate base
+     - mamba env create -f environment.yml
+     - conda activate doc-tttrlib
+     - mamba install tttrlib --use-local
+     - make html
+     - rm -rf /srv/docs.peulen.xyz/project/$CI_PROJECT_TITLE/$CI_COMMIT_REF_NAME
+     - mv -v _build/html/stable /srv/docs.peulen.xyz/project/$CI_PROJECT_TITLE/$CI_COMMIT_REF_NAME
+
+wheel:linux:
+  stage: build
+  tags:
+    - linux  
+  image: python:3.8
+  # make a docker daemon available for cibuildwheel to use
+  services:
+    - name: docker:dind
+      entrypoint: ["env", "-u", "DOCKER_HOST"]
+      command: ["dockerd-entrypoint.sh"]
+  variables:
+    DOCKER_HOST: tcp://docker:2375/
+    DOCKER_DRIVER: overlay2
+    # See https://github.com/docker-library/docker/pull/166
+    DOCKER_TLS_CERTDIR: ""
+  script:
+    - curl -sSL https://get.docker.com/ | sh
+    - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
+    - python -m pip install cibuildwheel==2.12.1
+    - git submodule update --init --recursive --remote
+    - cibuildwheel --output-dir bld-dir --platform linux
+    - python -m pip install --upgrade pip
+    - pip install twine
+    - export TWINE_PASSWORD="$PYPI_TOKEN"
+    - if [ "$CI_COMMIT_REF_NAME" != "master" ]; then export TWINE_REPOSITORY=testpypi; export TWINE_PASSWORD="$PYPI_TEST_TOKEN"; fi
+    - twine upload --skip-existing bld-dir/*.whl
+  artifacts:
+    paths:
+      - bld-dir/
+
+wheel:osx:
+  stage: build
+  tags:
+    - osx  
+  script:
+    - conda deactivate
+    - python3 -m pip install cibuildwheel==2.12.1
+    - git submodule update --init --recursive --remote
+    - cibuildwheel --output-dir bld-dir --platform macos
+    - python3 -m pip install --upgrade pip
+    - python3 -m pip install twine
+    - export TWINE_PASSWORD="$PYPI_TOKEN"
+    - if [ "$CI_COMMIT_REF_NAME" != "master" ]; then export TWINE_REPOSITORY=testpypi; export TWINE_PASSWORD="$PYPI_TEST_TOKEN"; fi
+    - python3 -m twine upload --skip-existing bld-dir/*.whl
+  artifacts:
+    paths:
+      - bld-dir/
+
+wheel:windows:
+  <<: *build  
+  image: wheel:win
+  script:
+      - |
+        conda deactivate
+        set-content $env:public\inline.cmd -Value @'
+        call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
+        py -m cibuildwheel --output-dir bld-dir --platform windows
+        set TWINE_PASSWORD=%PYPI_TOKEN%
+        if %CI_COMMIT_REF_NAME% NEQ master set TWINE_REPOSITORY=testpypi
+        if %CI_COMMIT_REF_NAME% NEQ master set TWINE_PASSWORD=%PYPI_TEST_TOKEN%
+        twine upload --skip-existing bld-dir/*.whl
+        '@
+        CMD.EXE /C $env:public\inline.cmd
+  tags:
+    - win

.gitmodules

@@ -1,6 +1,3 @@
 [submodule "tools"]
 	path = tools
 	url = https://gitlab.peulen.xyz/build/tools
-[submodule "cmake"]
-	path = cmake
-	url = https://gitlab.peulen.xyz/build/cmake

.readthedocs.yaml

@@ -0,0 +1,16 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+   configuration: doc/rtd/conf.py

AUTHORS

@@ -6,6 +6,7 @@
 # source control.
 
 Sali Lab, UCSF
+Heinze Lab, University Wuerzburg
 Thomas-Otavio Peulen
 
 AG Seidel Lab, HHU Dueseldorf