Refactor to run all calculations in separate directories (#220)

Summary
-------
This refactor of the code contains several changes designed to make `koopmans` more robust and amenable to integration with `AiiDA`. Namely...
- each calculation is run in a separate directory following a regular pattern
- there are no shared `tmp` directories
- file reading/writing/moving is done in an abstract way (see the new `FilePointer` class) so that it will be possible to "move" files on a remote server
- the introduction of a `Process` class to make python operations on files etc. more standardized (and able to be executed remotely)

These changes to treat calculations more like pure functions, and the introduction of a `Process` class also starts us in the direction of following [CWL](https://www.commonwl.org/)'s design pattern more closely. In the long term, we would like to be as close as possible to CWL (perhaps even with composite workflows being able to be written in CWL).

Other changes
---------------------
- the machine-learning workflows have been simplified
- the output of `koopmans` markdown-compliant, making the output files easier to read for humans
- `.kwf` files have been replaced by `.pkl` files (removing the responsibility of writing `Workflows` to file from custom code and using the widely-used `dill` package instead)

.github/workflows/tests.yml

@@ -21,10 +21,10 @@ jobs:
            uses: actions/checkout@v3
            with:
               submodules: true
-         - name: Set up Python 3.8
+         - name: Set up Python
            uses: actions/setup-python@v4
            with:
-              python-version: 3.8
+              python-version: '3.10'
          - name: Install dependencies
            run: |
               python -m pip install --upgrade pip

.github/workflows/tutorials.yml

@@ -21,10 +21,10 @@ jobs:
            uses: actions/checkout@v3
            with:
               submodules: true
-         - name: Set up Python 3.8
+         - name: Set up Python
            uses: actions/setup-python@v4
            with:
-              python-version: 3.8
+              python-version: '3.10'
          - name: Install dependencies
            run: |
               python -m pip install --upgrade pip

.github/workflows/typechecking.yml

@@ -19,10 +19,10 @@ jobs:
            uses: actions/checkout@v3
            with:
               submodules: false
-         - name: Set up Python 3.8
+         - name: Set up Python
            uses: actions/setup-python@v4
            with:
-              python-version: 3.8
+              python-version: '3.10'
          - name: Install dependencies
            run: |
               python -m pip install --upgrade pip

.github/workflows/update_citation.yml

@@ -17,10 +17,10 @@ jobs:
          - name: Get branch name
            id: branch-name
            uses: tj-actions/branch-names@v8
-         - name: Set up Python 3.8
+         - name: Set up Python
            uses: actions/setup-python@v4
            with:
-              python-version: 3.8
+              python-version: '3.10'
          - name: Install dependencies
            run: |
               python -m pip install --upgrade pip
@@ -30,7 +30,9 @@ jobs:
            run: |
               python bin/update_cff.py
          - name: Validate CITATION.cff
-           uses: dieghernan/cff-validator@main
+           uses: dieghernan/cff-validator@v3
+           with:
+             install-r: true
          - name: Commit changes
            uses: test-room-7/action-update-file@v1
            with:

.gitignore

@@ -17,15 +17,6 @@ coverage.xml
 .mypy_cache/
 TMP-CP/
 TMP/
-calc_alpha/
-calculate_eps/
-dft_bands/
-final/
-init/
-pdos/
-postproc/
-wannier/
-screening/
-hamiltonian/
+??-*/
 build/
 dist/

.pre-commit-config.yaml

@@ -2,7 +2,7 @@
 # See https://pre-commit.com/hooks.html for more hooks
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v3.2.0
+    rev: v4.6.0
     hooks:
     -   id: trailing-whitespace
         types: [file, python]
@@ -16,14 +16,14 @@ repos:
       - id: isort
 
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v0.961
+    rev: v1.10.0
     hooks:
       - id: mypy
         files: src/koopmans/
         args: [--ignore-missing-imports]
 
   - repo: https://github.com/pre-commit/mirrors-autopep8
-    rev: v1.6.0
+    rev: v2.0.4
     hooks:
     - id: autopep8
       args: [--max-line-length=120, -i]

CITATION.cff

@@ -71,4 +71,4 @@ preferred-citation:
 repository-code: https://github.com/epfl-theos/koopmans/
 title: koopmans
 type: software
-version: 1.0.1
+version: 1.1.0

bin/update_cff.py

@@ -1,5 +1,7 @@
 import sys
+
 import yaml
+
 if sys.version_info >= (3, 8):
     from importlib import metadata
 else:

docs/_static/tutorials/fetch.py

@@ -0,0 +1,71 @@
+from shutil import copy
+
+
+def extract(filename_in, filename_out, start=0, end=None, heading=None):
+    with open(filename_in) as fd:
+        flines = fd.readlines()
+
+    # If heading is provided, find the line number of the heading
+    if heading:
+        for i, line in enumerate(flines):
+            if heading in line:
+                start = i
+                break
+        else:
+            raise ValueError(f'Heading {heading} not found in file {filename_in}')
+
+        indent = len(flines[start]) - len(flines[start].lstrip())
+
+        for i in range(start + 1, len(flines)):
+            line = flines[i]
+            if not line.startswith(' ' * (indent + 1)):
+                end = i
+                break
+        else:
+            raise ValueError(f'Could not find the end of the {heading} block')
+
+    flines = flines[start:end]
+
+    # Find the shortest leading whitespace and strip this from all lines (as otherwise the markdown will be rendered as a code block)
+    min_indent = min(len(line) - len(line.lstrip()) for line in flines if line.strip())
+    flines = [line if line == "\n" else line[min_indent:]for line in flines]
+
+    # Manual conversion of double spaces to /
+    flines = [line[:-2] + '\ \n' if (line.endswith('  \n')
+                                     and not line.strip().startswith('-')) else line for line in flines]
+
+    with open(filename_out, 'w') as fd:
+        fd.writelines(flines)
+
+
+if __name__ == '__main__':
+    # Tutorial 1
+    extract('../../../tutorials/tutorial_1/ozone.md', 'tutorial_1/md_excerpts/ozone_init.md', heading='Initialization')
+    extract('../../../tutorials/tutorial_1/ozone.md', 'tutorial_1/md_excerpts/ozone_alpha.md', 24, 33)
+    extract('../../../tutorials/tutorial_1/ozone.md', 'tutorial_1/md_excerpts/ozone_alpha_10.md', 45, 49)
+    extract('../../../tutorials/tutorial_1/ozone.md', 'tutorial_1/md_excerpts/ozone_tables.md', 50, 62)
+    extract('../../../tutorials/tutorial_1/ozone.md', 'tutorial_1/md_excerpts/ozone_final.md', -4)
+
+    # Tutorial 2
+    extract('../../../tutorials/tutorial_2/si_wannierize.md', 'tutorial_2/md_excerpts/si_wannierize.md', start=18)
+    extract('../../../tutorials/tutorial_2/si_ki.md', 'tutorial_2/md_excerpts/si_ki_wannierize.md', heading="Wannierize")
+    extract('../../../tutorials/tutorial_2/si_ki.md', 'tutorial_2/md_excerpts/si_ki_fold.md', heading="Fold To Supercell")
+    extract('../../../tutorials/tutorial_2/si_ki.md', 'tutorial_2/md_excerpts/si_ki_screening.md', 51, 61)
+    extract('../../../tutorials/tutorial_2/si_ki.md',
+            'tutorial_2/md_excerpts/si_ki_postproc.md', heading="Unfold And Interpolate")
+
+    # Tutorial 3
+    extract('../../../tutorials/tutorial_3/01-ki/zno.md',
+            'tutorial_3/md_excerpts/zno_wannierize_section.md', heading='Wannierize')
+    extract('../../../tutorials/tutorial_3/01-ki/zno.md', 'tutorial_3/md_excerpts/zno_w2kc.md', -4, -3)
+    extract('../../../tutorials/tutorial_3/01-ki/zno.md', 'tutorial_3/md_excerpts/zno_ham.md', -3, -2)
+    copy('../../../tutorials/tutorial_3/01-ki/01-koopmans-dfpt/Koopmans_DFPT_bandstructure.png', 'tutorial_3/')
+
+    # Tutorial 4
+    extract('../../../tutorials/tutorial_4/h2o_conv.md', 'tutorial_4/md_excerpts/h2o_conv.md', 18, 36)
+
+    # Tutorial 5
+    extract('../../../tutorials/tutorial_5/01-train/h2o_train.md', 'tutorial_5/md_excerpts/train.md', 45, 92)
+    extract('../../../tutorials/tutorial_5/02-predict/h2o_predict.md',
+            'tutorial_5/md_excerpts/predict.md', heading='Calculate Screening Via DSCF')
+    extract('../../../tutorials/tutorial_5/03-test/h2o_test.md', 'tutorial_5/md_excerpts/test.md', 45, 96)

docs/_static/tutorials/tutorial_1/md_excerpts/ozone_alpha.md

@@ -0,0 +1,9 @@
+- **Calculate Screening Via DSCF**
+  - **Iteration 1**
+    - ✅ `01-ki` completed  
+    - **Orbital 1**
+      - ✅ `01-dft_n-1` completed  
+    - **Orbital 2**
+      - ✅ `01-dft_n-1` completed  
+    - **Orbital 3**
+      - ✅ `01-dft_n-1` completed  

docs/_static/tutorials/tutorial_1/md_excerpts/ozone_alpha_10.md

@@ -0,0 +1,4 @@
+- **Orbital 10**
+  - ✅ `01-dft_n+1_dummy` completed  
+  - ✅ `02-pz_print` completed  
+  - ✅ `03-dft_n+1` completed  

docs/_static/tutorials/tutorial_1/md_excerpts/ozone_final.md

@@ -0,0 +1,4 @@
+
+  - ✅ `03-ki_final` completed  
+
+**Workflow complete** 🎉

docs/_static/tutorials/tutorial_1/md_excerpts/ozone_init.md

@@ -0,0 +1,5 @@
+- **Initialization**
+  - ✅ `01-dft_init_nspin1` completed  
+  - ✅ `02-dft_init_nspin2_dummy` completed  
+  - ✅ `03-convert_files_from_spin1to2` completed  
+  - ✅ `04-dft_init_nspin2` completed  

docs/_static/tutorials/tutorial_1/md_excerpts/ozone_tables.md

@@ -0,0 +1,12 @@
+**α**
+|    |        1 |        2 |       3 |        4 |        5 |        6 |        7 |        8 |        9 |       10 |
+|---:|---------:|---------:|--------:|---------:|---------:|---------:|---------:|---------:|---------:|---------:|
+|  0 | 0.6      | 0.6      | 0.6     | 0.6      | 0.6      | 0.6      | 0.6      | 0.6      | 0.6      | 0.6      |
+|  1 | 0.655691 | 0.727571 | 0.78386 | 0.663859 | 0.772354 | 0.726848 | 0.729968 | 0.741899 | 0.779264 | 0.717389 |
+
+**ΔE<sub>i</sub> - λ<sub>ii</sub> (eV)**
+|    |        1 |         2 |        3 |         4 |        5 |         6 |         7 |         8 |        9 |       10 |
+|---:|---------:|----------:|---------:|----------:|---------:|----------:|----------:|----------:|---------:|---------:|
+|  0 | -0.47736 | -0.920499 | -1.12409 | -0.452149 | -1.05055 | -0.830893 | -0.785139 | -0.888597 | -1.05016 | 0.711209 |
+
+

docs/_static/tutorials/tutorial_2/md_excerpts/si_ki_fold.md

@@ -0,0 +1,3 @@
+- **Fold To Supercell**
+  - ✅ `01-convert_block_1_to_supercell` completed  
+  - ✅ `02-convert_block_2_to_supercell` completed  

docs/_static/tutorials/tutorial_2/md_excerpts/si_ki_postproc.md

@@ -0,0 +1,16 @@
+- **Unfold And Interpolate**
+  - **Wannierize**
+    - ✅ `01-scf` completed  
+    - ✅ `02-nscf` completed  
+    - **Wannierize Block 1**
+      - ✅ `01-wannier90_preproc` completed  
+      - ✅ `02-pw2wannier90` completed  
+      - ✅ `03-wannier90` completed  
+    - **Wannierize Block 2**
+      - ✅ `01-wannier90_preproc` completed  
+      - ✅ `02-pw2wannier90` completed  
+      - ✅ `03-wannier90` completed  
+    - ✅ `05-bands` completed  
+    - ✅ `06-projwfc` completed  
+  - ✅ `02-unfold_and_interpolate_occ` completed  
+  - ✅ `03-unfold_and_interpolate_emp` completed  

docs/_static/tutorials/tutorial_2/md_excerpts/si_ki_screening.md

@@ -0,0 +1,10 @@
+- **Calculate Screening Via DSCF**
+  - **Iteration 1**
+    - ✅ `01-ki` completed  
+    - **Orbital 32**
+      - ✅ `01-dft_n-1` completed  
+    - **Orbital 33**
+      - ✅ `01-dft_n+1_dummy` completed  
+      - ✅ `02-pz_print` completed  
+      - ✅ `03-dft_n+1` completed  
+

docs/_static/tutorials/tutorial_2/md_excerpts/si_ki_wannierize.md

@@ -0,0 +1,11 @@
+- **Wannierize**
+  - ✅ `01-scf` completed  
+  - ✅ `02-nscf` completed  
+  - **Wannierize Block 1**
+    - ✅ `01-wannier90_preproc` completed  
+    - ✅ `02-pw2wannier90` completed  
+    - ✅ `03-wannier90` completed  
+  - **Wannierize Block 2**
+    - ✅ `01-wannier90_preproc` completed  
+    - ✅ `02-pw2wannier90` completed  
+    - ✅ `03-wannier90` completed  

docs/_static/tutorials/tutorial_2/md_excerpts/si_wannierize.md

@@ -0,0 +1,14 @@
+- ✅ `01-scf` completed  
+- ✅ `02-nscf` completed  
+- **Wannierize Block 1**
+  - ✅ `01-wannier90_preproc` completed  
+  - ✅ `02-pw2wannier90` completed  
+  - ✅ `03-wannier90` completed  
+- **Wannierize Block 2**
+  - ✅ `01-wannier90_preproc` completed  
+  - ✅ `02-pw2wannier90` completed  
+  - ✅ `03-wannier90` completed  
+- ✅ `05-bands` completed  
+- ✅ `06-projwfc` completed  
+
+**Workflow complete** 🎉

docs/_static/tutorials/tutorial_3/md_excerpts/zno_ham.md

@@ -0,0 +1 @@
+- ✅ `03-kc_ham` completed  

docs/_static/tutorials/tutorial_3/md_excerpts/zno_w2kc.md

@@ -0,0 +1 @@
+- ✅ `02-wann2kc` completed  

docs/_static/tutorials/tutorial_3/md_excerpts/zno_wannierize_section.md

@@ -0,0 +1,28 @@
+- **Wannierize**
+  - ✅ `01-scf` completed  
+  - ✅ `02-nscf` completed  
+  - **Wannierize Block 1**
+    - ✅ `01-wannier90_preproc` completed  
+    - ✅ `02-pw2wannier90` completed  
+    - ✅ `03-wannier90` completed  
+  - **Wannierize Block 2**
+    - ✅ `01-wannier90_preproc` completed  
+    - ✅ `02-pw2wannier90` completed  
+    - ✅ `03-wannier90` completed  
+  - **Wannierize Block 3**
+    - ✅ `01-wannier90_preproc` completed  
+    - ✅ `02-pw2wannier90` completed  
+    - ✅ `03-wannier90` completed  
+  - **Wannierize Block 4**
+    - ✅ `01-wannier90_preproc` completed  
+    - ✅ `02-pw2wannier90` completed  
+    - ✅ `03-wannier90` completed  
+  - **Wannierize Block 5**
+    - ✅ `01-wannier90_preproc` completed  
+    - ✅ `02-pw2wannier90` completed  
+    - ✅ `03-wannier90` completed  
+  - ✅ `08-merge_occ_wannier_hamiltonian` completed  
+  - ✅ `09-merge_occ_wannier_u` completed  
+  - ✅ `10-merge_occ_wannier_centers` completed  
+  - ✅ `11-bands` completed  
+  - ✅ `12-projwfc` completed  

docs/_static/tutorials/tutorial_4/md_excerpts/h2o_conv.md

@@ -0,0 +1,18 @@
+- **Singlepoint ecutwfc 20_0 celldm1 11_3**
+  - **DFTCP**
+    - ✅ `01-dft` completed  
+- **Singlepoint ecutwfc 20_0 celldm1 12_3**
+  - **DFTCP**
+    - ✅ `01-dft` completed  
+- **Singlepoint ecutwfc 20_0 celldm1 13_3**
+  - **DFTCP**
+    - ✅ `01-dft` completed  
+- **Singlepoint ecutwfc 30_0 celldm1 11_3**
+  - **DFTCP**
+    - ✅ `01-dft` completed  
+- **Singlepoint ecutwfc 30_0 celldm1 12_3**
+  - **DFTCP**
+    - ✅ `01-dft` completed  
+- **Singlepoint ecutwfc 30_0 celldm1 13_3**
+  - **DFTCP**
+    - ✅ `01-dft` completed  

docs/_static/tutorials/tutorial_5/md_excerpts/predict.md

@@ -0,0 +1,19 @@
+- **Calculate Screening Via DSCF**
+  - **Iteration 1**
+    - ✅ `01-ki` completed  
+    - **Power Spectrum Decomposition**
+      - **Convert Orbital Files To XML**
+        - ✅ `01-bin2xml_total_density` completed  
+        - ✅ `02-bin2xml_occ_spin_0_orb_1_density` completed  
+        - ✅ `03-bin2xml_occ_spin_0_orb_2_density` completed  
+        - ✅ `04-bin2xml_occ_spin_0_orb_3_density` completed  
+        - ✅ `05-bin2xml_occ_spin_0_orb_4_density` completed  
+        - ✅ `06-bin2xml_emp_spin_0_orb_5_density` completed  
+        - ✅ `07-bin2xml_emp_spin_0_orb_6_density` completed  
+      - ✅ `02-extract_coefficients_from_xml` completed  
+      - ✅ `03-compute_power_spectrum_orbital_1` completed  
+      - ✅ `04-compute_power_spectrum_orbital_2` completed  
+      - ✅ `05-compute_power_spectrum_orbital_3` completed  
+      - ✅ `06-compute_power_spectrum_orbital_4` completed  
+      - ✅ `07-compute_power_spectrum_orbital_5` completed  
+      - ✅ `08-compute_power_spectrum_orbital_6` completed