Migrate to mpi4py.futures, simplify bash script

Migrate the Python script from mpi4py.MPI to mpi4py.futures. Note that mpi4py.futures requires a separate launcher script to create the MPI environment and launch the main script.

When using mpi4py.futures, the MPIPoolExecutor needs to be started in a separate process. This is because the MPIPoolExecutor must be created and submitted to tasks within an if __name__ == '__main__': block to prevent recursive creation of MPIPoolExecutor instances.

Also, in the bash commands:
- Modify the bash code to remove or clear the python-test directory, if it already exists. This ensures a clean environment for every test run.
- Simplify the scp file transfer code, to one line.

scripts/DelftBlue commands.txt

@@ -1,16 +1,32 @@
-## Login
+# Login (CMD)
 ssh -J [email protected] [email protected]
 
-## Make new directory
-mkdir python-test
+# Move to scratch folder
+cd ../../scratch/eterhoeven
+
+# Check if directory exists
+if [ -d "python-test" ]; then
+  # Remove all files and subdirectories inside the directory
+  rm -r python-test/*
+else
+  # Create new directory
+  mkdir python-test
+fi
+
+# Change to the python-test directory
 cd python-test
 
-## Move files
-cd C:\Users\Ewout\OneDrive\Studie\CapitaSelecta\
-scp -J [email protected] my_model.py [email protected]:/scratch/eterhoeven/python-test
-scp -J [email protected] test_script.sh [email protected]:/scratch/eterhoeven/python-test
+# Move files (NEW CMD)
+cd C:\Users\Ewout\Documents\GitHub\EMAworkbench\scripts
+scp -J [email protected] my_model.py launcher.py test_script.sh [email protected]:/scratch/eterhoeven/python-test
+
+# Check if files are correctly tranfered
+ls
+
+# Note: bash scripts (sh) needs LF line endings. Do that in atom.
 
-## Note: bash scripts (sh) needs LF line endings. Do that in atom.
+# Run batch script
+sbatch test_script.sh
 
-## Run batch script
-sbatch test_script.sh
+# Check queue
+squeue -u eterhoeven

scripts/launcher.py

@@ -0,0 +1,7 @@
+# launcher.py
+from mpi4py.futures import MPIPoolExecutor
+import my_model
+
+if __name__ == '__main__':
+    with MPIPoolExecutor() as executor:
+        executor.submit(my_model)

scripts/my_model.py

@@ -2,34 +2,23 @@
 from mpi4py import MPI
 import numpy as np
 import pickle
+from mpi4py.futures import MPIPoolExecutor
 
-def my_model(x, y):
+def my_model(data):
     # This could be any function that takes inputs and produces outputs
+    x, y = data
     return x**2 + y**2
 
-comm = MPI.COMM_WORLD
-rank = comm.Get_rank()
-size = comm.Get_size()
+if __name__ == '__main__':
+    # Define the input data
+    data = np.arange(20).reshape(-1, 2)  # Let's say we have 10 sets of (x, y) pairs
 
-# Define the input data
-data = np.arange(20).reshape(-1, 2)  # Let's say we have 10 sets of (x, y) pairs
+    with MPIPoolExecutor() as executor:
+        # Apply the model to the data
+        results = list(executor.map(my_model, data))
 
-# Divide the data among the available processes
-chunks = np.array_split(data, size)
-
-# Each process gets its chunk of the data
-chunk = chunks[rank]
-
-# Each process applies the model to its chunk of the data
-results = np.array([my_model(x, y) for x, y in chunk])
-
-# Gather the results back to the root process
-all_results = comm.gather(results, root=0)
-
-if rank == 0:
-    # Root process prints out the results
-    print(np.concatenate(all_results))
+    # Print out the results
+    print(results)
 
     with open('py_test.pickle', 'wb') as handle:
         pickle.dump(results, handle, protocol=pickle.HIGHEST_PROTOCOL)
-

scripts/test_script.sh

@@ -14,4 +14,4 @@ module load python
 module load py-numpy
 module load py-mpi4py
 
-srun python my_model.py > py_test.log
+srun python launcher.py > py_test.log