Skip to content

Commit

Permalink
Merge pull request #396 from nicholasjnelson/ModelSetup
Browse files Browse the repository at this point in the history
Reviewed Model Setup
  • Loading branch information
feathern authored Sep 15, 2022
2 parents d46eadb + cdf6964 commit 0b27224
Show file tree
Hide file tree
Showing 2 changed files with 106 additions and 105 deletions.
105 changes: 1 addition & 104 deletions doc/source/User_Guide/model_setup.rst
Original file line number Diff line number Diff line change
Expand Up @@ -607,6 +607,7 @@ Classification of the Convective Regimes in Rotating Stars.” The Astrophysical


::
&problemsize_namelist
n_r = 64
n_theta = 192
Expand Down Expand Up @@ -706,107 +707,3 @@ Classification of the Convective Regimes in Rotating Stars.” The Astrophysical
kappa_top = 4.d12
/

.. _surveys:

Ensemble Mode
-------------

Rayleigh can also be used to run multiple simulations under the umbrella
of a single executable. This functionality is particularly useful for
running parameter space studies, which often consist of mulitple,
similarly-sized simulations, in one shot. Moreover, as some queuing
systems favor large jobs over small jobs, an ensemble mode is useful for
advancing multiple small simulations through the queue in a reasonable
timeframe.

Running Rayleigh in ensemble mode is relatively straightforward. To
begin with, create a directory for each simulation as you normally
would, and place an appropriately modified main_input into each
directory. These directories should all reside within the same parent
directory. Within that parent directory, you should place a copy of the
Rayleigh executable (or a softlink). In addition, you should create a
text file named **run_list** that contains the name of each simulation
directory, one name per line. An ensemble job may then be executed by
calling Rayleigh with **nruns** command line flag as:

::

user@machinename ~/runs/ $ mpiexec -np Y ./rayleigh.opt -nruns X

Here, Y is the total number of cores needed by all X simulations listed
in run_list.

**Example:** Suppose you wish to run three simulations at once from
within a parent directory named *ensemble* and that the simulation
directories are named run1, run2, and run3. When performing an *ls* from
within *ensemble*, you should see 5 items.

::

user@machinename ~/runs/ $ cd ensemble
user@machinename ~/runs/ensemble $ ls
rayleigh.opt run1 run2 run3 run_list

In this example, the contents of run_list should be the *local* names of
your ensemble run-directories, namely run1, run2, and run3.

::

user@machinename ~runs/ensemble $ more run_list
run1
run2
run3
<-- place an empty line here

Note that some Fortran implementations will not read the last line in
run_list unless it ends in a newline character. Avoid unexpected crashes
by hitting "enter" following your final entry in run_list.

Before running Rayleigh, make sure you know how many cores each
simulation needs by examining the main_input files:

::

user@machinename ~runs/ensemble $ head run1/main_input
&problemsize_namelist
n_r = 128
n_theta = 192
nprow = 16
npcol = 16
/

user@machinename ~runs/ensemble $ head run2/main_input
&problemsize_namelist
n_r = 128
n_theta = 384
nprow = 32
npcol = 16
/

user@machinename ~runs/ensemble $ head run3/main_input
&problemsize_namelist
n_r = 64
n_theta = 192
nprow = 16
npcol = 16
/

In this example, we need a total of 1024 cores (256+512+256) to execute
three simulations, and so the relevant call to Rayleigh would be:

::

user@machinename ~/runs/ $ mpiexec -np 1024 ./rayleigh.opt -nruns 3

**Closing Notes:** When running in ensemble mode, it is *strongly
recommended* that you redirect standard output for each simulation to a
text file (see §\ :ref:`io`). Otherwise, all simulations
write to the same default (machine-dependent) log file, making it
difficult to read. Moreover, some machines such as NASA Pleiades will
terminate a run if the log file becomes too long. This is easy to do
when multiple simulations are writing to the same file.

Finally, The flags -nprow and -npcol **are ignored** when -nruns is
specified. The row and column configuration for all simulations needs to
be specified in their respective main_input files instead.

106 changes: 105 additions & 1 deletion doc/source/User_Guide/run_at_scale.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,108 @@ Need text here.
Sample Jobscripts
-----------------

Need text here.
Need text here.

.. _surveys:

Ensemble Mode
-------------

Rayleigh can also be used to run multiple simulations under the umbrella
of a single executable. This functionality is particularly useful for
running parameter space studies, which often consist of mulitple,
similarly-sized simulations, in one shot. Moreover, as some queuing
systems favor large jobs over small jobs, an ensemble mode is useful for
advancing multiple small simulations through the queue in a reasonable
timeframe.

Running Rayleigh in ensemble mode is relatively straightforward. To
begin with, create a directory for each simulation as you normally
would, and place an appropriately modified main_input into each
directory. These directories should all reside within the same parent
directory. Within that parent directory, you should place a copy of the
Rayleigh executable (or a softlink). In addition, you should create a
text file named **run_list** that contains the name of each simulation
directory, one name per line. An ensemble job may then be executed by
calling Rayleigh with **nruns** command line flag as:

::

user@machinename ~/runs/ $ mpiexec -np Y ./rayleigh.opt -nruns X

Here, Y is the total number of cores needed by all X simulations listed
in run_list.

**Example:** Suppose you wish to run three simulations at once from
within a parent directory named *ensemble* and that the simulation
directories are named run1, run2, and run3. When performing an *ls* from
within *ensemble*, you should see 5 items.

::

user@machinename ~/runs/ $ cd ensemble
user@machinename ~/runs/ensemble $ ls
rayleigh.opt run1 run2 run3 run_list

In this example, the contents of run_list should be the *local* names of
your ensemble run-directories, namely run1, run2, and run3.

::

user@machinename ~runs/ensemble $ more run_list
run1
run2
run3
<-- place an empty line here

Note that some Fortran implementations will not read the last line in
run_list unless it ends in a newline character. Avoid unexpected crashes
by hitting "enter" following your final entry in run_list.

Before running Rayleigh, make sure you know how many cores each
simulation needs by examining the main_input files:

::

user@machinename ~runs/ensemble $ head run1/main_input
&problemsize_namelist
n_r = 128
n_theta = 192
nprow = 16
npcol = 16
/

user@machinename ~runs/ensemble $ head run2/main_input
&problemsize_namelist
n_r = 128
n_theta = 384
nprow = 32
npcol = 16
/

user@machinename ~runs/ensemble $ head run3/main_input
&problemsize_namelist
n_r = 64
n_theta = 192
nprow = 16
npcol = 16
/

In this example, we need a total of 1024 cores (256+512+256) to execute
three simulations, and so the relevant call to Rayleigh would be:

::

user@machinename ~/runs/ $ mpiexec -np 1024 ./rayleigh.opt -nruns 3

**Closing Notes:** When running in ensemble mode, it is *strongly
recommended* that you redirect standard output for each simulation to a
text file (see §\ :ref:`io`). Otherwise, all simulations
write to the same default (machine-dependent) log file, making it
difficult to read. Moreover, some machines such as NASA Pleiades will
terminate a run if the log file becomes too long. This is easy to do
when multiple simulations are writing to the same file.

Finally, The flags -nprow and -npcol **are ignored** when -nruns is
specified. The row and column configuration for all simulations needs to
be specified in their respective main_input files instead.

0 comments on commit 0b27224

Please sign in to comment.