From 5fcf1ae9a7e5615731c4d29da877e24e41739588 Mon Sep 17 00:00:00 2001 From: "Keaton J. Burns" Date: Mon, 17 Aug 2020 10:59:57 -0400 Subject: [PATCH] Update readme and start expanding documentation --- README.md | 45 ++++++++++++-------- docs/conf.py | 2 +- docs/getting_started.rst | 42 ------------------- docs/index.rst | 32 +++++++++------ docs/pages/configuration.rst | 20 +++++++++ docs/pages/general_functions.rst | 4 ++ docs/{ => pages}/installation.rst | 20 ++++----- docs/pages/methodology.rst | 14 +++++++ docs/pages/output_format.rst | 4 ++ docs/pages/parallel_data.rst | 4 ++ docs/pages/performance_tips.rst | 68 +++++++++++++++++++++++++++++++ docs/pages/restarting.rst | 4 ++ docs/pages/troubleshooting.rst | 50 +++++++++++++++++++++++ docs/pages/tutorials.rst | 37 +++++++++++++++++ docs/pages/user_guide.rst | 16 ++++++++ 15 files changed, 278 insertions(+), 84 deletions(-) delete mode 100644 docs/getting_started.rst create mode 100644 docs/pages/configuration.rst create mode 100644 docs/pages/general_functions.rst rename docs/{ => pages}/installation.rst (94%) create mode 100644 docs/pages/methodology.rst create mode 100644 docs/pages/output_format.rst create mode 100644 docs/pages/parallel_data.rst create mode 100644 docs/pages/performance_tips.rst create mode 100644 docs/pages/restarting.rst create mode 100644 docs/pages/troubleshooting.rst create mode 100644 docs/pages/tutorials.rst create mode 100644 docs/pages/user_guide.rst diff --git a/README.md b/README.md index 229d60fb..bca7a971 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,39 @@ -# Dedalus Project - -## About Dedalus + +

+ Dedalus Project +

+ + +

+ + Repo status + + + Read the Docs + + PyPI - Python Version + PyPI + PyPI - License +

Dedalus is a flexible framework for solving partial differential equations using spectral methods. -The code is open-source and developed by a team of researchers working on problems in astrophysical and geophysical fluid dynamics. +The code is open-source and developed by a team of researchers studying astrophysical and geophysical fluid dynamics. -The code is written primarily in Python and features an easy-to-use interface, including text-based equation entry. -Our numerical algorithm produces highly sparse systems for a wide variety of equations on spectrally-discretized domains. -These systems are efficiently solved using compiled libraries and multidimensional parallelization though MPI. +Dedalus is written primarily in Python and features an easy-to-use interface with symbolic equation entry. +Our numerical algorithm produces sparse systems for a wide variety of equations and spectrally-discretized domains. +These systems are efficiently solved using compiled libraries and are automatically parallelized using MPI. ## Links -Learn more about Dedalus by visiting the - * Project homepage: * Code repository: * Documentation: -* User mailing list: -* Development mailing list: +* Mailing list: ## Developers -The core development team consists of - -* Keaton Burns () -* Ben Brown () -* Daniel Lecoanet () -* Jeff Oishi () -* Geoff Vasil () +* [Keaton Burns (@kburns)](https://github.com/kburns) +* [Geoff Vasil (@gmvasil)](https://github.com/gmvasil) +* [Jeff Oishi (@jsoishi)](https://github.com/jsoishi) +* [Daniel Lecoanet (@lecoanet)](https://github.com/lecoanet/) +* [Ben Brown (@bpbrown)](https://github.com/bpbrown) diff --git a/docs/conf.py b/docs/conf.py index db5e57f9..014bf930 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -20,7 +20,7 @@ # -- Project information ----------------------------------------------------- project = 'Dedalus Project' -copyright = '2018 Dedalus Collaboration' +copyright = '2020 Dedalus Collaboration' author = 'Dedalus Collaboration' # The short X.Y version diff --git a/docs/getting_started.rst b/docs/getting_started.rst deleted file mode 100644 index bdc1cdba..00000000 --- a/docs/getting_started.rst +++ /dev/null @@ -1,42 +0,0 @@ -Getting started with Dedalus -**************************** - -Tutorial Notebooks -================== - -This tutorial on using Dedalus consists of three short IPython notebooks, which can be downloaded and ran interactively, or viewed on-line through the links below. - -The notebooks cover the basics of setting up and interacting with the primary facets of the code, culminating in the setup and simulation of the 1D KdV-Burgers equation. - -.. toctree:: - :maxdepth: 1 - - Tutorial 1: Bases and Domains - Tutorial 2: Fields and Operators - Tutorial 3: Problems and Solvers - Tutorial 4: Analysis and Post-processing - -Example Notebooks -================= - -Below are several notebooks that walk through the setup and execution of more complicated multidimensional example problems. - -.. toctree:: - :maxdepth: 1 - - Example 1: Kelvin-Helmholtz Instability - Example 2: Taylor-Couette Flow - -Example Scripts -=============== - -A wider range of examples are available under the ``examples`` subdirectory of the main code repository, which you can browse `here `_. -These examples cover a wider range of use cases, including larger multidimensional problems designed for parallel execution. -Basic post-processing and plotting scripts are also provided with many problems. - -These simulation and processing scripts may be useful as a starting point for implementing different problems and equation sets. - -Contributions & Suggestions -=========================== - -If you have a script that you'd like to make available as an example, or to request an additional example covering different functionality or use cases, please get in touch on the `dev list `_! diff --git a/docs/index.rst b/docs/index.rst index e9e2682c..32c386bf 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,15 +1,12 @@ Dedalus Project *************** -About Dedalus -============= - Dedalus is a flexible framework for solving partial differential equations using spectral methods. -The code is open-source and developed by a team of researchers working on problems in astrophysical and geophysical fluid dynamics. +The code is open-source and developed by a team of researchers studying astrophysical, geophysical, and biological fluid dynamics. -The code is written primarily in Python and features an easy-to-use interface, including text-based equation entry. -Our numerical algorithm produces highly sparse systems for a wide variety of equations on spectrally-discretized domains. -These systems are efficiently solved using compiled libraries and multidimensional parallelization though MPI. +Dedalus is written primarily in Python and features an easy-to-use interface with symbolic equation entry. +Our numerical algorithm produces sparse systems for a wide variety of equations and spectrally-discretized domains. +These systems are efficiently solved using compiled libraries and are automatically parallelized using MPI. Doc Contents ============ @@ -17,16 +14,25 @@ Doc Contents .. toctree:: :maxdepth: 2 - installation - getting_started + pages/installation + pages/tutorials + pages/user_guide + pages/methodology Dedalus API reference -Other Links -=========== +Links +===== * Project homepage: http://dedalus-project.org * Code repository: https://github.com/DedalusProject/dedalus * Documentation: http://dedalus-project.readthedocs.org -* User mailing list: https://groups.google.com/forum/#!forum/dedalus-users -* Development mailing list: https://groups.google.com/forum/#!forum/dedalus-dev +* Mailing list: https://groups.google.com/forum/#!forum/dedalus-users + +Developers +========== +* `Keaton Burns `_ +* `Geoff Vasil `_ +* `Jeff Oishi `_ +* `Daniel Lecoanet `_ +* `Ben Brown `_ diff --git a/docs/pages/configuration.rst b/docs/pages/configuration.rst new file mode 100644 index 00000000..feec062f --- /dev/null +++ b/docs/pages/configuration.rst @@ -0,0 +1,20 @@ +.. _configuration: + +Configuring Dedalus +******************* + +Various aspects of the underlying numerics, logging behavior, and output behavior of Dedalus can be modified through a configuration interface using Python's standard-library `ConfigParser` structure. + +A ``dedalus.cfg`` file with the default configuration and descriptions of each option is included in the package. +This file can be copied to any working directory with the command ``python3 -m dedalus get_config``. +The configuration settings can be modified by, in order of increasing precedence: + +1. Modifying the package's ``dedalus.cfg`` file (not recommended). +2. Creating a user-based config file at ``~/.dedalus/dedalus.cfg``. +3. Creating a local config file called ``dedalus.cfg`` in the directory where you execute a Dedalus script. +4. Modifying configuration values at the top of a Dedalus script like: + +.. code-block:: python + + from dedalus.tools.config import config + config['logging']['stdout_level'] = 'debug' diff --git a/docs/pages/general_functions.rst b/docs/pages/general_functions.rst new file mode 100644 index 00000000..4f10c1a4 --- /dev/null +++ b/docs/pages/general_functions.rst @@ -0,0 +1,4 @@ +General Functions +***************** + +Documentation coming soon. diff --git a/docs/installation.rst b/docs/pages/installation.rst similarity index 94% rename from docs/installation.rst rename to docs/pages/installation.rst index ebe72d01..cbb044b5 100644 --- a/docs/installation.rst +++ b/docs/pages/installation.rst @@ -121,16 +121,16 @@ Below are instructions for building the dependency stack on a variety of machine .. toctree:: :maxdepth: 1 - machines/bridges/bridges - machines/cedar/cedar - machines/engaging/engaging - machines/janus/janus - machines/mac_os/mac_os - machines/nasa_discover/discover - machines/nasa_pleiades/pleiades - machines/savio/savio - machines/stampede/stampede - machines/trestles/trestles + /machines/bridges/bridges + /machines/cedar/cedar + /machines/engaging/engaging + /machines/janus/janus + /machines/mac_os/mac_os + /machines/nasa_discover/discover + /machines/nasa_pleiades/pleiades + /machines/savio/savio + /machines/stampede/stampede + /machines/trestles/trestles Once the dependency stack has been installed, Dedalus can be installed `as described above <#installing-the-dedalus-package>`_. diff --git a/docs/pages/methodology.rst b/docs/pages/methodology.rst new file mode 100644 index 00000000..51211175 --- /dev/null +++ b/docs/pages/methodology.rst @@ -0,0 +1,14 @@ +Methodology +*********** + +Dedalus Methods Paper +===================== + +This is a work-in-progress section on the mathematical and computational methodologies underlying Dedalus. +Until this section is completed, please consult the Dedalus methods paper: + + K J Burns, G M Vasil, J S Oishi, D Lecoanet, B P Brown, "Dedalus: A Flexible Framework for Numerical Simulations with Spectral Methods," Physical Review Research, vol. 2, no. 2, p. 838, Apr. 2020. + `[doi] `_ + `[arxiv] `_ + `[bibtex] `_ + `[examples] `_ diff --git a/docs/pages/output_format.rst b/docs/pages/output_format.rst new file mode 100644 index 00000000..824fc15b --- /dev/null +++ b/docs/pages/output_format.rst @@ -0,0 +1,4 @@ +Output Formatting +***************** + +Documentation coming soon. diff --git a/docs/pages/parallel_data.rst b/docs/pages/parallel_data.rst new file mode 100644 index 00000000..d762bb22 --- /dev/null +++ b/docs/pages/parallel_data.rst @@ -0,0 +1,4 @@ +Parallel Data Interaction +************************* + +Documentation coming soon. diff --git a/docs/pages/performance_tips.rst b/docs/pages/performance_tips.rst new file mode 100644 index 00000000..a6f46973 --- /dev/null +++ b/docs/pages/performance_tips.rst @@ -0,0 +1,68 @@ +.. _performance_tips: + +Performance Tips +**************** + +Domain Specification +==================== + +Resolutions for faster transforms +--------------------------------- + +The transforms for the ``Fourier``, ``SinCos``, and ``Chebyshev`` bases are computed using fast Fourier transforms (FFTs). +The underlying FFT algorithms are most efficient when the transform sizes are products of small primes. +We recommend choosing basis resolutions that are powers of two, or powers of two multiplied by other small factors. + +Process meshes for better load balancing +---------------------------------------- + +Dedalus uses multidimensional block distributions when decomposing domains in parallel. +By default, problems are parallelized over a 1D process mesh of all available MPI processes. +Multidimensional parallelization is easily enabled by specifing a mesh shape using the ``mesh`` keyword when instantiating a ``Domain`` object. +The specified mesh shape should be a tuple with a length one less than the problem dimension. + +Ideal load balancing occurs when the size of a distributed dimension is evenly divisible by the corresponding mesh size. +We recommend choosing mesh sizes that are powers of two, or powers of two multiplied by other small factors. +An "isotropic" mesh with the same number of processes in each mesh dimension, e.g. ``(8, 8)``, will theoretically be the most efficient for a given number of processes. + +Avoid empty cores +----------------- + +Note that it is possible to end up with empty cores in certain layouts if the mesh shape is chosen improperly or if a mesh is not specified for a 3D problem ran over many cores. +For example, consider a problem with global shape ``N = (64, 64, 64)`` distributed on ``P = 256`` cores. +Keeping the default 1D mesh or choosing a mesh of shape ``(128, 2)`` or ``(2, 128)`` will result in many empty cores -- a better choice would be a mesh of shape ``(16, 16)``. + +Problem Formulation +=================== + +Minimize the number of problem variables +---------------------------------------- + +The number of variables used in a problem can have a large impact on the overall simulation performance. +We recommend formulating problems to use as few variables as possible, within the constraints posed by Dedalus itself, which require that PDEs are written as first-order systems in terms of temporal and non-Fourier spatial derivatives. +Often the ``problem.substitutions`` interface can be used to simplify the entry of complex equations without introducing new variables into a problem. + +Formulate boundary conditions as Dirichlet conditions +----------------------------------------------------- + +Maintaining a high degree of bandedness in the LHS matrices can greatly improve the performance of Dedalus. +This requires using Dirichlet rather than integral, Neumann, or other mixed boundary conditions. +Note that the first-order formulation required by Dedalus often means that such boundary conditions can be posed as Dirichlet conditions on the first-order variables, e.g. the Neumann condition ``dz(u) = 0`` for an elliptic or parabolic problem can be written as the Dirichlet condition ``uz = 0``, assuming the first-order reduction definition ``"uz - dz(u) = 0"`` is part of the PDE system. + +Avoid non-smooth or rational-function NCCs +------------------------------------------ + +High bandedness also requires that non-constant coefficients (NCCs) appearing on the LHS are spectrally smooth. +An amplitude cutoff and a limit to the number of terms used when expanding the LHS NCCs can be specified with the ``ncc_cutoff`` and ``max_ncc_terms`` keywords when instantiating a problem. +These settings can have a large performance impact for problems with NCCs that are not low-degree polynomials. +Problems with NCCs that are rational functions (such as ``1/r**2`` terms that appear in problems in curvilinear coordinates) should usually be multiplied through to clear the polynomial denominators, resulting in purely polynomial and hence well-banded NCC operators. + +Timestepping +============ + +Avoid changing the simulation timestep unless necessary +------------------------------------------------------- + +Changing the simulation timestep requires refactorizing the LHS matrices, so the timestep should be changed as infrequently as possible for maximum performance. +If you are using the built-in CFL tools to calculate the simulation timestep, be sure to specify a ``threshold`` parameter greater than zero to prevent the timestep from changing due to small variations in the flow. + diff --git a/docs/pages/restarting.rst b/docs/pages/restarting.rst new file mode 100644 index 00000000..e20fcbfe --- /dev/null +++ b/docs/pages/restarting.rst @@ -0,0 +1,4 @@ +Restarting Simulations +********************** + +Documentation coming soon. diff --git a/docs/pages/troubleshooting.rst b/docs/pages/troubleshooting.rst new file mode 100644 index 00000000..b38a95f1 --- /dev/null +++ b/docs/pages/troubleshooting.rst @@ -0,0 +1,50 @@ +Troubleshooting +*************** + +Singular matrix errors +====================== + +If you come across an error in the linear solver stating that a matrix/factor is singular, that means that the linear LHS portion of the PDE system is not solvable. +This error indicates that some degrees of freedom of the solution are unconstrained and some of the specified equations are redundant (these are equivalent since the LHS matrices must be square). + +These errors are often due to imposing boundary conditions that are redundant for some set of modes and/or failing to constrain a gauge freedom in the solution. +For instance, when simulating incompressibly hydrodynamics in a channel, specifying the mean wall-normal velocity on both walls is redundant with the incompressiblity constraint (the mean wall-normal velocity at one wall must be the same as at the other wall for the flow to be divergence-free). +Instead, the wall-normal velocity boundary condition at one wall should be removed and replaced with a gauge condition setting the constant portion of the pressure. +This is why you'll see these types of boundary conditions for the wall-normal velocity (e.g. ``w``) in Dedalus scripts for incompressible flow: + +.. code-block:: python + + problem.add_bc("left(w) = 0") + problem.add_bc("right(w) = 0", condition="(nx != 0)") + problem.add_bc("right(p) = 0", condition="(nx == 0)") + +Maintaining Hermitian symmetry with real variables +================================================== + +In certain problems with real variables, numerical instabilities may arise due to the loss of Hermitian symmetry in the Fourier components of the solution. +When the grid datatype is set to ``float`` or ``np.float64`` in Dedalus, that means that the first Fourier transform (say in the x direction) will be computed with a real-to-complex FFT where the negative half of the spectrum (kx < 0) is discarded. +Subsequent transforms are then complex-to-complex. + +This strategy maintains the proper Hermitian symmetry for the majority of the modes, since only the non-negative kx half of the spectrum is evolved and the negative kx modes are computed from the conjugates of the positive kx modes whenever they are needed (i.e. for interpolation along x). +However, modes with kx = 0 should have coefficients that are real after the x-transform, and should have Hermitian symmetry in e.g. ky if there's a subsequent Fourier transform in y. +This is not enforced by Dedalus, so if the PDE has a linear instability for modes with kx = 0, then the imaginary part of the solution can grow tihout bound (the nonlinear terms are computed in grid space, so they only affect the real portion of the solution and can’t saturate a growing imaginary part). + +A simple solution is to periodically transform all the state variables to grid space, which will project away any imaginary component that may have been building up during timestepping. +This can be done with a simple statement in the main loop like: + +.. code-block:: python + + if solver.iteration % 100 == 0: + for field in solver.state.fields: + field.require_grid_space() + +Out of memory errors +==================== + +Spectral simulations with implicit timestepping can require a large amount of memory to store the LHS matrices and their factorizations. +The best way to minimize the required memory is to minimize the LHS matrix size by using as few variables as possible and to minimize the LHS matrix bandwidth (see the :ref:`performance_tips` page). +Beyond this, several of the Dedalus configuration options can be changed the minimize the simulation's memory footprint, potentially at the cost of reduced performance (see the :ref:`configuration` page). + +Reducing memory consumption in Dedalus is an ongoing effort. +Any assistance with memory profiling and contributions reducing the code's memory footprint would be greatly appreciated! + diff --git a/docs/pages/tutorials.rst b/docs/pages/tutorials.rst new file mode 100644 index 00000000..066bcecd --- /dev/null +++ b/docs/pages/tutorials.rst @@ -0,0 +1,37 @@ +Tutorials \& Examples +********************* + +Tutorial Notebooks +================== + +This tutorial on using Dedalus consists of four short IPython notebooks, which can be downloaded and ran interactively, or viewed on-line through the links below. + +The notebooks cover the basics of setting up and interacting with the primary facets of the code, culminating in the setup and simulation of the 1D KdV-Burgers equation. + +.. toctree:: + :maxdepth: 1 + + Tutorial 1: Bases and Domains + Tutorial 2: Fields and Operators + Tutorial 3: Problems and Solvers + Tutorial 4: Analysis and Post-processing + +Example Notebooks +================= + +Below are several notebooks that walk through the setup and execution of more complicated multidimensional example problems. + +.. toctree:: + :maxdepth: 1 + + Example 1: Kelvin-Helmholtz Instability + Example 2: Taylor-Couette Flow + +Example Scripts +=============== + +A wider range of examples are available under the ``examples`` subdirectory of the main code repository, which you can browse `here `_. +These example scripts can be copied to any working directory with the command ``python3 -m dedalus get_examples``. +These examples cover a wider range of use cases, including larger multidimensional problems designed for parallel execution. +Basic post-processing and plotting scripts are also provided with many problems. +These simulation and processing scripts may be useful as a starting point for implementing different problems and equation sets. diff --git a/docs/pages/user_guide.rst b/docs/pages/user_guide.rst new file mode 100644 index 00000000..f938c6fb --- /dev/null +++ b/docs/pages/user_guide.rst @@ -0,0 +1,16 @@ +User Guide \& How-To's +********************** + +This is a work-in-progress user guide and how-to section for Dedalus. +Contributions and suggestions for additions to this section are very welcome! + +.. toctree:: + :maxdepth: 2 + + configuration + performance_tips + troubleshooting + general_functions + parallel_data + output_format + restarting