Documentation (spcl#1051)

.github/workflows/fpga-ci.yml

@@ -8,6 +8,7 @@ on:
 
 jobs:
   test-fpga:
+    if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
     runs-on: [self-hosted, linux, intel-fpga, xilinx-fpga]
     steps:
     - uses: actions/checkout@v2

.github/workflows/general-ci.yml

@@ -8,6 +8,7 @@ on:
 
 jobs:
   test:
+    if: "!contains(github.event.pull_request.labels.*.name, 'no-ci')"
     runs-on: ubuntu-latest
     strategy:
       matrix:

.github/workflows/gpu-ci.yml

@@ -8,6 +8,7 @@ on:
 
 jobs:
   test-gpu:
+    if: "!contains(github.event.pull_request.labels.*.name, 'no-ci')"
     runs-on: [self-hosted, linux, gpu]
     steps:
     - uses: actions/checkout@v2

.github/workflows/heterogeneous-ci.yml

@@ -8,6 +8,7 @@ on:
 
 jobs:
   test-heterogeneous:
+    if: "!contains(github.event.pull_request.labels.*.name, 'no-ci')"
     runs-on: [self-hosted, linux]
     steps:
     - uses: actions/checkout@v2

.gitignore

@@ -189,3 +189,4 @@ _build/
 
 
 .*cache/
+/doc/source/config_schema.rst

.readthedocs.yaml

@@ -5,6 +5,11 @@
 # Required
 version: 2
 
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.9"
+
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
   configuration: doc/conf.py

CONTRIBUTING.md

@@ -13,6 +13,30 @@ We follow the [Google Python Style Guide](https://google.github.io/styleguide/py
 * **Type Hints**: New functions must include proper Python [typing information](https://docs.python.org/3/library/typing.html), in order to support type checking and smoother development.
 * **Importing classes and functions directly**: This is disallowed, with the exception of directly importing the following main graph components (which are heavily reused throughout the framework): `SDFG, SDFGState, Memlet, InterstateEdge`.
 * **Inline imports**: Imports usually go at the top of a Python file, after the copyright statement and the file docstring. If you must place an `import` statement anywhere else, indicate the reason with an adjacent comment (e.g., `# Avoid import loop`).
+* **docstrings**: We use [Sphinx](https://www.sphinx-doc.org/) for documentation. Use type hints as much as possible (this will be automatically integrated into the documentation) and the following format:
+
+```python
+def example_function(param_a: str, *args: Optional[SDFG]) -> bool:
+    """
+    Explain what the function does. Note the double line break below, after
+    description and before parameter declaration! Without it Sphinx does not work.
+
+    :param param_a: What ``param_a`` does. Double backticks indicate code format in Sphinx.
+    :param args: Variable-length arguments are documented just like standard parameters.
+    :return: True if example, False otherwise.
+    :note: Some notes can be used here. See Sphinx for the full list of available annotations.
+    :note: Numbered and itemized lists must also have a blank line and must be indented.
+
+    If you want to include a code sample, use:
+
+    .. code-block:: python
+
+        # Note the empty line above
+        example_use = example_function('hello', None, None, SDFG('world'))
+    """
+    ...
+```
+
 
 For automatic styling, we use the [yapf](https://github.com/google/yapf) file formatter.
 **Please run `yapf` before making your pull request ready for review.**

README.md

@@ -11,9 +11,9 @@
 
 _Decoupling domain science from performance optimization._
 
-DaCe is a parallel programming framework that takes code in Python/NumPy and 
-other programming languages, and maps it to high-performance **CPU, GPU, and FPGA** 
-programs, which can be optimized to achieve state-of-the-art. Internally, DaCe 
+DaCe is a [fast](https://nbviewer.org/github/spcl/dace/blob/master/tutorials/benchmarking.ipynb) parallel programming
+framework that takes code in Python/NumPy and other programming languages, and maps it to high-performance 
+**CPU, GPU, and FPGA** programs, which can be optimized to achieve state-of-the-art. Internally, DaCe 
 uses the Stateful DataFlow multiGraph (SDFG) *data-centric intermediate 
 representation*: A transformable, interactive representation of code based on 
 data movement.
@@ -30,19 +30,16 @@ DaCe generates high-performance programs for:
  * Xilinx and Intel FPGAs
 
 DaCe can be written inline in Python and transformed in the command-line/Jupyter 
-Notebooks or SDFGs can be interactively modified using the Data-centric 
-Interactive Optimization Development Environment [Visual Studio Code extension](https://marketplace.visualstudio.com/items?itemName=phschaad.sdfv).
+Notebooks or SDFGs can be interactively modified using our [Visual Studio Code extension](https://marketplace.visualstudio.com/items?itemName=phschaad.sdfv).
 
-For more information, see our [paper](http://www.arxiv.org/abs/1902.10345).
-
-See an example SDFG [in the standalone viewer (SDFV)](https://spcl.github.io/dace/sdfv.html?url=https://spcl.github.io/dace/examples/gemm.sdfg).
+## [For more information, see the documentation](https://spcldace.readthedocs.io/en/latest/)
 
 Quick Start
 -----------
 
 Install DaCe with pip: `pip install dace`
 
-Having issues? See [Troubleshooting](#Troubleshooting)
+Having issues? See our full [Installation and Troubleshooting Guide](https://spcldace.readthedocs.io/en/latest/setup/installation.html).
 
 Using DaCe in Python is as simple as adding a `@dace` decorator:
 ```python
@@ -56,8 +53,7 @@ def myprogram(a):
     return np.sum(a)
 ```
 
-Calling `myprogram` with any NumPy array or 
-`__{cuda_}array_interface__`-supporting tensor (e.g., PyTorch, Numba) will 
+Calling `myprogram` with any NumPy array or GPU array (e.g., PyTorch, Numba, CuPy) will 
 generate data-centric code, compile, and run it. From here on out, you can 
 _optimize_ (interactively or automatically), _instrument_, and _distribute_ 
 your code. The code creates a shared library (DLL/SO file) that can readily 
@@ -73,38 +69,12 @@ For more information on how to use DaCe, see the [samples](samples) or tutorials
 * [Using and Creating Transformations](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/transformations.ipynb)
 * [Extending the Code Generator](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/codegen.ipynb)
 
-Dependencies
-------------
-
-Runtime dependencies:
- * A C++14-capable compiler (e.g., gcc 5.3+)
- * Python 3.7 or newer (Python 3.6 is supported but not actively tested)
- * CMake 3.15 or newer
-
-Running
--------
-
-**Python scripts:** Run DaCe programs (in implicit or explicit syntax) using Python directly.
-
-**[SDFV (standalone SDFG viewer)](https://spcl.github.io/dace/sdfv.html):** To view SDFGs separately, run the `sdfv` installed script with the `.sdfg` file as an argument. Alternatively, you can use the link or open `dace/viewer/webclient/sdfv.html` directly and choose a file in the browser.
-
-**Visual Studio Code extension**: Install from the [VSCode marketplace](https://marketplace.visualstudio.com/items?itemName=phschaad.sdfv) or open an `.sdfg` file for interactive SDFG viewing and transformation.
-
-**The sdfgcc tool:** Compile `.sdfg` files with `sdfgcc program.sdfg`. Interactive command-line optimization is possible with the `--optimize` flag.
-
-**Jupyter Notebooks:** DaCe is Jupyter-compatible. If a result is an SDFG or a state, it will show up directly in the notebook. See the [tutorials](tutorials) for examples.
-
-**Octave scripts (experimental):** `.m` files can be run using the installed script `dacelab`, which will create the appropriate SDFG file.
-
-**Note for Windows/Visual C++ users:** If compilation fails in the linkage phase, try setting the following environment variable to force Visual C++ to use Multi-Threaded linkage:
-```
-X:\path\to\dace> set _CL_=/MT
-```
-
-
 Publication
 -----------
 
+The paper for the SDFG IR can be found [here](http://www.arxiv.org/abs/1902.10345).
+Other DaCe-related publications are available on our [website](http://spcl.inf.ethz.ch/dace).
+
 If you use DaCe, cite us:
 ```bibtex
 @inproceedings{dace,
@@ -116,57 +86,6 @@ If you use DaCe, cite us:
 }
 ```
 
-Troubleshooting
----------------
-
-* If you are using DaCe from the git repository and getting missing dependencies or missing include files, make sure you cloned the repository recursively (with `git clone --recursive`) and that the submodules are up to date.
-* If you are running on Mac OS and getting compilation errors when calling DaCe programs, make sure you have OpenMP installed and configured with Apple Clang. Otherwise, you can use GCC to compile the code by following these steps:
-    * Run `brew install gcc`
-	* Set your `~/.dace.conf` compiler configuration to use the installed GCC. For example, if you installed version 9 (`brew install gcc@9`), run `which g++-9` and set the config entry called `compiler.cpu.executable` (empty string by default) to the resulting path
-	* Remove any `.dacecache` folders to clear the cache
-
-Other issues? Look for similar issues or start a discussion on our [GitHub Discussions](https://github.com/spcl/dace/discussions)!
-
-
-Configuration
--------------
-
-DaCe creates a file called `.dace.conf` in the user's home directory. It provides useful settings that can be modified either directly in the file (YAML) or overridden on a case-by-case basis using environment variables that begin with `DACE_` and specify the setting (where categories are separated by underscores). The full configuration schema is located [here](dace/config_schema.yml).
-
-The priority order for configuration files is as follows:
- 1. If a `DACE_*` environment variable is found, its value will always be used
- 2. If `with dace.config.set_temporary(...)` is used ([see example here](samples/simple/laplace.py#L43))
- 3. A `.dace.conf` located in the current working directory
- 4. The `.dace.conf` located in the user's home directory or the path pointed to by the `DACE_CONFIG` environment variable
-
-If no configuration file can be created in any of the above paths, default settings will be used.
-
-Useful environment variable configurations include:
-
- * `DACE_CONFIG` (default: `~/.dace.conf`): Override DaCe configuration file choice.
-
-General configuration:
- * `DACE_debugprint` (default: False): Print debugging information.
- * `DACE_compiler_use_cache` (default: False): Uses DaCe program cache instead of re-optimizing and compiling programs.
- * `DACE_compiler_default_data_types` (default: `Python`): Chooses default types for integer and floating-point values. If `Python` is chosen, `int` and `float` are both 64-bit wide. If `C` is chosen, `int` and `float` are 32-bit wide.
-
-Profiling:
- * `DACE_profiling` (default: False): Enables profiling measurement of the DaCe program runtime in milliseconds. Produces a log file and prints out median runtime.
- * `DACE_treps` (default: 100): Number of repetitions to run a DaCe program when profiling is enabled.
-
-GPU programming and debugging:
- * `DACE_compiler_cuda_backend` (default: `cuda`): Chooses the GPU backend to use (can be `cuda` for NVIDIA GPUs or `hip` for AMD GPUs).
- * `DACE_compiler_cuda_syncdebug` (default: False): If True, calls device-synchronization after every GPU kernel and checks for errors. Good for checking crashes or invalid memory accesses.
-
-FPGA programming:
- * `DACE_compiler_fpga_vendor`: (default: `xilinx`): Can be `xilinx` for Xilinx FPGAs, or `intel_fpga` for Intel FPGAs.
-
-SDFG interactive transformation:
- * `DACE_optimizer_transform_on_call` (default: False): Uses the transformation command line interface every time a `@dace` function is called.
- * `DACE_optimizer_interface` (default: `dace.transformation.optimizer.SDFGOptimizer`): Controls the SDFG optimization process if `transform_on_call` is enabled. By default, uses the transformation command line interface.
- * `DACE_optimizer_automatic_simplification` (default: True): If False, skips automatic simplification in the Python frontend (see transformations tutorial for more information).
-
-
 Contributing
 ------------
 DaCe is an open-source project. We are happy to accept Pull Requests with your contributions! Please follow the [contribution guidelines](CONTRIBUTING.md) before submitting a pull request.

dace/cli/sdfv.py

@@ -24,13 +24,15 @@ class NewCls(cls):
 
 
 def view(sdfg: dace.SDFG, filename: Optional[Union[str, int]] = None):
-    """View an sdfg in the system's HTML viewer
-       :param sdfg: the sdfg to view, either as `dace.SDFG` object or a json string
-       :param filename: the filename to write the HTML to. If `None`, a
-                        temporary file will be created. If an integer,
-                        the generated HTML and related sources will be
-                        served using a basic web server on that port,
-                        blocking the current thread.
+    """
+    View an sdfg in the system's HTML viewer
+
+    :param sdfg: the sdfg to view, either as `dace.SDFG` object or a json string
+    :param filename: the filename to write the HTML to. If `None`, a
+                    temporary file will be created. If an integer,
+                    the generated HTML and related sources will be
+                    served using a basic web server on that port,
+                    blocking the current thread.
     """
     # If vscode is open, try to open it inside vscode
     if filename is None:

dace/codegen/codegen.py

@@ -147,10 +147,12 @@ def _get_codegen_targets(sdfg: SDFG, frame: framecode.DaCeCodeGenerator):
 
 
 def generate_code(sdfg, validate=True) -> List[CodeObject]:
-    """ Generates code as a list of code objects for a given SDFG.
-        :param sdfg: The SDFG to use
-        :param validate: If True, validates the SDFG before generating the code.
-        :return: List of code objects that correspond to files to compile.
+    """
+    Generates code as a list of code objects for a given SDFG.
+
+    :param sdfg: The SDFG to use
+    :param validate: If True, validates the SDFG before generating the code.
+    :return: List of code objects that correspond to files to compile.
     """
     from dace.codegen.targets.target import TargetCodeGenerator  # Avoid import loop
 

dace/codegen/common.py

@@ -41,6 +41,7 @@ def _sym2cpp(s, arrayexprs):
 def sym2cpp(s, arrayexprs: Optional[Set[str]] = None) -> Union[str, List[str]]:
     """ 
     Converts an array of symbolic variables (or one) to C++ strings. 
+    
     :param s: Symbolic expression to convert.
     :param arrayexprs: Set of names of arrays, used to convert SymPy 
                        user-functions back to array expressions.

dace/codegen/compiled_sdfg.py

@@ -18,14 +18,17 @@
 
 
 class ReloadableDLL(object):
-    """ A reloadable shared object (or dynamically linked library), which
-        bypasses Python's dynamic library reloading issues. """
-
+    """
+    A reloadable shared object (or dynamically linked library), which
+    bypasses Python's dynamic library reloading issues.
+    """
     def __init__(self, library_filename, program_name):
-        """ Creates a new reloadable shared object.
-            :param library_filename: Path to library file.
-            :param program_name: Name of the DaCe program (for use in finding
-                                 the stub library loader).
+        """
+        Creates a new reloadable shared object.
+        
+        :param library_filename: Path to library file.
+        :param program_name: Name of the DaCe program (for use in finding
+                             the stub library loader).
         """
         self._stub_filename = os.path.join(
             os.path.dirname(os.path.realpath(library_filename)),
@@ -145,6 +148,7 @@ def _array_interface_ptr(array: Any, array_type: dt.Array) -> int:
     If the given array implements ``__array_interface__`` (see
     ``dtypes.is_array``), returns the base host or device pointer to the
     array's allocated memory.
+
     :param array: Array object that implements NumPy's array interface.
     :param array_type: Data descriptor of the array (used to get storage
                        location to determine whether it's a host or GPU device
@@ -192,6 +196,7 @@ def get_exported_function(self, name: str, restype=None) -> Optional[Callable[..
         """
         Tries to find a symbol by name in the compiled SDFG, and convert it to a callable function
         with the (optionally) given return type (void by default). If no such function exists, returns None.
+
         :param name: Name of the function to query.
         :return: Callable to the function, or None if doesn't exist.
         """
@@ -205,7 +210,8 @@ def get_state_struct(self) -> ctypes.Structure:
             consecutive entries in the struct that are pointers. As soon as a non-pointer or other unparseable field is
             encountered, the method exits early. All fields defined until then will nevertheless be available in the
             structure.
-            :returns: the ctypes.Structure representation of the state struct.
+            
+            :return: the ctypes.Structure representation of the state struct.
         """
 
         return ctypes.cast(self._libhandle, ctypes.POINTER(self._try_parse_state_struct())).contents
@@ -265,6 +271,7 @@ def _initialize(self, argtuple):
     def initialize(self, *args, **kwargs):
         """
         Initializes the compiled SDFG without invoking it. 
+
         :param args: Arguments to call SDFG with.
         :param kwargs: Keyword arguments to call SDFG with.
         :return: If successful, returns the library handle (as a ctypes pointer).

dace/codegen/compiler.py

@@ -183,6 +183,8 @@ def configure_and_compile(program_folder, program_name=None, output_stream=None)
 
     cmake_command.append("-DDACE_LIBS=\"{}\"".format(" ".join(sorted(libraries))))
 
+    cmake_command.append(f"-DCMAKE_BUILD_TYPE={Config.get('compiler', 'build_type')}")
+
     # Set linker and linker arguments, iff they have been specified
     cmake_linker = Config.get('compiler', 'linker', 'executable') or ''
     cmake_linker = cmake_linker.strip()
@@ -195,6 +197,9 @@ def configure_and_compile(program_folder, program_name=None, output_stream=None)
         cmake_command.append(f'-DCMAKE_SHARED_LINKER_FLAGS="{cmake_link_flags}"')
     cmake_command = ' '.join(cmake_command)
 
+    if Config.get('debugprint') == 'verbose':
+        print(f'Running CMake: {cmake_command}')
+
     cmake_filename = os.path.join(build_folder, 'cmake_configure.sh')
     ##############################################
     # Configure
@@ -252,6 +257,7 @@ def get_environment_flags(environments) -> Tuple[List[str], Set[str]]:
     """
     Returns the CMake environment and linkage flags associated with the
     given input environments/libraries.
+    
     :param environments: A list of ``@dace.library.environment``-decorated
                          classes.
     :return: A 2-tuple of (environment CMake flags, linkage CMake flags)