deploy: b3396a9

cubed-dev · Nov 16, 2024 · ee94b24 · ee94b24
1 parent e7ef49d
commit ee94b24
Show file tree

Hide file tree

Showing 62 changed files with 266 additions and 261 deletions.
diff --git a/.doctrees/api.doctree b/.doctrees/api.doctree
diff --git a/.doctrees/array-api.doctree b/.doctrees/array-api.doctree
diff --git a/.doctrees/design.doctree b/.doctrees/design.doctree
diff --git a/.doctrees/environment.pickle b/.doctrees/environment.pickle
diff --git a/.doctrees/generated/cubed.rechunk.doctree b/.doctrees/generated/cubed.rechunk.doctree
diff --git a/.doctrees/operations.doctree b/.doctrees/operations.doctree
diff --git a/.doctrees/user-guide/optimization.doctree b/.doctrees/user-guide/optimization.doctree
diff --git a/_images/design.svg b/_images/design.svg
diff --git a/_images/map_direct.svg b/_images/map_direct.svg
diff --git a/_images/map_selection.svg b/_images/map_selection.svg
diff --git a/_images/ops.dot.svg b/_images/ops.dot.svg
diff --git a/_sources/design.md b/_sources/design.md
@@ -4,7 +4,7 @@ Cubed is composed of five layers: from the storage layer at the bottom, to the A
 
 ![Five layer diagram](images/design.svg)
 
-Blue blocks are implemented in Cubed, green in Rechunker, and red in other projects like Zarr and Beam.
+Blue blocks are implemented in Cubed; red blocks in other projects like Zarr and Lithops.
 
 Let's go through the layers from the bottom:
 
@@ -14,7 +14,7 @@ Every _array_ in Cubed is backed by a Zarr array. This means that the array type
 
 ## Runtime
 
-Cubed uses external runtimes for computation. It follows the Rechunker model (and uses its algorithm) to delegate tasks to stateless executors, which include Python (in-process), Lithops, Modal, and Apache Beam.
+Cubed uses external runtimes for computation, delegating tasks to stateless executors, which include Python (in-process), Lithops, Modal, and Apache Beam.
 
 
 ## Primitive operations
@@ -45,8 +45,7 @@ These are built on top of the primitive operations, and provide functions that a
 
     elemwise
     map_blocks
-    map_direct
-    index
+    map_selection
     reduction
     arg_reduction
 ```

diff --git a/_sources/operations.md b/_sources/operations.md
@@ -39,18 +39,20 @@ This example shows how the `squeeze` operation is implemented using `map_blocks`
 
 ![The map_blocks core operation](images/map_blocks.svg)
 
-## `map_direct`
+## `map_selection`
 
-The `map_direct` operation is more general than `map_blocks` since it allows input arrays to be read directly, without regard for block boundaries.
+The `map_selection` operation selects subsets of an input array using standard NumPy indexing notation.
 
 * No input array attributes are preserved in general
-* __Multiple__ inputs, __single__ output
+* __Single__ input, __single__ output
 
-It is a core operation that is implemented using `map_blocks`, which is in turn implemented using `blockwise`. It works by creating an empty array that has the same shape and chunk structure as the output, and calling `map_blocks` on this empty array, passing in the input arrays as side inputs to the function, which may access them in whatever way is needed.
+It is a core operation that is implemented using `blockwise` on the output's blocks. It works by converting indexing selections, such as slices, to keys that refer to blocks in the input array, then retrieving these blocks, slicing them, and assembling them into the final output block.
 
-This example shows how `concat` is implemented using `map_direct`. Each block in the output array is read directly from one or more blocks from the inputs.
+This example shows how `index` is implemented using `map_selection`. Each block in the output array is read directly from one or more blocks from the inputs.
 
-![The map_direct core operation](images/map_direct.svg)
+![The map_selection core operation](images/map_selection.svg)
+
+Note: previously, operations that now use `map_selection` were written using `map_direct`, which allowed input arrays to be read directly. The main difference between the two operations is that `map_selection` tracks which blocks are read by the operation, which allows the optimizer to fuse operations. This is unlike `map_direct`, which does not provide information about block inputs, and therefore cannot be fused by the optimizer.
 
 ## `blockwise`
 
@@ -63,6 +65,13 @@ This example shows how `outer` is implemented using `blockwise`. Each block from
 
 ![The blockwise primitive operation](images/blockwise.svg)
 
+Note: the `general_blockwise` operation is a more general form of `blockwise` that uses a function to specify the block mapping, rather than an index notation, and which supports multiple outputs.
+
+* No input array attributes are preserved in general
+* __Multiple__ inputs, __multiple__ outputs
+
+For multiple outputs, all output arrays must have matching `numblocks`.
+
 (rechunk-operation)=
 ## `rechunk`
 

diff --git a/_sources/user-guide/optimization.md b/_sources/user-guide/optimization.md
@@ -81,9 +81,9 @@ e.visualize()
 The output explains which operations can or can't be fused, and why:
 
 ```
-DEBUG:cubed.core.optimization:can't fuse op-001 since it is not a primitive operation, or it uses map_direct
-DEBUG:cubed.core.optimization:can't fuse op-002 since it is not a primitive operation, or it uses map_direct
-DEBUG:cubed.core.optimization:can't fuse op-003 since it is not a primitive operation, or it uses map_direct
+DEBUG:cubed.core.optimization:can't fuse op-001 since it is not a primitive operation, or it uses an operation that can't be fused (concat or stack)
+DEBUG:cubed.core.optimization:can't fuse op-002 since it is not a primitive operation, or it uses an operation that can't be fused (concat or stack)
+DEBUG:cubed.core.optimization:can't fuse op-003 since it is not a primitive operation, or it uses an operation that can't be fused (concat or stack)
 DEBUG:cubed.core.optimization:can't fuse op-004 since no predecessor ops can be fused
 DEBUG:cubed.primitive.blockwise:can fuse op-005 since num tasks of predecessor ops match
 ```

diff --git a/_static/documentation_options.js b/_static/documentation_options.js
@@ -1,5 +1,5 @@
 const DOCUMENTATION_OPTIONS = {
-    VERSION: 'c75556f',
+    VERSION: 'b3396a9',
     LANGUAGE: 'en',
     COLLAPSE_INDEX: false,
     BUILDER: 'html',

diff --git a/api.html b/api.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="_static/documentation_options.js?v=d5542f40"></script>
     <script src="_static/doctools.js?v=9bcbadda"></script>
     <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="Diagnostics" href="user-guide/diagnostics.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 
@@ -414,7 +414,7 @@ <h2>Chunk-specific functions<a class="headerlink" href="#chunk-specific-function
 <td><p>Apply a function to corresponding blocks from multiple input arrays with some overlap.</p></td>
 </tr>
 <tr class="row-even"><td><p><a class="reference internal" href="generated/cubed.rechunk.html#cubed.rechunk" title="cubed.rechunk"><code class="xref py py-obj docutils literal notranslate"><span class="pre">rechunk</span></code></a></p></td>
-<td><p></p></td>
+<td><p>Change the chunking of an array without changing its shape or data.</p></td>
 </tr>
 </tbody>
 </table>

diff --git a/array-api.html b/array-api.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="_static/documentation_options.js?v=d5542f40"></script>
     <script src="_static/doctools.js?v=9bcbadda"></script>
     <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="cubed.measure_reserved_mem" href="generated/cubed.measure_reserved_mem.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/computation.html b/computation.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="_static/documentation_options.js?v=d5542f40"></script>
     <script src="_static/doctools.js?v=9bcbadda"></script>
     <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="Operations" href="operations.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/configuration.html b/configuration.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="_static/documentation_options.js?v=d5542f40"></script>
     <script src="_static/doctools.js?v=9bcbadda"></script>
     <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="Python Array API" href="array-api.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/contributing.html b/contributing.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="_static/documentation_options.js?v=d5542f40"></script>
     <script src="_static/doctools.js?v=9bcbadda"></script>
     <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -49,7 +49,7 @@
     <link rel="prev" title="Computation" href="computation.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/design.html b/design.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="_static/documentation_options.js?v=d5542f40"></script>
     <script src="_static/doctools.js?v=9bcbadda"></script>
     <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="Related Projects" href="related-projects.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 
@@ -353,15 +353,15 @@ <h2> Contents </h2>
 <h1>Design<a class="headerlink" href="#design" title="Link to this heading">#</a></h1>
 <p>Cubed is composed of five layers: from the storage layer at the bottom, to the Array API layer at the top:</p>
 <p><img alt="Five layer diagram" src="_images/design.svg" /></p>
-<p>Blue blocks are implemented in Cubed, green in Rechunker, and red in other projects like Zarr and Beam.</p>
+<p>Blue blocks are implemented in Cubed; red blocks in other projects like Zarr and Lithops.</p>
 <p>Let’s go through the layers from the bottom:</p>
 <section id="storage">
 <h2>Storage<a class="headerlink" href="#storage" title="Link to this heading">#</a></h2>
 <p>Every <em>array</em> in Cubed is backed by a Zarr array. This means that the array type inherits Zarr attributes including the underlying store (which may be on local disk, or a cloud store, for example), as well as the shape, dtype, and chunks. Chunks are the unit of storage and computation in this system.</p>
 </section>
 <section id="runtime">
 <h2>Runtime<a class="headerlink" href="#runtime" title="Link to this heading">#</a></h2>
-<p>Cubed uses external runtimes for computation. It follows the Rechunker model (and uses its algorithm) to delegate tasks to stateless executors, which include Python (in-process), Lithops, Modal, and Apache Beam.</p>
+<p>Cubed uses external runtimes for computation, delegating tasks to stateless executors, which include Python (in-process), Lithops, Modal, and Apache Beam.</p>
 </section>
 <section id="primitive-operations">
 <h2>Primitive operations<a class="headerlink" href="#primitive-operations" title="Link to this heading">#</a></h2>
@@ -394,16 +394,13 @@ <h2>Core operations<a class="headerlink" href="#core-operations" title="Link to
 <tr class="row-even"><td><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_blocks</span></code></p></td>
 <td><p>Apply a function to corresponding blocks from multiple input arrays.</p></td>
 </tr>
-<tr class="row-odd"><td><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_direct</span></code></p></td>
-<td><p>Apply a function across blocks of a new array, reading directly from side inputs (not necessarily in a blockwise fashion).</p></td>
+<tr class="row-odd"><td><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_selection</span></code></p></td>
+<td><p>Apply a function to selected subsets of an input array using standard NumPy indexing notation.</p></td>
 </tr>
-<tr class="row-even"><td><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">index</span></code></p></td>
-<td><p>Subset an array, along one or more axes.</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">reduction</span></code></p></td>
+<tr class="row-even"><td><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">reduction</span></code></p></td>
 <td><p>Apply a function to reduce an array along one or more axes.</p></td>
 </tr>
-<tr class="row-even"><td><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">arg_reduction</span></code></p></td>
+<tr class="row-odd"><td><p><code class="xref py py-obj docutils literal notranslate"><span class="pre">arg_reduction</span></code></p></td>
 <td><p>A reduction that returns the array indexes, not the values.</p></td>
 </tr>
 </tbody>

diff --git a/generated/cubed.Array.compute.html b/generated/cubed.Array.compute.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="../_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="../_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="../_static/documentation_options.js?v=d5542f40"></script>
     <script src="../_static/doctools.js?v=9bcbadda"></script>
     <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="../_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="cubed.Array" href="cubed.Array.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/generated/cubed.Array.html b/generated/cubed.Array.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="../_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="../_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="../_static/documentation_options.js?v=d5542f40"></script>
     <script src="../_static/doctools.js?v=9bcbadda"></script>
     <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="../_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="API Reference" href="../api.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/generated/cubed.Array.rechunk.html b/generated/cubed.Array.rechunk.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="../_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="../_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="../_static/documentation_options.js?v=d5542f40"></script>
     <script src="../_static/doctools.js?v=9bcbadda"></script>
     <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="../_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="cubed.Array.compute" href="cubed.Array.compute.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/generated/cubed.Array.visualize.html b/generated/cubed.Array.visualize.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="../_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="../_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="../_static/documentation_options.js?v=d5542f40"></script>
     <script src="../_static/doctools.js?v=9bcbadda"></script>
     <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="../_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="cubed.Array.rechunk" href="cubed.Array.rechunk.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/generated/cubed.Callback.html b/generated/cubed.Callback.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="../_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="../_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="../_static/documentation_options.js?v=d5542f40"></script>
     <script src="../_static/doctools.js?v=9bcbadda"></script>
     <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="../_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="cubed.random.random" href="cubed.random.random.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>
 

diff --git a/generated/cubed.Spec.html b/generated/cubed.Spec.html
@@ -39,7 +39,7 @@
   <link rel="preload" as="script" href="../_static/scripts/bootstrap.js?digest=26a4bc78f4c0ddb94549" />
 <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=26a4bc78f4c0ddb94549" />
 
-    <script src="../_static/documentation_options.js?v=9e750ebb"></script>
+    <script src="../_static/documentation_options.js?v=d5542f40"></script>
     <script src="../_static/doctools.js?v=9bcbadda"></script>
     <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
     <script src="../_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
@@ -50,7 +50,7 @@
     <link rel="prev" title="cubed.Callback" href="cubed.Callback.html" />
   <meta name="viewport" content="width=device-width, initial-scale=1"/>
   <meta name="docsearch:language" content="en"/>
-  <meta name="docsearch:version" content="c75556f
+  <meta name="docsearch:version" content="b3396a9
 " />
   </head>