Add stateless reader domains #103
Labels
enhancement
New feature or request
m-materialization
Related to materialization (full or otherwise)
m-operators
Related to operator implementations and semantics
research
This issue represents an open research question, and will require careful thought to fix.
It came up in a discussion today that it'd be neat to have stateless domains below
Reader
nodes that reads could "process" things they extract fromReader
state through. Since the domain is stateless, readers could all process this data-flow in parallel using combinations of data they've read without worrying about synchronization with other executors. This would allow us to move certain compute to the readers, which might let us materialize a less verboseReader
further "up" the graph (e.g., above an expensivejoin
).The text was updated successfully, but these errors were encountered: