Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

Commit

Permalink
mention limits
Browse files Browse the repository at this point in the history
Signed-off-by: Yuan Zhou <[email protected]>
  • Loading branch information
zhouyuan committed Mar 29, 2021
1 parent d67f215 commit 8d91fb2
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 2 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ orders.createOrReplaceTempView("orders")
spark.sql("select * from orders where o_orderdate > date '1998-07-26'").show(20000, false)
```

The result should showup on Spark console and you can check the DAG diagram with some Columnar Processing stage.
The result should showup on Spark console and you can check the DAG diagram with some Columnar Processing stage. Native SQL engine still lacks some features, please check out the [limitations](./docs/limitations.md).


## Performance data
Expand Down
4 changes: 3 additions & 1 deletion docs/limitation.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@
Native SQL engine currenlty works with Spark 3.0.0 only. There are still some trouble with latest Shuffle/AQE API from Spark 3.0.1, 3.0.2 or 3.1.x.

## Operator limitations
All performance critical operators in TPC-H/TPC-DS should be supported. For those unsupported operators, Native SQL engine will automatically fallback to row operators in vanilla Spark.

### Columnar Projection with Filter
We used 16 bit selection vector for filter so the max batch size need to be < 65536

### Columnar Sort
To reduce the peak memory usage, we used smaller data structure(uin16_t). This limits
Columnar Sort does not support spill to disk yet. To reduce the peak memory usage, we used smaller data structure(uin16_t), so this limits
- the max batch size to be < 65536
- the number of batches in one partiton to be < 65536

Expand Down

0 comments on commit 8d91fb2

Please sign in to comment.