From d67f2155c73e2436f215f04e65a45cf29dfcfbe7 Mon Sep 17 00:00:00 2001 From: Yuan Zhou Date: Mon, 29 Mar 2021 11:42:55 +0800 Subject: [PATCH] adding limitations Signed-off-by: Yuan Zhou --- docs/limitation.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 docs/limitation.md diff --git a/docs/limitation.md b/docs/limitation.md new file mode 100644 index 000000000..92887d4ff --- /dev/null +++ b/docs/limitation.md @@ -0,0 +1,15 @@ +# Limitations for Native SQL Engine + +## Spark compability +Native SQL engine currenlty works with Spark 3.0.0 only. There are still some trouble with latest Shuffle/AQE API from Spark 3.0.1, 3.0.2 or 3.1.x. + +## Operator limitations +### Columnar Projection with Filter +We used 16 bit selection vector for filter so the max batch size need to be < 65536 + +### Columnar Sort +To reduce the peak memory usage, we used smaller data structure(uin16_t). This limits +- the max batch size to be < 65536 +- the number of batches in one partiton to be < 65536 + +