[NSE-383] Release SMJ input data immediately after being used #387
Conversation
zhztheplayer
changed the title
zhztheplayer
changed the title
|
Arrow patch oap-project/arrow#25 |
|
@zhouyuan @weiting-chen Another required change is this branch needs arrow-dataset-jni.so to be dynamically linked. This may alter the standard build process so we may have another discussion if needed |
|
@zhztheplayer |
|
will checkout Arrow JNI iterator codes to avoid introducing more binary dependencies |
|
@zhouyuan Based on previous discussion I am about to get the refactor on Arrow jni util work https://github.com/oap-project/arrow/pull/27/files. As a result we'll not have to change the c++ dependency of Gazelle. |
zhztheplayer
force-pushed
the
NSE-383
branch
2 times, most recently
from
7106960 to
d16744f
Compare
| @@ -568,7 +592,7 @@ case class ColumnarWholeStageCodegenExec(child: SparkPlan)(val codegenStageId: I | |||
| def close = { | |||
| closed = true | |||
| pipelineTime += (eval_elapse + build_elapse) / 1000000 | |||
| buildRelationBatchHolder.foreach(_.close) | |||
| buildRelationBatchHolder.foreach(_.close) // fixing: ref cnt goes nagative | |||
There was a problem hiding this comment.
this seems not necessary now as the iter will be closed when kernel close?
There was a problem hiding this comment.
It seems like BHJ & SHJ are still using this for caching data
| @@ -586,7 +586,7 @@ arrow::Status CompileCodes(std::string codes, std::string signature) { | |||
| char* env_codegen_option_ = std::getenv("CODEGEN_OPTION"); | |||
|
|
|||
| if (env_codegen_option_ == nullptr) { | |||
| env_codegen_option_ = " -O3 -march=native "; | |||
| env_codegen_option_ = " -O0 -g "; | |||
There was a problem hiding this comment.
This is not intended. Will remove
|
@zhztheplayer rebase? |
zhztheplayer
changed the title
Also, add switch option spark.oap.sql.columnar.sortmergejoin.lazyread to Spark config.
|
A switch added to enable this, set |
No description provided.