Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix merge conflict from branch-24.02 #10544

Merged
merged 5 commits into from
Mar 5, 2024
Merged

Fix merge conflict from branch-24.02 #10544

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e1c2f77
Update rapids jni and private dependency version to 24.02.1 (#10511)
NvTimLiu Feb 29, 2024
c4abf14
Add missed shims for scala2.13 (#10465)
NvTimLiu Mar 1, 2024
23f1bc4
Update changelog for v24.02.0 release (#10525)
NvTimLiu Mar 1, 2024
4d14b75
Fix Delta log cache size settings during integration tests (#10541)
jlowe Mar 4, 2024
82158aa
Merge branch 'branch-24.02' into fix-merge
jlowe Mar 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion integration_tests/run_pyspark_from_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,9 @@ else
export PYSP_TEST_spark_ui_enabled=${PYSP_TEST_spark_ui_enabled:-false}

# Set the Delta log cache size to prevent the driver from caching every Delta log indefinitely
export PYSP_TEST_spark_driver_extraJavaOptions="-ea -Duser.timezone=$TZ -Ddelta.log.cacheSize=10 $COVERAGE_SUBMIT_FLAGS"
export PYSP_TEST_spark_databricks_delta_delta_log_cacheSize=${PYSP_TEST_spark_databricks_delta_delta_log_cacheSize:-10}
deltaCacheSize=$PYSP_TEST_spark_databricks_delta_delta_log_cacheSize
export PYSP_TEST_spark_driver_extraJavaOptions="-ea -Duser.timezone=$TZ -Ddelta.log.cacheSize=$deltaCacheSize $COVERAGE_SUBMIT_FLAGS"
export PYSP_TEST_spark_executor_extraJavaOptions="-ea -Duser.timezone=$TZ"
export PYSP_TEST_spark_ui_showConsoleProgress='false'
export PYSP_TEST_spark_sql_session_timeZone=$TZ
Expand Down Expand Up @@ -384,6 +386,7 @@ EOF

# avoid double processing of variables passed to spark in
# spark_conf_init
unset PYSP_TEST_spark_databricks_delta_delta_log_cacheSize
unset PYSP_TEST_spark_driver_extraClassPath
unset PYSP_TEST_spark_driver_extraJavaOptions
unset PYSP_TEST_spark_jars
Expand All @@ -395,6 +398,7 @@ EOF
--driver-java-options "$driverJavaOpts" \
$SPARK_SUBMIT_FLAGS \
--conf 'spark.rapids.memory.gpu.allocSize='"$gpuAllocSize" \
--conf 'spark.databricks.delta.delta.log.cacheSize='"$deltaCacheSize" \
"${RUN_TESTS_COMMAND[@]}" "${TEST_COMMON_OPTS[@]}"
fi
fi
Loading