[FEA] disallow nested with_spark_session
and throw an exception if it is nested in integration test
#9412
Labels
test
Only impacts tests
Is your feature request related to a problem? Please describe.
with_spark_session
should not allowed to be nested because it will lead to some unexpected behaviors.with_spark_session
callsreset_spark_session_conf
which does more than just reset the conf.It clears out the catalog too with no way to get the original config or catalog back after it exits.
That means
with_gpu_session
->with_cpu_session
will result in the GPU query running on the CPU after thewith_cpu_session
.Describe the solution you'd like
Throw an exception if
with_spark_session
is nested inside anotherwith_spark_session
.If there are some nested
with_spark_session
in current integration test, we should also fix them.Additional context
This is a follow-on issue of #9405
The text was updated successfully, but these errors were encountered: