Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-23809][SQL][backport] Active SparkSession should be set by getOrCreate #20971

Closed
wants to merge 3 commits into from
Closed

[SPARK-23809][SQL][backport] Active SparkSession should be set by getOrCreate #20971

wants to merge 3 commits into from

Commits on Apr 4, 2018

  1. [SPARK-23809][SQL] Active SparkSession should be set by getOrCreate 

    ## What changes were proposed in this pull request?

Currently, the active spark session is set inconsistently (e.g., in createDataFrame, prior to query execution). Many places in spark also incorrectly query active session when they should be calling activeSession.getOrElse(defaultSession) and so might get None even if a Spark session exists.

The semantics here can be cleaned up if we also set the active session when the default session is set.

Related: https://github.com/apache/spark/pull/20926/files

## How was this patch tested?

Unit test, existing test. Note that if apache#20926 merges first we should also update the tests there.

Author: Eric Liang <[email protected]>

Closes apache#20927 from ericl/active-session-cleanup.
    @ericl
    ericl committed Apr 4, 2018
    Configuration menu
    Copy the full SHA
    f2303dc View commit details
    Browse the repository at this point in the history

  2. Tue Apr 3 17:30:50 PDT 2018

    @ericl
    ericl committed Apr 4, 2018
    Configuration menu
    Copy the full SHA
    e429af1 View commit details
    Browse the repository at this point in the history

  3. Tue Apr 3 17:45:47 PDT 2018

    @ericl
    ericl committed Apr 4, 2018
    Configuration menu
    Copy the full SHA
    36fa1bd View commit details
    Browse the repository at this point in the history