Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-19148][SQL] do not expose the external table concept in Catalog #16528

Closed
wants to merge 3 commits into from
Closed

[SPARK-19148][SQL] do not expose the external table concept in Catalog #16528

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b0c252a
do not expose the external table concept in Catalog
cloud-fan Jan 10, 2017
55cf0c3
fix python api
cloud-fan Jan 11, 2017
318dc04
revert useless changes
cloud-fan Jan 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions python/pyspark/sql/catalog.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
# limitations under the License.
#

import warnings
from collections import namedtuple

from pyspark import since
Expand Down Expand Up @@ -138,7 +139,27 @@ def listColumns(self, tableName, dbName=None):

@since(2.0)
def createExternalTable(self, tableName, path=None, source=None, schema=None, **options):
"""Creates an external table based on the dataset in a data source.
"""Creates a table based on the dataset in a data source.

It returns the DataFrame associated with the external table.

The data source is specified by the ``source`` and a set of ``options``.
If ``source`` is not specified, the default data source configured by
``spark.sql.sources.default`` will be used.

Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
created external table.

:return: :class:`DataFrame`
"""
warnings.warn(
"createExternalTable is deprecated since Spark 2.2, please use createTable instead.",
DeprecationWarning)
return self.createTable(tableName, path, source, schema, **options)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

**options -> options?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's python syntax, like what we do in scala: func(options: _*)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Got it. I also manually tried it in pyspark. It works fine.


@since(2.2)
def createTable(self, tableName, path=None, source=None, schema=None, **options):
"""Creates a table based on the dataset in a data source.

It returns the DataFrame associated with the external table.

Expand All @@ -157,12 +178,12 @@ def createExternalTable(self, tableName, path=None, source=None, schema=None, **
source = self._sparkSession.conf.get(
"spark.sql.sources.default", "org.apache.spark.sql.parquet")
if schema is None:
df = self._jcatalog.createExternalTable(tableName, source, options)
df = self._jcatalog.createTable(tableName, source, options)
else:
if not isinstance(schema, StructType):
raise TypeError("schema should be StructType")
scala_datatype = self._jsparkSession.parseDataType(schema.json())
df = self._jcatalog.createExternalTable(tableName, source, scala_datatype, options)
df = self._jcatalog.createTable(tableName, source, scala_datatype, options)
return DataFrame(df, self._sparkSession._wrapped)

@since(2.0)
Expand Down