Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add generic JDBC data source connector #3105

Closed
wants to merge 16 commits into from
Closed

Add generic JDBC data source connector #3105

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
591cacf
Make default target result size configurable in ExecutingStatementRes…
tooptoop4 Dec 25, 2019
9a3902a
address review comments
tooptoop4 Dec 29, 2019
6ee8a38
orig master
tooptoop4 Jan 28, 2020
ae536e0
1st attempt
tooptoop4 Jan 29, 2020
5a0bb71
correct jconn version and case insensitive matching
tooptoop4 Mar 1, 2020
9d17bc0
remove jconn from proj level pom
tooptoop4 Mar 1, 2020
647acb7
Merge branch 'master' into sybasev16
tooptoop4 Mar 1, 2020
812bd92
generic jdbc working with oracle and sybase at least!
tooptoop4 Mar 15, 2020
071cead
update docs
tooptoop4 Mar 17, 2020
3221217
remove -SNAPSHOT from pom
tooptoop4 Mar 17, 2020
c8e2d98
pom version fix and some review comments
tooptoop4 Mar 27, 2020
904e2f1
fix sphinx docs
tooptoop4 Mar 28, 2020
965a4d1
fix test
tooptoop4 Mar 29, 2020
fccb318
impala support
tooptoop4 Sep 18, 2020
15bf43e
Update GenericJdbcClient.java
tooptoop4 Sep 20, 2020
207b513
new 348-snapshot version with fixes
tooptoop4 Jan 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
<packaging>pom</packaging>

<name>presto-root</name>
Expand Down Expand Up @@ -137,6 +137,7 @@
<module>presto-iceberg</module>
<module>presto-google-sheets</module>
<module>presto-bigquery</module>
<module>presto-genericjdbc</module>
tooptoop4 marked this conversation as resolved.
Show resolved Hide resolved
</modules>

<dependencyManagement>
Expand Down Expand Up @@ -436,6 +437,12 @@
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>io.prestosql</groupId>
<artifactId>presto-genericjdbc</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>io.prestosql.hadoop</groupId>
<artifactId>hadoop-apache</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-accumulo/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-accumulo</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-array/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-array</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-atop/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-atop</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-base-jdbc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-base-jdbc</artifactId>
Expand Down
13 changes: 13 additions & 0 deletions presto-base-jdbc/src/main/java/io/prestosql/plugin/jdbc/BaseJdbcConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ public class BaseJdbcConfig
private Set<String> jdbcTypesMappedToVarchar = ImmutableSet.of();
private Duration metadataCacheTtl = new Duration(0, MINUTES);
private boolean cacheMissing;
private String driverClass;

@NotNull
public String getConnectionUrl()
Expand All @@ -49,6 +50,18 @@ public BaseJdbcConfig setConnectionUrl(String connectionUrl)
return this;
}

public String getDriverClass()
{
return driverClass;
}

@Config("driver-class")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This property is unrelated to JDBC base connector. Please move to the generic module.

Suggested change
@Config("driver-class")
@Config("generic-jdbc.driver-class")

public BaseJdbcConfig setDriverClass(String driverClass)
{
this.driverClass = driverClass;
return this;
}

public boolean isCaseInsensitiveNameMatching()
{
return caseInsensitiveNameMatching;
Expand Down
4 changes: 3 additions & 1 deletion presto-base-jdbc/src/main/java/io/prestosql/plugin/jdbc/JdbcErrorCode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,14 @@
import io.prestosql.spi.ErrorType;

import static io.prestosql.spi.ErrorType.EXTERNAL;
import static io.prestosql.spi.ErrorType.INTERNAL_ERROR;

public enum JdbcErrorCode
implements ErrorCodeSupplier
{
JDBC_ERROR(0, EXTERNAL),
JDBC_NON_TRANSIENT_ERROR(1, EXTERNAL);
JDBC_NON_TRANSIENT_ERROR(1, EXTERNAL),
DRIVER_NOT_FOUND(2, INTERNAL_ERROR);
tooptoop4 marked this conversation as resolved.
Show resolved Hide resolved

private final ErrorCode errorCode;

Expand Down
2 changes: 1 addition & 1 deletion presto-benchmark-driver/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-benchmark-driver</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchmark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-benchmark</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchto-benchmarks/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-benchto-benchmarks</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-bigquery/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-bigquery</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-blackhole/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-blackhole</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cassandra/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-cassandra</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cli/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-cli</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-client</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-docs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-docs</artifactId>
Expand Down
1 change: 1 addition & 0 deletions presto-docs/src/main/sphinx/connector.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ from different data sources.
connector/blackhole
connector/cassandra
connector/elasticsearch
connector/genericjdbc
connector/googlesheets
connector/hive
connector/hive-gcs-tutorial
Expand Down
73 changes: 73 additions & 0 deletions presto-docs/src/main/sphinx/connector/genericjdbc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
====================
Generic JDBC Connector
====================

The Generic JDBC connector allows querying tables in an
external database (using the database's native JDBC driver). This can be used to join data between
different systems like Sybase and Hive, or between two different
Oracle instances.

Configuration
-------------

To configure the Generic JDBC connector, create a catalog properties file
in ``etc/catalog`` named, for example, ``sybase.properties``, to
mount the Generic JDBC connector as the ``sybase`` catalog.
Create the file with the following contents, replacing the
connection properties as appropriate for your setup:

.. code-block:: none

connector.name=genericjdbc
driver-class=
connection-url=jdbc:
connection-user=root
connection-password=secret
case-insensitive-name-matching=true

Some examples:

.. code-block:: none

connector.name=genericjdbc
driver-class=com.sybase.jdbc4.jdbc.SybDataSource
connection-url=jdbc:sybase:Tds:localhost:8000/MYSYBASE
connection-user=sa
connection-password=myPassword
case-insensitive-name-matching=true

.. code-block:: none

connector.name=genericjdbc
driver-class=oracle.jdbc.driver.OracleDriver
connection-url=jdbc:oracle:thin:@localhost:49161:xe
connection-user=system
connection-password=rootpw
case-insensitive-name-matching=true

Multiple Generic JDBC Databases or Servers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Generic JDBC connector can only access a single database within
a SGeneric JDBC server. Thus, if you have multiple Generic JDBC databases,
tooptoop4 marked this conversation as resolved.
Show resolved Hide resolved
or want to connect to multiple instances of the Generic JDBC, you must configure
multiple catalogs, one for each instance.

To add another catalog, simply add another properties file to ``etc/catalog``
with a different name, making sure it ends in ``.properties``. For example,
if you name the property file ``sales.properties``, Presto creates a
catalog named ``sales`` using the configured connector.

Specific driver jars
-------------------

You must place the specific JAR for your database (ie ojdbc6.jar, jconn4.jar etc) containing the driver-class into the plugin/genericjdbc folder.

Generic JDBC Connector Limitations
--------------------------------

Read-only access
Not all data types can be mapped
Not all databases will work
Performance
Experimental
2 changes: 1 addition & 1 deletion presto-elasticsearch/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-elasticsearch</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-example-http/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331-SNAPSHOT</version>
<version>331</version>
</parent>

<artifactId>presto-example-http</artifactId>
Expand Down
106 changes: 106 additions & 0 deletions presto-genericjdbc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>io.prestosql</groupId>
<artifactId>presto-root</artifactId>
<version>331</version>
tooptoop4 marked this conversation as resolved.
Show resolved Hide resolved
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>presto-genericjdbc</artifactId>
<description>Presto - Generic JDBC Connector</description>
<packaging>presto-plugin</packaging>

<properties>
<air.main.basedir>${project.parent.basedir}</air.main.basedir>
</properties>

<dependencies>
<dependency>
<groupId>io.prestosql</groupId>
<artifactId>presto-base-jdbc</artifactId>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>configuration</artifactId>
</dependency>

<dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
</dependency>

<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>slice</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>io.prestosql</groupId>
<artifactId>presto-spi</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.openjdk.jol</groupId>
<artifactId>jol-core</artifactId>
<scope>provided</scope>
</dependency>

<!-- for testing -->
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.prestosql</groupId>
<artifactId>presto-main</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.prestosql</groupId>
<artifactId>presto-tpch</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.prestosql</groupId>
<artifactId>presto-testing</artifactId>
<scope>test</scope>
</dependency>


tooptoop4 marked this conversation as resolved.
Show resolved Hide resolved
<dependency>
<groupId>io.airlift</groupId>
<artifactId>testing</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>net.jodah</groupId>
<artifactId>failsafe</artifactId>
<scope>test</scope>
</dependency>
tooptoop4 marked this conversation as resolved.
Show resolved Hide resolved
</dependencies>
</project>
Loading