ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] #6343

andygrove · 2020-02-03T16:03:59Z

This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.

  @Test
  public void executeQuery() throws SQLException {
    try (Connection conn = driver.connect("jdbc:arrow://localhost:50051", new Properties())) {
      try (Statement stmt = conn.createStatement()) {
        try (ResultSet rs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
          List<Integer> ids = new ArrayList<>();
          while (rs.next()) {
            ids.add(rs.getInt(1));
          }
          assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
        }
      }
    }
  }

github-actions · 2020-02-03T16:37:42Z

https://issues.apache.org/jira/browse/ARROW-7744

lidavidm

Added some minor notes on the Flight stuff. Overall this is pretty interesting...

lidavidm · 2020-02-09T00:12:25Z

java/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java

+  private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);
+
+  /** JDBC connection string prefix. */
+  private static final String PREFIX = "jdbc:arrow://";


Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.

lidavidm · 2020-02-09T00:12:50Z

java/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java

+  @Override
+  public Connection connect(String url, Properties properties) throws SQLException {
+    logger.info("connect() url={}", url);
+    //TODO this needs much more work to parse full URLs but this is enough to get end to end tests running


java.net.URI should be able to do all the parsing.

lidavidm · 2020-02-09T00:15:17Z

java/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

+  @Override
+  public ResultSet executeQuery(String query) throws SQLException {
+
+    FlightClient client = FlightClient.builder()


Ideally this client should be able to live in the driver instance and get shared between statements.

lidavidm · 2020-02-09T00:16:13Z

java/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

+
+    CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);
+
+    Ticket ticket = new Ticket(query.getBytes());


I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.

lidavidm · 2020-02-09T00:16:53Z

java/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

+        .location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
+        .build();
+
+    CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);


Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.

julianhyde · 2020-02-13T03:12:09Z

It's worth considering Calcite's Avatica as the basis for the JDBC driver.

Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.

Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.

Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.

andygrove · 2020-02-16T15:46:04Z

@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.

liyafan82 · 2020-02-19T10:17:07Z

java/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

+  public boolean next() throws SQLException {
+    if (batchIndex + 1 == root.getRowCount()) {
+      if (getNextBatch()) {
+        batchIndex++;


Here we should reset batchIndex to 0?

liyafan82 · 2020-02-19T10:21:21Z

java/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

+
+  @Override
+  public int getInt(String columnName) throws SQLException {
+    return ResultSetHelper.getInt(getObject(columnName));


There are boxing/unboxing overheads here, which should be eliminated in the future.

liyafan82 · 2020-02-19T10:25:21Z

java/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

+
+  @Override
+  public BigDecimal getBigDecimal(String s) throws SQLException {
+    throw new SQLFeatureNotSupportedException();


It seems this can also be implemented in a way similar to getBigDecimal(int i)?

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.

andygrove · 2020-02-21T17:22:12Z

@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?

andygrove · 2020-03-29T15:52:18Z

I'm closing this PR for now since it is stale.

andygrove added the Component: Java label Feb 3, 2020

andygrove self-assigned this Feb 3, 2020

andygrove changed the title ~~ARROW-7744: [Java] Implement Flight JDBC Driver~~ ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] Feb 3, 2020

andygrove marked this pull request as ready for review February 3, 2020 22:43

kszucs force-pushed the master branch from b18ed44 to e79c251 Compare February 7, 2020 07:41

andygrove added 11 commits February 7, 2020 11:08

Basic skeleton for JDBC driver

9a12bc6

add unit test

acbaf12

test executing query and fetching one row

f15f31a

rename classes to use Flight prefix to simplify imports

3c70a5b

move type conversion code to helper class for easier unit testing

d91884f

make most methods throw SQLFeatureNotSupportedException

fbd9549

Fix mvn verify issues

78b3083

ignore test

ba9fd9f

Implement ResultSet.next() and more data type conversion logic

767e9e7

ResultSetMetaData and bug fixes

3e174d2

Add java.sql.Driver

3abf8bb

kszucs force-pushed the ARROW-7744 branch from 319f9e3 to 3abf8bb Compare February 7, 2020 10:08

lidavidm reviewed Feb 9, 2020

View reviewed changes

liyafan82 reviewed Feb 19, 2020

View reviewed changes

andygrove closed this Mar 29, 2020

asfimport mentioned this pull request Sep 30, 2022

[FlightRPC][Java] Implement Flight SQL JDBC Driver #23982

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] #6343

ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] #6343

andygrove commented Feb 3, 2020 •

edited

Loading

github-actions bot commented Feb 3, 2020

lidavidm left a comment

lidavidm Feb 9, 2020

lidavidm Feb 9, 2020

lidavidm Feb 9, 2020

lidavidm Feb 9, 2020

lidavidm Feb 9, 2020

andygrove Feb 11, 2020

lidavidm Feb 9, 2020

julianhyde commented Feb 13, 2020

andygrove commented Feb 16, 2020

liyafan82 Feb 19, 2020

liyafan82 Feb 19, 2020

liyafan82 Feb 19, 2020

andygrove Feb 21, 2020

andygrove commented Feb 21, 2020

andygrove commented Mar 29, 2020


		CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

		Ticket ticket = new Ticket(query.getBytes());

ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] #6343

ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] #6343

Conversation

andygrove commented Feb 3, 2020 • edited Loading

github-actions bot commented Feb 3, 2020

lidavidm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julianhyde commented Feb 13, 2020

andygrove commented Feb 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andygrove commented Feb 21, 2020

andygrove commented Mar 29, 2020

andygrove commented Feb 3, 2020 •

edited

Loading