Issue #70 - Add Serverless Support #73

currantw · 2024-11-13T23:56:57Z

Description

Adds support for OpenSearch Serverless to the ODBC driver.

Issues Resolved

Check List

New functionality includes testing.
Automated testing has been deferred to future issue (see below). See comments for summary of manual testing.
All tests pass, including unit test, integration test and doctest.
Pipeline job build-mac is failing. This is an existing failure on main, and I have raised a new issue to fix it (see below)
New functionality has been documented.
New functionality has javadoc added - Not applicable
New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Related Issues

Signed-off-by: currantw <[email protected]>

.gitignore

src/sqlodbc/dlg_specific.h

src/sqlodbc/opensearch_communication.cpp

src/sqlodbc/dlg_specific.c

Signed-off-by: currantw <[email protected]>

src/sqlodbc/opensearch_communication.cpp

src/sqlodbc/opensearch_communication.h

Signed-off-by: currantw <[email protected]>

docs/user/configuration_options.md

currantw · 2024-11-18T18:50:29Z

The build_mac action is also failing on the main project (https://github.com/opensearch-project/sql-odbc). Currently, building locally on Mac fails unless the -Werror flag is removed, so this is perhaps the cause of this failure.

Raised this issue to address this: #74

src/sqlodbc/opensearch_communication.cpp

…e` for consistency with CLI implementation. Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

currantw · 2024-11-18T21:33:34Z

Automated testing is non-trivial, and has been deferred to #75. In the meantime, we have performed the following to test this new behaviour on Mac:

TESTING SERVERLESS

Amazon OpenSearch Serverless cluster was setup for testing.
See comments for more details.

✅ ODBC connects successfully with iodbctest.

TESTING NON-SERVERLESS WITH EDITED HOST FILE

A local OpenSearch cluster was setup for testing.

✅ ODBC connects successfully with iodbctest.

NOTES ON TESTING

The following steps where taken to setup and test the Serverless cluster:

Created a new OpenSearch Serverless cluster that includes SQL.
Updated my AWS credentials file ( to add opensearchodbc profile with the appropriate access key, secret access key, and session token.
Load some test data to OpenSearch using Postman.
Downloaded the repo and build the .dll using CMake.
Updated my odbc.ini and odbcinst.ini files to configure the ODBC (see below)
Download iodbctest (brew install libiodbc).
Run iodbctest, connect to the corresponding DSN, and query the test data.

# odbc.ini

[ODBC Data Sources]
OpenSearchServerlessDSN = OpenSearchODBC

[OpenSearchServerlessDSN]
Host                 = https://<HOST-URL>.aoss.amazonaws.com
Port                 = 443
UseSSL               = 1
ResponseTimeout      = 10
HostnameVerification = 1
Region               = <AWS_REGION>
Auth                 = AWS_SIGV4
LogOutput            = /Library/ODBC/opensearch-sql-odbc
Driver               = <PROJECT_ROOT>/build/odbc/lib/libsqlodbc.dylib
ServerlessOverride   = 1

# odbcinst.ini
[ODBC Drivers]
OpenSearchODBC = Installed

[OpenSearchODBC]
Driver = <PROJECT_ROOT>/build/odbc/lib/libsqlodbc.dylib
Setup  = <PROJECT_ROOT>//build/odbc/lib/libsqlodbc.dylib

src/sqlodbc/opensearch_communication.cpp

src/sqlodbc/opensearch_communication.h

Signed-off-by: currantw <[email protected]>

MaxKsyunz · 2024-11-19T01:42:08Z

src/sqlodbc/opensearch_types.h

@@ -272,6 +272,7 @@ typedef struct connection_options {
    std::string port;
    std::string timeout;
    std::string fetch_size;
+    std::string is_serverless;


Can you make this a bool please?

Signed-off-by: currantw <[email protected]>

…g _boolean_ configuration value (which defaults to false). Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

currantw · 2024-11-21T03:54:49Z

Automated testing is non-trivial, and has been deferred to #75. In the meantime, we have performed the following to test this new behaviour on Mac:

TESTING SERVERLESS

Amazon OpenSearch Serverless cluster was setup for testing. See comments for more details.

✅ ODBC connects successfully with iodbctest.

TESTING NON-SERVERLESS WITH EDITED HOST FILE

A local OpenSearch cluster was setup for testing.

✅ ODBC connects successfully with iodbctest.

NOTES ON TESTING

The following steps where taken to setup and test the Serverless cluster:

Created a new OpenSearch Serverless cluster that includes SQL.

Updated my AWS credentials file ( to add opensearchodbc profile with the appropriate access key, secret access key, and session token.

Load some test data to OpenSearch using Postman.

Downloaded the repo and build the .dll using CMake.

Updated my odbc.ini and odbcinst.ini files to configure the ODBC (see below)

Download iodbctest (brew install libiodbc).

Run iodbctest, connect to the corresponding DSN, and query the test data.

``` # odbc.ini
[ODBC Data Sources]
OpenSearchServerlessDSN = OpenSearchODBC

[OpenSearchServerlessDSN]
Host = https://.aoss.amazonaws.com
Port = 443
UseSSL = 1
ResponseTimeout = 10
HostnameVerification = 1
Region = <AWS_REGION>
Auth = AWS_SIGV4
LogOutput = /Library/ODBC/opensearch-sql-odbc
Driver = <PROJECT_ROOT>/build/odbc/lib/libsqlodbc.dylib
ServerlessOverride = 1

odbcinst.ini

[ODBC Drivers]
OpenSearchODBC = Installed

[OpenSearchODBC]
Driver = <PROJECT_ROOT>/build/odbc/lib/libsqlodbc.dylib
Setup = <PROJECT_ROOT>//build/odbc/lib/libsqlodbc.dylib

currantw requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, vamsimanohar, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto and GumpacG as code owners November 13, 2024 23:56

currantw added 2 commits November 13, 2024 15:58

Initial support for OpenSearch Serverless, based simply on parsing URL

b401f48

Signed-off-by: currantw <[email protected]>

Minor unrelated cleanup

b855e2d

Signed-off-by: currantw <[email protected]>

currantw force-pushed the AOS-2291_serverless branch from 0f13e9e to b855e2d Compare November 13, 2024 23:58

currantw added 5 commits November 14, 2024 13:58

Initial updates to add IsServerless configuration option.

8491671

Signed-off-by: currantw <[email protected]>

Add .idea/* to project .gitignore

c490706

Signed-off-by: currantw <[email protected]>

Fix small errors

0820b35

Signed-off-by: currantw <[email protected]>

Update documentation

44336bf

Signed-off-by: currantw <[email protected]>

Minor cleanuo

f39036d

Signed-off-by: currantw <[email protected]>

currantw commented Nov 15, 2024

View reviewed changes

.gitignore Show resolved Hide resolved

src/sqlodbc/dlg_specific.h Show resolved Hide resolved

src/sqlodbc/opensearch_communication.cpp Show resolved Hide resolved

src/sqlodbc/dlg_specific.c Outdated Show resolved Hide resolved

currantw mentioned this pull request Nov 15, 2024

[FEATURE] SQL Serverless support #70

Open

currantw added 2 commits November 15, 2024 11:40

Fix format string error

29ac366

Signed-off-by: currantw <[email protected]>

Exclude .pyc files from test runner, which supports Python in venv

929088e

Signed-off-by: currantw <[email protected]>

jduo reviewed Nov 15, 2024

View reviewed changes

src/sqlodbc/opensearch_communication.cpp Outdated Show resolved Hide resolved

jduo reviewed Nov 15, 2024

View reviewed changes

src/sqlodbc/opensearch_communication.h Outdated Show resolved Hide resolved

currantw added 2 commits November 15, 2024 15:41

Add is_serverless member variable to avoid repeatedly determining it.

aa76d77

Signed-off-by: currantw <[email protected]>

Also exclude .o and .d object files from test runner.

63ffb0c

Signed-off-by: currantw <[email protected]>

jduo reviewed Nov 18, 2024

View reviewed changes

docs/user/configuration_options.md Outdated Show resolved Hide resolved

jduo reviewed Nov 18, 2024

View reviewed changes

src/sqlodbc/opensearch_communication.cpp Outdated Show resolved Hide resolved

currantw added 4 commits November 18, 2024 11:50

Rename configuration option from IsServerless to `ServerlessOverrid…

7fcfa87

…e` for consistency with CLI implementation. Signed-off-by: currantw <[email protected]>

Add default is_servrless value to connection tests.

f0b3abf

Signed-off-by: currantw <[email protected]>

Revert unintentional change.

caefd49

Signed-off-by: currantw <[email protected]>

Add clarifying comments.

ec3386a

Signed-off-by: currantw <[email protected]>

acarbonetto approved these changes Nov 19, 2024

View reviewed changes

src/sqlodbc/opensearch_communication.cpp Outdated Show resolved Hide resolved

src/sqlodbc/opensearch_communication.h Outdated Show resolved Hide resolved

Add constant for OpenSearch distribution.

caf9d39

Signed-off-by: currantw <[email protected]>

MaxKsyunz requested changes Nov 19, 2024

View reviewed changes

currantw added 5 commits November 19, 2024 11:56

Rename from is_serverless to is_aoss_serverless.

4e55ffe

Signed-off-by: currantw <[email protected]>

Update to first determine if serverless via URL, then default to usin…

8e2f718

…g _boolean_ configuration value (which defaults to false). Signed-off-by: currantw <[email protected]>

Simplify logic, remove tertiary

bed6886

Signed-off-by: currantw <[email protected]>

Remove ServerlessOverride configuration option.

c5b48f7

Signed-off-by: currantw <[email protected]>

Minor comment cleanup

effccef

Signed-off-by: currantw <[email protected]>

MaxKsyunz self-requested a review November 21, 2024 04:31

MaxKsyunz approved these changes Nov 21, 2024

View reviewed changes

MaxKsyunz merged commit 1ac3e6d into opensearch-project:main Nov 21, 2024
5 of 8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #70 - Add Serverless Support #73

Issue #70 - Add Serverless Support #73

currantw commented Nov 13, 2024 •

edited

Loading

currantw commented Nov 18, 2024 •

edited

Loading

currantw commented Nov 18, 2024 •

edited

Loading

MaxKsyunz Nov 19, 2024

currantw Nov 19, 2024

currantw commented Nov 21, 2024

odbcinst.ini

Issue #70 - Add Serverless Support #73

Issue #70 - Add Serverless Support #73

Conversation

currantw commented Nov 13, 2024 • edited Loading

Description

Issues Resolved

Check List

Related Issues

currantw commented Nov 18, 2024 • edited Loading

currantw commented Nov 18, 2024 • edited Loading

MaxKsyunz Nov 19, 2024

Choose a reason for hiding this comment

currantw Nov 19, 2024

Choose a reason for hiding this comment

currantw commented Nov 21, 2024

odbcinst.ini

currantw commented Nov 13, 2024 •

edited

Loading

currantw commented Nov 18, 2024 •

edited

Loading

currantw commented Nov 18, 2024 •

edited

Loading