-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add EMR client to spark connector #1790
Conversation
Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Codecov Report
@@ Coverage Diff @@
## main #1790 +/- ##
============================================
+ Coverage 97.33% 97.38% +0.04%
- Complexity 4490 4569 +79
============================================
Files 394 401 +7
Lines 11118 11340 +222
Branches 795 828 +33
============================================
+ Hits 10822 11043 +221
- Misses 289 290 +1
Partials 7 7
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
core/src/main/java/org/opensearch/sql/data/model/ExprBooleanValue.java
Outdated
Show resolved
Hide resolved
spark/src/main/java/org/opensearch/sql/spark/data/constants/SparkConstants.java
Outdated
Show resolved
Hide resolved
.../java/org/opensearch/sql/spark/functions/response/DefaultSparkSqlFunctionResponseHandle.java
Outdated
Show resolved
Hide resolved
.../java/org/opensearch/sql/spark/functions/response/DefaultSparkSqlFunctionResponseHandle.java
Outdated
Show resolved
Hide resolved
.../java/org/opensearch/sql/spark/functions/response/DefaultSparkSqlFunctionResponseHandle.java
Outdated
Show resolved
Hide resolved
spark/src/main/java/org/opensearch/sql/spark/storage/SparkMetricScan.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/opensearch/sql/spark/functions/scan/SparkSqlFunctionTableScanOperator.java
Show resolved
Hide resolved
spark/src/main/java/org/opensearch/sql/spark/client/EmrClientImpl.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Rupal Mahajan <[email protected]>
when do we delete the query result? |
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
We need to set ISM policy for result index to delete results. |
spark/src/main/java/org/opensearch/sql/spark/client/EmrClientImpl.java
Outdated
Show resolved
Hide resolved
spark/src/main/java/org/opensearch/sql/spark/client/EmrClientImpl.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/opensearch/sql/spark/functions/scan/SparkSqlFunctionTableScanOperator.java
Outdated
Show resolved
Hide resolved
spark/src/main/java/org/opensearch/sql/spark/storage/implementor/SparkDefaultImplementor.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Added delete query result after retrieving from index |
The backport to
To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.9 2.9
# Navigate to the new working tree
cd .worktrees/backport-2.9
# Create a new branch
git switch --create backport/backport-1790-to-2.9
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 392a72000224f36a75222b22398357933d18c015
# Push it to GitHub
git push --set-upstream origin backport/backport-1790-to-2.9
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.9 Then, create a pull request where the |
* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Add emr client Signed-off-by: Rupal Mahajan <[email protected]> * Set default values for flint args Signed-off-by: Rupal Mahajan <[email protected]> * Validate emr auth type Signed-off-by: Rupal Mahajan <[email protected]> * Add default constants for flint Signed-off-by: Rupal Mahajan <[email protected]> * Update unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * tests draft Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Rename classes and update tests Signed-off-by: Rupal Mahajan <[email protected]> * Update scan operator test Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> * Fix Connection pool shut down issue Signed-off-by: Rupal Mahajan <[email protected]> * Update emr unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Update doc and tests Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Update EMR clinet impl tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Make spark & flint jars configurable Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Add spark application id in logs Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Delete result when not required Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> (cherry picked from commit 392a720)
* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Add emr client Signed-off-by: Rupal Mahajan <[email protected]> * Set default values for flint args Signed-off-by: Rupal Mahajan <[email protected]> * Validate emr auth type Signed-off-by: Rupal Mahajan <[email protected]> * Add default constants for flint Signed-off-by: Rupal Mahajan <[email protected]> * Update unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * tests draft Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Rename classes and update tests Signed-off-by: Rupal Mahajan <[email protected]> * Update scan operator test Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> * Fix Connection pool shut down issue Signed-off-by: Rupal Mahajan <[email protected]> * Update emr unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Update doc and tests Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Update EMR clinet impl tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Make spark & flint jars configurable Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Add spark application id in logs Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Delete result when not required Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> (cherry picked from commit 392a720) Co-authored-by: Rupal Mahajan <[email protected]>
* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Add emr client Signed-off-by: Rupal Mahajan <[email protected]> * Set default values for flint args Signed-off-by: Rupal Mahajan <[email protected]> * Validate emr auth type Signed-off-by: Rupal Mahajan <[email protected]> * Add default constants for flint Signed-off-by: Rupal Mahajan <[email protected]> * Update unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * tests draft Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Rename classes and update tests Signed-off-by: Rupal Mahajan <[email protected]> * Update scan operator test Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> * Fix Connection pool shut down issue Signed-off-by: Rupal Mahajan <[email protected]> * Update emr unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Update doc and tests Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Update EMR clinet impl tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Make spark & flint jars configurable Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Add spark application id in logs Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Delete result when not required Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> Signed-off-by: Mitchell Gale <[email protected]>
* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Add emr client Signed-off-by: Rupal Mahajan <[email protected]> * Set default values for flint args Signed-off-by: Rupal Mahajan <[email protected]> * Validate emr auth type Signed-off-by: Rupal Mahajan <[email protected]> * Add default constants for flint Signed-off-by: Rupal Mahajan <[email protected]> * Update unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * tests draft Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Rename classes and update tests Signed-off-by: Rupal Mahajan <[email protected]> * Update scan operator test Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> * Fix Connection pool shut down issue Signed-off-by: Rupal Mahajan <[email protected]> * Update emr unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Update doc and tests Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Update EMR clinet impl tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Make spark & flint jars configurable Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Add spark application id in logs Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Delete result when not required Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> Signed-off-by: Mitchell Gale <[email protected]>
* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Add emr client Signed-off-by: Rupal Mahajan <[email protected]> * Set default values for flint args Signed-off-by: Rupal Mahajan <[email protected]> * Validate emr auth type Signed-off-by: Rupal Mahajan <[email protected]> * Add default constants for flint Signed-off-by: Rupal Mahajan <[email protected]> * Update unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * tests draft Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Rename classes and update tests Signed-off-by: Rupal Mahajan <[email protected]> * Update scan operator test Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> * Fix Connection pool shut down issue Signed-off-by: Rupal Mahajan <[email protected]> * Update emr unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Update doc and tests Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Update EMR clinet impl tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Make spark & flint jars configurable Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Add spark application id in logs Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Delete result when not required Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]>
* Create Spark Connector * Add spark client and engine * Remove vars * Spark connector draft * nit * Fix checkstyle errors * nit * Fix license header * Add spark storage test * Update comments * Fix checkstyle in comments * Update tests * Add emr client * Set default values for flint args * Validate emr auth type * Add default constants for flint * Update unit tests * Address PR comments * tests draft * Refactor class name * Rename classes and update tests * Update scan operator test * Address PR comment * Fix Connection pool shut down issue * Update emr unit tests * Update doc and tests * nit * Update EMR clinet impl tests * Address PR comments * Make spark & flint jars configurable * Address comments * Add spark application id in logs * nit * Delete result when not required --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]>
* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Add emr client Signed-off-by: Rupal Mahajan <[email protected]> * Set default values for flint args Signed-off-by: Rupal Mahajan <[email protected]> * Validate emr auth type Signed-off-by: Rupal Mahajan <[email protected]> * Add default constants for flint Signed-off-by: Rupal Mahajan <[email protected]> * Update unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * tests draft Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Rename classes and update tests Signed-off-by: Rupal Mahajan <[email protected]> * Update scan operator test Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> * Fix Connection pool shut down issue Signed-off-by: Rupal Mahajan <[email protected]> * Update emr unit tests Signed-off-by: Rupal Mahajan <[email protected]> * Update doc and tests Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Update EMR clinet impl tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Make spark & flint jars configurable Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Add spark application id in logs Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Delete result when not required Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> Signed-off-by: Mitchell Gale <[email protected]>
Description
DO NOT MERGE before #1780
Issues Resolved
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.