Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Manual Backport 2.x] Flint query scheduler part1 - integrate job scheduler plugin #2834 #2888

Closed

Conversation

noCharger
Copy link
Collaborator

Description

Backport 3daf64f from #2834.

opensearch-trigger-bot bot and others added 30 commits May 30, 2023 13:24
…#1680)

Signed-off-by: Andrew Carbonetto <[email protected]>
(cherry picked from commit 567cabc)

Co-authored-by: Andrew Carbonetto <[email protected]>
…earch-project#1677)

* Create new anonymizer for new engine (opensearch-project#266)

* Created anonymizer listener for anonymizing SQL queries through the new engine
Signed-off-by: Matthew Wells <[email protected]>

* Update for review comments

Signed-off-by: Andrew Carbonetto <[email protected]>

* added missing file header, change public variable to private

Signed-off-by: Matthew Wells <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: Matthew Wells <[email protected]>
Co-authored-by: Andrew Carbonetto <[email protected]>
(cherry picked from commit 62120fd)

Co-authored-by: Matthew Wells <[email protected]>
…rch-project#1682)

* Support Alternate Datetime Formats (opensearch-project#268)

* Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types

Signed-off-by: Andrew Carbonetto <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: GabeFernandez310 <[email protected]>
Signed-off-by: Guian Gumpac <[email protected]>
Signed-off-by: MaxKsyunz <[email protected]>
Co-authored-by: Andrew Carbonetto <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>
Co-authored-by: MaxKsyunz <[email protected]>
(cherry picked from commit 986db39)

Co-authored-by: Guian Gumpac <[email protected]>
…<table>` queries (opensearch-project#1684)

* Support for pagination in v2 engine of `SELECT * FROM <table>` queries (opensearch-project#1666)

v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON.

Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details.
---------

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Max Ksyunz <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>
Co-authored-by: Andrew Carbonetto <[email protected]>
(cherry picked from commit 57ce303)

* Fix test build failure.

Somehow the import is required in 2.x but not 3.0

Signed-off-by: MaxKsyunz <[email protected]>

---------

Signed-off-by: MaxKsyunz <[email protected]>
Co-authored-by: Max Ksyunz <[email protected]>
…pensearch-project#1689)

* Allow backtick around fields in sort cmd

Signed-off-by: Joshua Li <[email protected]>

* Add test

Signed-off-by: Joshua Li <[email protected]>

* Update test setup

Signed-off-by: Joshua Li <[email protected]>

* revert ignoring sql-cli repo if exists

Signed-off-by: Joshua Li <[email protected]>

---------

Signed-off-by: Joshua Li <[email protected]>
(cherry picked from commit 691012d)

Co-authored-by: Joshua Li <[email protected]>
Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
…nsearch-project#272) (opensearch-project#1716) (opensearch-project#1734)

* Fixed bug of byte/short values not handling divide/modulus arithmetic equations

Signed-off-by: Matthew Wells <[email protected]>
(cherry picked from commit 2c80631)

Co-authored-by: Matthew Wells <[email protected]>
…lause and for functions and expressions in the query. (opensearch-project#1500) (opensearch-project#1741)

* Add support for `WHERE` clause, column list in `SELECT` clause and for functions and expressions in the query.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix merge issue and address PR feedback by updating comments.

Signed-off-by: Yury-Fridlyand <[email protected]>

* More comments.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add extra check for unset `initialSearchRequest`.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>
(cherry picked from commit da386e5)

Co-authored-by: Yury-Fridlyand <[email protected]>
…ROM`. (opensearch-project#1599) (opensearch-project#1745)

* Support `ORDER BY` clauses in pagination and queries without `FROM`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix IT.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>
(cherry picked from commit 94d5479)

Co-authored-by: Yury-Fridlyand <[email protected]>
* Fix ML-commons missing dependency.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix `mockito` dependency.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Revert changes in `:opensearch` since it is not needed anymore.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>
(cherry picked from commit f6e2a97)

Co-authored-by: Yury-Fridlyand <[email protected]>
…s upon project failure in startOpenSearch (opensearch-project#1763)

* fixed buildgradle merge conflicts

Signed-off-by: Mitchell Gale <[email protected]>

* Fix CI (opensearch-project#1760)

* Fix ML-commons missing dependency.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix `mockito` dependency.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Revert changes in `:opensearch` since it is not needed anymore.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Mitchell Gale <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>
…ct#1725) (opensearch-project#1770)

* fixed round documentation

Signed-off-by: Matthew Wells <[email protected]>
(cherry picked from commit 34cad6e)

Co-authored-by: Matthew Wells <[email protected]>
(cherry picked from commit c7dfdb3)

Signed-off-by: forestmvey <[email protected]>
Co-authored-by: Forest Vey <[email protected]>
* Update SQL plugin for core refactor (opensearch-project#1571)

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix plugin compilation (opensearch-project#1580)

* Changed gradle version and removed values iterator

Signed-off-by: Guian Gumpac <[email protected]>

* Update a test to match new indexResponse.aliases() type.

Signed-off-by: MaxKsyunz <[email protected]>

* Ran ./gradlew wrapper

Signed-off-by: Guian Gumpac <[email protected]>

---------

Signed-off-by: Guian Gumpac <[email protected]>
Signed-off-by: MaxKsyunz <[email protected]>
Co-authored-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>

* Update sqlite-jdbc to 3.41.2.2 to address CVE-2023-32697 (opensearch-project#1667)

* Update sqlite-jdbc to 3.41.2.2 to address CVE-2023-32697

Signed-off-by: MaxKsyunz <[email protected]>

* Don't check column names on H2 results for correctness tests as described in opensearch-project#1667 (comment).

Signed-off-by: Yury-Fridlyand <[email protected]>

* Address PR review comment.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Guian Gumpac <[email protected]>
Co-authored-by: MaxKsyunz <[email protected]>
…ct#1737) (opensearch-project#1791)

* Add support for Array and ExprValue Parsing With Inner Hits



* Adding schema validation for IT test, and another UT for nested arrays.



* Making handleAggregationResponse a private function.



---------


(cherry picked from commit 9fbcf11)

Signed-off-by: forestmvey <[email protected]>
Co-authored-by: Forest Vey <[email protected]>
…t#1789) (opensearch-project#1794)

* Add Support for Nested Function in Order By Clause (#280)

* Adding order by clause support for nested function.

Signed-off-by: forestmvey <[email protected]>

* Adding test coverage for nested in ORDER BY clause.

Signed-off-by: forestmvey <[email protected]>

* Added nested function validation to NestedAnalyzer.

Signed-off-by: forestmvey <[email protected]>

---------

Signed-off-by: forestmvey <[email protected]>

* Adding semantic check for missing arguments in function and unit test.

Signed-off-by: forestmvey <[email protected]>

---------

Signed-off-by: forestmvey <[email protected]>
(cherry picked from commit 3302ec8)

Co-authored-by: Forest Vey <[email protected]>
* Fix CSV/RAW output header being application/json rather than plain/text (opensearch-project#1779)

* Fix CI (opensearch-project#1760)

* Fix ML-commons missing dependency.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix `mockito` dependency.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Revert changes in `:opensearch` since it is not needed anymore.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Matthew Wells <[email protected]>

* Fix CSV/RAW outputting wrong format (opensearch-project#279)

* Fixed bug where CSV/RAW outputs as JSON rather than plain text

Signed-off-by: Matthew Wells <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Matthew Wells <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>
(cherry picked from commit 1ec696d)

* updated tests

Signed-off-by: Matthew Wells <[email protected]>

* updated tests to return name

Signed-off-by: Matthew Wells <[email protected]>

* changed tests to return value

Signed-off-by: Matthew Wells <[email protected]>

* removed unneeded imports

Signed-off-by: Matthew Wells <[email protected]>

---------

Signed-off-by: Matthew Wells <[email protected]>
…ensearch-project#1802)

* Keep up with refactoring in OpenSearch.



* Updating code formatting.



---------


(cherry picked from commit 3fc11a4)

Signed-off-by: MaxKsyunz <[email protected]>
Co-authored-by: Max Ksyunz <[email protected]>
…earch-project#1812)

* Updating imports to reflect changes in opensearch core. (opensearch-project#1645)

Signed-off-by: MaxKsyunz <[email protected]>
(cherry picked from commit 8e5d766)
Signed-off-by: Max Ksyunz <[email protected]>

* Forcing httpclient to 4.5.14

Signed-off-by: Max Ksyunz <[email protected]>

---------

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Max Ksyunz <[email protected]>
Co-authored-by: Max Ksyunz <[email protected]>
Co-authored-by: Max Ksyunz <[email protected]>
Signed-off-by: Sean Kao <[email protected]>
(cherry picked from commit 41f3405)

Co-authored-by: Sean Kao <[email protected]>
… (opensearch-project#1797)

* Add Support for Field Star in Nested Function.



* Removing toString for NestedAllTupleFields.



* Adding IT test for nested all fields in invalid clause of SQL statement.



* Use utility function for checking is nested in NestedAnalyzer.



* Formatting fixes.



---------


(cherry picked from commit fa840e0)

Signed-off-by: forestmvey <[email protected]>
Co-authored-by: Forest Vey <[email protected]>
) (opensearch-project#1832)

* Added new datetime functions and aliases to PPL (opensearch-project#281)

* Added new functions, and alias' to PPL

Signed-off-by: Matthew Wells <[email protected]>

* removed message about functions not working with dates with 0 month/day as those dates don't exist

Signed-off-by: Matthew Wells <[email protected]>

* changed documentation to be in alphabetical order

Signed-off-by: Matthew Wells <[email protected]>

---------

Signed-off-by: Matthew Wells <[email protected]>
(cherry picked from commit e44d983)

Co-authored-by: Matthew Wells <[email protected]>
) (opensearch-project#1821) (opensearch-project#1830)

* Support user-defined and incomplete date formats (opensearch-project#273)

* Check custom formats for characters

Signed-off-by: Guian Gumpac <[email protected]>

* Removed duplicated code

Signed-off-by: Guian Gumpac <[email protected]>

* Reworked checking for exprcoretype

Signed-off-by: Guian Gumpac <[email protected]>

* Changed check for time

Signed-off-by: Guian Gumpac <[email protected]>

* Rework processing custom and incomplete formats and add tests.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Values of incomplete and incorrect formats to be returned as `TIMESTAMP` instead of `STRING`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Complete fix and update tests.

Signed-off-by: Yury-Fridlyand <[email protected]>

* More fixes for god of fixes.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Guian Gumpac <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>

* Refactoring.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Guian Gumpac <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: Guian Gumpac <[email protected]>
(cherry picked from commit a60b222)

Co-authored-by: Yury-Fridlyand <[email protected]>
opensearch-trigger-bot bot and others added 29 commits June 21, 2024 15:29
(cherry picked from commit 9ad4e02)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…project#2770)

(cherry picked from commit fbff4a3)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…nsearch-project#2744) (opensearch-project#2752)

(cherry picked from commit 7b40c2c)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…ect#2771)

* Abstract FlintIndex client



* Fix log



* Fix test function name



---------


(cherry picked from commit b2403ca)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…pensearch-project#2779)

(cherry picked from commit b9f544b)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…2778) (opensearch-project#2782)

(cherry picked from commit 8eae36f)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
) (opensearch-project#2785)

(cherry picked from commit 49e2e0e)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
…-project#2723) (opensearch-project#2811)

(cherry picked from commit d639796)

Signed-off-by: Frank Dattalo <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peng Huo <[email protected]>
…-project#2720) (opensearch-project#2810)

(cherry picked from commit c063d5e)

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peng Huo <[email protected]>
…ject#2819)

* Fix node issue in CI



* fix linux ci



---------


(cherry picked from commit f241f34)

Signed-off-by: Rupal Mahajan <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-project#2791)

* Fix SQLQueryUtils to extract multiple tables



* Improve test coverage



---------


(cherry picked from commit 883cc7e)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…ect#2786) (opensearch-project#2792)

(cherry picked from commit 00f82f5)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…opensearch-project#2822)

* Pass accountId to EMRServerlessClientFactory.getClient (opensearch-project#2783)

Signed-off-by: Tomoyuki Morita <[email protected]>
(cherry picked from commit e24b51f)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix build failure

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-project#2823)

(cherry picked from commit eb5c824)

Signed-off-by: Derek Ho <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…IndexDescriptors (opensearch-project#2772) (opensearch-project#2817)

* Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors



* Add Spark Buffer index pattern



* Upgrade to checkout v4



* Fix CI issues



* Move env



* Revert "Upgrade to checkout v4"

This reverts commit b35573b.

---------




(cherry picked from commit d3ecb2d)

Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Derek Ho <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Derek Ho <[email protected]>
…pensearch-project#2824)

* Introduce SparkParameterComposerCollection

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix comments

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix integ test

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>
(cherry picked from commit a151a7d)
…e fields as part of Lucene query (opensearch-project#2762) (opensearch-project#2849)

Github Issue - opensearch-project#2700


(cherry picked from commit 0fad56d)

Signed-off-by: Manasvini B S <[email protected]>
…nsearch-project#2839)

(cherry picked from commit 607354c)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…earch-project#2838) (opensearch-project#2855)

(cherry picked from commit 2117650)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…roject#2861)

---------
(cherry picked from commit a5ede64)

Signed-off-by: Chen Dai <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…pensearch-project#2865) (opensearch-project#2867)

(cherry picked from commit 41100be)

Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
… method (opensearch-project#2866) (opensearch-project#2872)

* Add RequestContext parameter to verifyDataSourceAccessAndGetRawMetadata method

Signed-off-by: Tomoyuki Morita <[email protected]>

* Add comments

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix style

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>

(cherry picked from commit ba82e12)
opensearch-project#2834)

* [Feature] Flint query scheduler part1 - integrate job scheduler plugin

Signed-off-by: Louis Chu <[email protected]>

* Add comments

Signed-off-by: Louis Chu <[email protected]>

* Add unit test

Signed-off-by: Louis Chu <[email protected]>

* Remove test rest API

Signed-off-by: Louis Chu <[email protected]>

* Fix doc test

Signed-off-by: Louis Chu <[email protected]>

* Add more tests

Signed-off-by: Louis Chu <[email protected]>

* Fix IT

Signed-off-by: Louis Chu <[email protected]>

* Fix IT with security

Signed-off-by: Louis Chu <[email protected]>

* Improve test coverage

Signed-off-by: Louis Chu <[email protected]>

* Fix integTest cluster

Signed-off-by: Louis Chu <[email protected]>

* Fix UT

Signed-off-by: Louis Chu <[email protected]>

* Update UT

Signed-off-by: Louis Chu <[email protected]>

* Fix bwc test

Signed-off-by: Louis Chu <[email protected]>

* Resolve comments

Signed-off-by: Louis Chu <[email protected]>

* Fix bwc test

Signed-off-by: Louis Chu <[email protected]>

* clean up doc test

Signed-off-by: Louis Chu <[email protected]>

* Resolve comments

Signed-off-by: Louis Chu <[email protected]>

* Fix UT

Signed-off-by: Louis Chu <[email protected]>

---------

Signed-off-by: Louis Chu <[email protected]>
(cherry picked from commit 3daf64f)
@noCharger noCharger closed this Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.