Merge feature/maximus-m1 to 2.x #1078

penghuo · 2022-11-16T03:59:47Z

Description

Merge feature/maximus-m1 to 2.x. There is no client facing API exposed.

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ject#822) Signed-off-by: penghuo <[email protected]> Signed-off-by: penghuo <[email protected]>

* Add create index function in opensearch node client Signed-off-by: Chen Dai <[email protected]> * Add create index function in opensearch rest client Signed-off-by: Chen Dai <[email protected]> * Add create index function in OpenSearchIndex table Signed-off-by: Chen Dai <[email protected]> * Separate index exist API from create index function Signed-off-by: Chen Dai <[email protected]> * Separate table exist API from create table function Signed-off-by: Chen Dai <[email protected]> * Add UT and Todo comment for OpenSearch system index Signed-off-by: Chen Dai <[email protected]> * Fix broken UT in core due to new API in Table interface Signed-off-by: Chen Dai <[email protected]> * Add more UT and javadoc for OpenSearch data type mapping Signed-off-by: Chen Dai <[email protected]> * Rename isExist to exists which looks more readable Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Peng Huo <[email protected]>

* Add window and tumbling window assigner Signed-off-by: Chen Dai <[email protected]> * Add sliding window assigner Signed-off-by: Chen Dai <[email protected]> * Address PR comments Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Peng Huo <[email protected]>

…imus-m1/merge Merge 2.x to feature/maximus-m1 branch

…earch-project#974) Signed-off-by: Peng Huo <[email protected]>

Signed-off-by: Peng Huo <[email protected]>

* Add watermark generator Signed-off-by: Chen Dai <[email protected]> * Update Javadoc and UT for readability Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>

* Add stream context and window trigger Signed-off-by: Chen Dai <[email protected]> * Update Javadoc with more details Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Peng Huo <[email protected]>

Signed-off-by: Chen Dai <[email protected]>

…ter-release Merge 2.x to Maximus feature branch

…urce Signed-off-by: Peng Huo <[email protected]>

Signed-off-by: Peng Huo <[email protected]>

* Make span collector not special Signed-off-by: Chen Dai <[email protected]> * Add more UT for span Signed-off-by: Chen Dai <[email protected]> * Prepare for removing span collector Signed-off-by: Chen Dai <[email protected]> * Delete span collector Signed-off-by: Chen Dai <[email protected]> * Delete unused allocate and locate code in Rounding class Signed-off-by: Chen Dai <[email protected]> * Fix broken UT Signed-off-by: Chen Dai <[email protected]> * Update javadoc Signed-off-by: Chen Dai <[email protected]> * Fix broken UT after merge Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>

* Add Streaming Source Impl Signed-off-by: Peng Huo <[email protected]> * update build.gradle Signed-off-by: Peng Huo <[email protected]> * change to hadoop-fs Signed-off-by: Peng Huo <[email protected]> * exclude FileSystemStreamSource from jacoco Signed-off-by: Peng Huo <[email protected]> * exclude unnecessary depedency Signed-off-by: Peng Huo <[email protected]> * Update integ-test depedency Signed-off-by: Peng Huo <[email protected]> * change from splits to split in batch Signed-off-by: Peng Huo <[email protected]> Signed-off-by: Peng Huo <[email protected]>

Signed-off-by: Peng Huo <[email protected]>

* Fix `FLOAT` -> `DOUBLE` cast. (opensearch-project#1025) Signed-off-by: Yury-Fridlyand <[email protected]> * Fix error messaging from prometheus. (opensearch-project#1029) (opensearch-project#1037) Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit 4a9cef3) Co-authored-by: vamsi-amazon <[email protected]> * Add `query` function as alternate syntax to `query_string` function (opensearch-project#1010) This maintains backwards compatibility with the v1 engine. * Update DATE and TIME functions to parse string input as datetime (opensearch-project#991) Add option to accept datetime like string in both TIME and DATE (eg. accept "1999-01-02 12:12:12" for both TIME and DATE. Strict check on date for testing for valid dates (eg. Don't accept Feb 30th as a valid date) and throws a SemanticCheckException. Co-authored-by: Yury-Fridlyand <[email protected]> Signed-off-by: MitchellGale-BitQuill <[email protected]> * back quote fix (opensearch-project#1041) (opensearch-project#1050) Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit d3bb902) Co-authored-by: vamsi-amazon <[email protected]> * Catalog to Datasource changes (opensearch-project#1027) (opensearch-project#1049) Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit 3e30379) * Bump jackson to 2.14.0 (opensearch-project#1058) Signed-off-by: Joshua Li <[email protected]> (cherry picked from commit 5a1adb2) * Add valueOf() to Expression (opensearch-project#1055) Signed-off-by: Joshua Li <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: MitchellGale-BitQuill <[email protected]> Signed-off-by: Joshua Li <[email protected]> Signed-off-by: Chen Dai <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: vamsi-amazon <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MitchellGale-BitQuill <[email protected]> Co-authored-by: Joshua Li <[email protected]>

Signed-off-by: Peng Huo <[email protected]>

…ximus/microbatchtask Add micro batch streaming execution

codecov-commenter · 2022-11-16T04:16:32Z

Codecov Report

Attention: Patch coverage is 99.35897% with 2 lines in your changes missing coverage. Please review.

Project coverage is 95.74%. Comparing base (662a938) to head (7f107a4).
Report is 453 commits behind head on 2.x.

Files with missing lines	Patch %	Lines
...search/sql/opensearch/security/SecurityAccess.java	0.00%	1 Missing ⚠️
...opensearch/sql/ppl/parser/AstStatementBuilder.java	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x    #1078      +/-   ##
============================================
- Coverage     98.27%   95.74%   -2.53%     
- Complexity     3351     3435      +84     
============================================
  Files           327      354      +27     
  Lines          8457     9218     +761     
  Branches        553      663     +110     
============================================
+ Hits           8311     8826     +515     
- Misses          142      334     +192     
- Partials          4       58      +54

Flag	Coverage Δ
query-workbench	`62.76% <ø> (?)`
sql-engine	`98.28% <99.35%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Peng Huo <[email protected]>

acarbonetto · 2022-11-16T17:41:41Z

core/src/main/java/org/opensearch/sql/executor/QueryId.java

+   * @return {@link QueryId}.
+   */
+  public static QueryId queryId() {
+    return new QueryId(RandomStringUtils.random(10, true, true));


May be a good idea to call this randomQueryId or randomId or similar.

Like UUID does: https://docs.oracle.com/javase/7/docs/api/java/util/UUID.html#randomUUID()

queryid is associate with each query. user could cancel the query with queryid. are you suggesting randomId as var name?

penghuo and others added 30 commits September 15, 2022 09:29

Change ApplicationContext lifecycle to per node level (opensearch-pro…

1bd0f35

…ject#822) Signed-off-by: penghuo <[email protected]> Signed-off-by: penghuo <[email protected]>

Merge branch '2.x' into feature/maximus-m1

c137ac3

Signed-off-by: Chen Dai <[email protected]>

Add missing API method in prometheus table

3a9d217

Signed-off-by: Chen Dai <[email protected]>

Merge branch '2.x' into feature/maximus-m1

5c7b22a

Signed-off-by: Peng Huo <[email protected]>

add table default impl of exist and create

b286742

Signed-off-by: Peng Huo <[email protected]>

Merge pull request opensearch-project#979 from penghuo/hp/feature/max…

2016d5c

…imus-m1/merge Merge 2.x to feature/maximus-m1 branch

Add metadatalog interface and default in memory implementation (opens…

63f3449

…earch-project#974) Signed-off-by: Peng Huo <[email protected]>

Add Statement, QueryExecution and QueryManager (opensearch-project#845)

30fcd79

Signed-off-by: Peng Huo <[email protected]>

Add Streaming Source Impl

7b6a1e2

Signed-off-by: Peng Huo <[email protected]>

update build.gradle

3231dfe

Signed-off-by: Peng Huo <[email protected]>

Add watermark generator (opensearch-project#959)

e8f3205

* Add watermark generator Signed-off-by: Chen Dai <[email protected]> * Update Javadoc and UT for readability Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>

Add stream context and window trigger (opensearch-project#958)

73787b7

* Add stream context and window trigger Signed-off-by: Chen Dai <[email protected]> * Update Javadoc with more details Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>

change to hadoop-fs

752168f

Signed-off-by: Peng Huo <[email protected]>

exclude FileSystemStreamSource from jacoco

91ef9b3

Signed-off-by: Peng Huo <[email protected]>

exclude unnecessary depedency

5fc1eb5

Signed-off-by: Peng Huo <[email protected]>

Update integ-test depedency

c8b0118

Signed-off-by: Peng Huo <[email protected]>

Merge branch '2.x' into merge-2.x-after-release

fbd6c7a

Signed-off-by: Chen Dai <[email protected]>

Merge pull request opensearch-project#1039 from dai-chen/merge-2.x-af…

eea2689

…ter-release Merge 2.x to Maximus feature branch

Merge branch 'feature/maximus-m1' into hp/feature/maximus/streamingso…

315c39a

…urce Signed-off-by: Peng Huo <[email protected]>

Add micro batch streaming execution

40535d7

Signed-off-by: Peng Huo <[email protected]>

Merge branch 'feature/maximus-m1' into hp/feature/maximus/microbatchtask

4eadb28

Signed-off-by: Peng Huo <[email protected]>

fix compile issue

d63345b

Signed-off-by: Peng Huo <[email protected]>

add PlanContext

6aee254

Signed-off-by: Peng Huo <[email protected]>

bump jackson version

5b99c08

Signed-off-by: Peng Huo <[email protected]>

Merge branch 'feature/maximus-m1' into hp/feature/maximus/microbatchtask

75c2194

penghuo added 2 commits November 11, 2022 19:31

fix bug

faff812

Signed-off-by: Peng Huo <[email protected]>

Merge pull request opensearch-project#1044 from penghuo/hp/feature/ma…

b0cca57

…ximus/microbatchtask Add micro batch streaming execution

Merge branch '2.x' into feature/maximus/2.x

7f107a4

Signed-off-by: Peng Huo <[email protected]>

penghuo force-pushed the feature/maximus/2.x branch from 809a963 to 7f107a4 Compare November 16, 2022 05:34

penghuo marked this pull request as ready for review November 16, 2022 15:48

penghuo requested a review from a team as a code owner November 16, 2022 15:48

dai-chen approved these changes Nov 16, 2022

View reviewed changes

acarbonetto approved these changes Nov 16, 2022

View reviewed changes

penghuo merged commit 0504c71 into opensearch-project:2.x Nov 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge feature/maximus-m1 to 2.x #1078

Merge feature/maximus-m1 to 2.x #1078

penghuo commented Nov 16, 2022 •

edited

Loading

codecov-commenter commented Nov 16, 2022 •

edited by codecov bot

Loading

acarbonetto Nov 16, 2022

penghuo Nov 16, 2022

Merge feature/maximus-m1 to 2.x #1078

Merge feature/maximus-m1 to 2.x #1078

Conversation

penghuo commented Nov 16, 2022 • edited Loading

Description

Check List

codecov-commenter commented Nov 16, 2022 • edited by codecov bot Loading

Codecov Report

acarbonetto Nov 16, 2022

Choose a reason for hiding this comment

penghuo Nov 16, 2022

Choose a reason for hiding this comment

penghuo commented Nov 16, 2022 •

edited

Loading

codecov-commenter commented Nov 16, 2022 •

edited by codecov bot

Loading