[2.x][Manual Backport ] Back port main to 2.x (#7477)

* Bump OUI to 1.8.0 (#7363) Signed-off-by: Miki <[email protected]> (cherry picked from commit c1bae75) * [Auto Suggest] OpenSearch SQL autosuggest with ANTLR (#7336) * cherry pick Signed-off-by: Eric <[email protected]> * SQL keyword suggestion Signed-off-by: Eric <[email protected]> * column and values suggestions Signed-off-by: Eric <[email protected]> * adjust values suggestions and ignored token Signed-off-by: Eric <[email protected]> * suggestion type casting Signed-off-by: Eric <[email protected]> * ignore functions Signed-off-by: Eric <[email protected]> * case insensitive Signed-off-by: Eric <[email protected]> * change to use grammar for insensitive case Signed-off-by: Eric <[email protected]> * fix multi-line issue Signed-off-by: Eric <[email protected]> * code cleanup Signed-off-by: Eric <[email protected]> * move ppl out of scope Signed-off-by: Eric <[email protected]> * rename folder and add ignoring rules Signed-off-by: Eric <[email protected]> * resolve type issue Signed-off-by: Eric <[email protected]> * fix ppl suggestion provider issue Signed-off-by: Eric <[email protected]> * remove function suggestion Signed-off-by: Eric <[email protected]> * some code clean up and adding tests Signed-off-by: Eric <[email protected]> * cursor tests Signed-off-by: Eric <[email protected]> * remove testing setup Signed-off-by: Eric <[email protected]> * add changelog Signed-off-by: Eric <[email protected]> * update yarn file Signed-off-by: Eric <[email protected]> * add missing testing library Signed-off-by: Eric <[email protected]> * MDS integration Signed-off-by: Eric <[email protected]> * minor interface change and disable word based suggestion Signed-off-by: Eric <[email protected]> * update ID_LITERAL and recompile grammar Signed-off-by: Eric <[email protected]> * column suggest for agg function Signed-off-by: Eric <[email protected]> * revert version Signed-off-by: Eric <[email protected]> * opensearch sql syntax highlighting Signed-off-by: Eric <[email protected]> * add utility tests Signed-off-by: Eric <[email protected]> * symbol table test Signed-off-by: Eric <[email protected]> * error listener test Signed-off-by: Eric <[email protected]> * parse test Signed-off-by: Eric <[email protected]> * add suggestion provider to single line query editor Signed-off-by: Eric <[email protected]> * remove one test Signed-off-by: Eric <[email protected]> * add connection service type Signed-off-by: Eric <[email protected]> --------- Signed-off-by: Eric <[email protected]> (cherry picked from commit 9348bd4) * Move TopNavLinks to new Nav Bar Discover (#7326) This PR aims to introduce a new Nav bar in Discover that would contain the Date Picker as well as the Navigation links that are currently part of headers. The Navigation Links have been replaced with the corresponding Icons to provide a modern UX. These setting are currently controlled by query:enhancements:enabled Advanced Setting Flag and would be visible when the above flag is turned on. Signed-off-by: Suchit Sahoo <[email protected]> (cherry picked from commit e46e54a) * [navigation-next]fix: breadcrumb issue found in data source management and other similar cases. (#7401) * feat: add scoped breadcrumbs Signed-off-by: SuZhou-Joe <[email protected]> * Changeset file for PR #7401 created/updated * feat: move data source management register logic up a little bit Signed-off-by: SuZhou-Joe <[email protected]> * feat: add unit test Signed-off-by: SuZhou-Joe <[email protected]> * feat: revert the home related change Signed-off-by: SuZhou-Joe <[email protected]> * fix: overview error in all use case when workspace is enabled Signed-off-by: SuZhou-Joe <[email protected]> * fix: update snapshot Signed-off-by: SuZhou-Joe <[email protected]> * Changeset file for PR #7401 created/updated * fix: hide nav groups that should be displayed Signed-off-by: SuZhou-Joe <[email protected]> * feat: hide expand icon in left navigation Signed-off-by: SuZhou-Joe <[email protected]> * feat: update Signed-off-by: SuZhou-Joe <[email protected]> --------- Signed-off-by: SuZhou-Joe <[email protected]> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> Co-authored-by: ZilongX <[email protected]> (cherry picked from commit 3f78c94) * [Bug][Data Source] Move data source manageable feature flag to DSM plugin (#7440) * Move data source manageable feature flag to DSM plugin Signed-off-by: yubonluo <[email protected]> * Changeset file for PR #7440 created/updated --------- Signed-off-by: yubonluo <[email protected]> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit dfcd2e1) * [Discover-next] data set picker (#7426) * update using query manager Signed-off-by: Kawika Avilla <[email protected]> * aggs not working Signed-off-by: Kawika Avilla <[email protected]> * almost there Signed-off-by: Kawika Avilla <[email protected]> * stablish Signed-off-by: Kawika Avilla <[email protected]> * thanks ashwin Signed-off-by: Kawika Avilla <[email protected]> * update ref name Signed-off-by: Kawika Avilla <[email protected]> * fix timefields Signed-off-by: Kawika Avilla <[email protected]> * clean up some console logs Signed-off-by: Kawika Avilla <[email protected]> * go safer route of setting language Signed-off-by: Kawika Avilla <[email protected]> * its working again Signed-off-by: Kawika Avilla <[email protected]> * fix names Signed-off-by: Kawika Avilla <[email protected]> * restore code editor and indices Signed-off-by: Kawika Avilla <[email protected]> * sql df Signed-off-by: Kawika Avilla <[email protected]> * fix external datasources again Signed-off-by: Kawika Avilla <[email protected]> --------- Signed-off-by: Kawika Avilla <[email protected]> (cherry picked from commit 6a079d3) * Fix add filter for numeric scripted field (#7022) * Fix add filter for scripted field Signed-off-by: abbyhu2000 <[email protected]> * add unit test Signed-off-by: abbyhu2000 <[email protected]> * change from nested ternary to if else Signed-off-by: abbyhu2000 <[email protected]> * add more unit tests Signed-off-by: abbyhu2000 <[email protected]> * change unit tests Signed-off-by: abbyhu2000 <[email protected]> * fix typo Signed-off-by: abbyhu2000 <[email protected]> --------- Signed-off-by: abbyhu2000 <[email protected]> (cherry picked from commit 7c1f8fc) * [Bug][Workspace] Add permission validation at workspace detail page (#7435) * [Bug][Workspace] Add permission validate at workspace detail page Signed-off-by: yubonluo <[email protected]> * Changeset file for PR #7435 created/updated --------- Signed-off-by: yubonluo <[email protected]> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit 45d34b5) * Revert "[Discover-next] data set picker (#7426)" (#7479) This reverts commit 6a079d3. --------- Co-authored-by: Miki <[email protected]> Co-authored-by: Eric Wei <[email protected]> Co-authored-by: SuZhou-Joe <[email protected]> Co-authored-by: yuboluo <[email protected]> Co-authored-by: Kawika Avilla <[email protected]> Co-authored-by: Qingyang(Abby) Hu <[email protected]> Co-authored-by: Sean Li <[email protected]> (cherry picked from commit 0515fb2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-project · Jul 25, 2024 · 4a3bf35 · 4a3bf35
1 parent 95b2ef6
commit 4a3bf35
Show file tree

Hide file tree

Showing 76 changed files with 44,032 additions and 115 deletions.
diff --git a/.eslintignore b/.eslintignore
@@ -30,3 +30,7 @@ target
 /packages/osd-test/src/functional_test_runner/lib/config/__tests__/fixtures/
 /packages/osd-ui-framework/dist
 /packages/osd-ui-shared-deps/flot_charts
+
+# antlr overrides
+/src/plugins/data/public/antlr/opensearch_sql/.generated/*
+/src/plugins/data/public/antlr/opensearch_sql/grammar/**/*
diff --git a/.eslintrc.js b/.eslintrc.js
@@ -751,5 +751,14 @@ module.exports = {
         'no-undef': 'off',
       },
     },
+    {
+      files: [
+        'src/plugins/data/public/antlr/opensearch_sql/.generated/*',
+        'src/plugins/data/public/antlr/opensearch_sql/grammar/**/*',
+      ],
+      rules: {
+        'filenames/match-regex': 'off',
+      },
+    },
   ],
 };
diff --git a/changelogs/fragments/7336.yml b/changelogs/fragments/7336.yml
@@ -0,0 +1,2 @@
+feat:
+- [Auto Suggest] OpenSearch SQL autosuggest with ANTLR ([#7336](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7336))
diff --git a/changelogs/fragments/7363.yml b/changelogs/fragments/7363.yml
@@ -0,0 +1,2 @@
+feat:
+- Bump OUI to 1.8.0 ([#7363](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7363))
diff --git a/config/opensearch_dashboards.yml b/config/opensearch_dashboards.yml
@@ -362,4 +362,4 @@
 # Set the backend roles in groups or users, whoever has the backend roles or exactly match the user ids defined in this config will be regard as dashboard admin.
 # Dashboard admin will have the access to all the workspaces(workspace.enabled: true) and objects inside OpenSearch Dashboards.
 # opensearchDashboards.dashboardAdmin.groups: ["dashboard_admin"]
-# opensearchDashboards.dashboardAdmin.users: ["dashboard_admin"]
+# opensearchDashboards.dashboardAdmin.users: ["dashboard_admin"]
diff --git a/package.json b/package.json
@@ -77,7 +77,12 @@
     "osd:bootstrap": "scripts/use_node scripts/build_ts_refs && scripts/use_node scripts/register_git_hook",
     "spec_to_console": "scripts/use_node scripts/spec_to_console",
     "pkg-version": "scripts/use_node -e \"console.log(require('./package.json').version)\"",
-    "release_note:generate": "scripts/use_node scripts/generate_release_note"
+    "release_note:generate": "scripts/use_node scripts/generate_release_note",
+    "cypress:run-without-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=false",
+    "cypress:run-with-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=true,openSearchUrl=https://localhost:9200,WAIT_FOR_LOADER_BUFFER_MS=500",
+    "osd:ciGroup10": "echo \"dashboard_sanity_test_spec.js\"",
+    "osd:ciGroup11": "echo \"apps/vis_builder/*.js\"",
+    "generate:opensearchsqlantlr": "./node_modules/antlr4ng-cli/index.js -Dlanguage=TypeScript -o ./src/plugins/data/public/antlr/opensearch_sql/.generated -visitor -no-listener -Xexact-output-dir ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLLexer.g4 ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLParser.g4"
   },
   "repository": {
     "type": "git",
@@ -109,7 +114,7 @@
     "**/qs": "^6.11.0",
     "**/semver": "^7.5.3",
     "**/set-value": "^4.1.0",
-    "**/tar":"^6.2.1",
+    "**/tar": "^6.2.1",
     "**/topo/hoek": "npm:@amoo-miki/[email protected]",
     "**/trim": "^0.0.3",
     "**/typescript": "4.0.2",
@@ -140,7 +145,7 @@
   "dependencies": {
     "@aws-crypto/client-node": "^3.1.1",
     "@elastic/datemath": "5.0.3",
-    "@elastic/eui": "npm:@opensearch-project/oui@1.7.0",
+    "@elastic/eui": "npm:@opensearch-project/oui@1.8.0",
     "@elastic/good": "^9.0.1-kibana3",
     "@elastic/numeral": "npm:@amoo-miki/[email protected]",
     "@elastic/request-crypto": "2.0.0",
@@ -177,6 +182,8 @@
     "JSONStream": "1.3.5",
     "abortcontroller-polyfill": "^1.4.0",
     "ajv": "^8.11.0",
+    "antlr4-c3": "^3.4.1",
+    "antlr4ng": "^3.0.4",
     "bluebird": "3.5.5",
     "chalk": "^4.1.0",
     "chokidar": "^3.4.2",
@@ -242,11 +249,13 @@
     "type-detect": "^4.0.8",
     "uuid": "3.3.2",
     "whatwg-fetch": "^3.0.0",
-    "yauzl": "^2.10.0"
+    "yauzl": "^2.10.0",
+    "@opensearch-dashboards-test/opensearch-dashboards-test-library": "https://github.com/opensearch-project/opensearch-dashboards-test-library/archive/refs/tags/1.0.6.tar.gz"
   },
   "devDependencies": {
     "@babel/core": "^7.22.9",
     "@babel/parser": "^7.22.9",
+    "@babel/plugin-transform-class-static-block": "^7.24.4",
     "@babel/register": "^7.22.9",
     "@babel/types": "^7.22.9",
     "@elastic/apm-rum": "^5.6.1",
@@ -361,6 +370,7 @@
     "@types/zen-observable": "^0.8.0",
     "@typescript-eslint/eslint-plugin": "^3.10.0",
     "@typescript-eslint/parser": "^3.10.0",
+    "antlr4ng-cli": "^2.0.0",
     "archiver": "^5.3.0",
     "axe-core": "^4.0.2",
     "babel-eslint": "^10.0.3",
@@ -461,6 +471,7 @@
     "regenerate": "^1.4.0",
     "reselect": "^4.0.0",
     "resize-observer-polyfill": "^1.5.1",
+    "rimraf": "^5.0.7",
     "selenium-webdriver": "^4.0.0-alpha.7",
     "simple-git": "^3.16.0",
     "sinon": "^7.4.2",

diff --git a/packages/osd-babel-preset/webpack_preset.js b/packages/osd-babel-preset/webpack_preset.js
@@ -55,6 +55,12 @@ module.exports = () => {
           fileName: false,
         },
       ],
+      [
+        require.resolve('@babel/plugin-transform-class-static-block'),
+        {
+          fileName: false,
+        },
+      ],
     ],
     env: {
       production: {

diff --git a/packages/osd-monaco/src/xjson/lexer_rules/index.ts b/packages/osd-monaco/src/xjson/lexer_rules/index.ts
@@ -33,6 +33,7 @@ import * as monaco from 'monaco-editor/esm/vs/editor/editor.api';
 import * as xJson from './xjson';
 import * as opensearchql from './opensearchql';
 import * as painless from './painless';
+import * as opensearchsql from './opensearchsql';
 
 export const registerLexerRules = (m: typeof monaco) => {
   m.languages.register({ id: xJson.ID });
@@ -41,4 +42,6 @@ export const registerLexerRules = (m: typeof monaco) => {
   m.languages.setMonarchTokensProvider(painless.ID, painless.lexerRules);
   m.languages.register({ id: opensearchql.ID });
   m.languages.setMonarchTokensProvider(opensearchql.ID, opensearchql.lexerRules);
+  m.languages.register({ id: opensearchsql.ID });
+  m.languages.setMonarchTokensProvider(opensearchsql.ID, opensearchsql.lexerRules);
 };
diff --git a/packages/osd-monaco/src/xjson/lexer_rules/opensearchsql.ts b/packages/osd-monaco/src/xjson/lexer_rules/opensearchsql.ts
@@ -0,0 +1,157 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+import { monaco } from '../../monaco';
+
+export const ID = 'SQL';
+
+const keywords = [
+  'ALL',
+  'AND',
+  'AS',
+  'ASC',
+  'BOOLEAN',
+  'BETWEEN',
+  'BY',
+  'CASE',
+  'CAST',
+  'CROSS',
+  'COLUMNS',
+  'DATETIME',
+  'DELETE',
+  'DESC',
+  'DESCRIBE',
+  'DISTINCT',
+  'DOUBLE',
+  'ELSE',
+  'EXISTS',
+  'FALSE',
+  'FLOAT',
+  'FIRST',
+  'FROM',
+  'GROUP',
+  'HAVING',
+  'IN',
+  'INNER',
+  'INT',
+  'INTEGER',
+  'IS',
+  'JOIN',
+  'LAST',
+  'LEFT',
+  'LIKE',
+  'LIMIT',
+  'LONG',
+  'MATCH',
+  'NATURAL',
+  'NOT',
+  'NULL',
+  'NULLS',
+  'ON',
+  'OR',
+  'ORDER',
+  'OUTER',
+  'OVER',
+  'PARTITION',
+  'REGEXP',
+  'RIGHT',
+  'SELECT',
+  'SHOW',
+  'STRING',
+  'THEN',
+  'TRUE',
+  'UNION',
+  'USING',
+  'WHEN',
+  'WHERE',
+  'EXCEPT',
+];
+
+const functions = [
+  'AVG',
+  'COUNT',
+  'MAX',
+  'MIN',
+  'SUM',
+  'VAR_POP',
+  'VAR_SAMP',
+  'VARIANCE',
+  'STD',
+  'STDDEV',
+  'STDDEV_POP',
+  'STDDEV_SAMP',
+  'SUBSTRING',
+  'TRIM',
+];
+
+const operators = [
+  '=',
+  '>',
+  '<',
+  '!',
+  '~',
+  '\\|',
+  '&',
+  '\\^',
+  '\\*',
+  '/',
+  '%',
+  '\\+',
+  '-',
+  'DIV',
+  'MOD',
+];
+
+const brackets = [
+  { open: '(', close: ')', token: 'delimiter.parenthesis' },
+  { open: '[', close: ']', token: 'delimiter.square' },
+];
+
+export const lexerRules = {
+  defaultToken: 'invalid',
+  ignoreCase: true,
+  tokenPostfix: '',
+  keywords,
+  functions,
+  operators,
+  brackets,
+  tokenizer: {
+    root: [
+      [
+        /[a-zA-Z_$][a-zA-Z0-9_$]*/,
+        {
+          cases: {
+            '@keywords': 'keyword',
+            '@functions': 'function',
+            '@default': 'identifier',
+          },
+        },
+      ],
+      { include: '@whitespace' },
+      [/[()]/, '@brackets'],
+      [new RegExp(operators.join('|')), 'operator'],
+      [/[0-9]+(\.[0-9]+)?/, 'number'],
+      [/'([^'\\]|\\.)*$/, 'string.invalid'], // non-terminated string
+      [/'/, 'string', '@string'],
+      [/"/, 'string', '@string'],
+    ],
+    whitespace: [
+      [/[ \t\r\n]+/, 'white'],
+      [/\/\*/, 'comment', '@comment'],
+      [/--.*$/, 'comment'],
+    ],
+    string: [
+      [/[^'\\]+/, 'string'],
+      [/\\./, 'string.escape'],
+      [/'/, 'string', '@pop'],
+      [/"/, 'string', '@pop'],
+    ],
+    comment: [
+      [/[^/*]+/, 'comment'],
+      [/\*\//, 'comment', '@pop'],
+      [/[\/*]/, 'comment'],
+    ],
+  },
+} as monaco.languages.IMonarchLanguage;
diff --git a/packages/osd-optimizer/src/worker/webpack.config.ts b/packages/osd-optimizer/src/worker/webpack.config.ts
@@ -256,6 +256,16 @@ export function getWebpackConfig(bundle: Bundle, bundleRefs: BundleRefs, worker:
             loader: 'raw-loader',
           },
         },
+        {
+          test: /\.cjs$/,
+          include: /node_modules/,
+          use: {
+            loader: 'babel-loader',
+            options: {
+              presets: [BABEL_PRESET_PATH],
+            },
+          },
+        },
       ],
     },
 

diff --git a/packages/osd-ui-framework/package.json b/packages/osd-ui-framework/package.json
@@ -23,7 +23,7 @@
     "enzyme-adapter-react-16": "^1.9.1"
   },
   "devDependencies": {
-    "@elastic/eui": "npm:@opensearch-project/oui@1.7.0",
+    "@elastic/eui": "npm:@opensearch-project/oui@1.8.0",
     "@osd/babel-preset": "1.0.0",
     "@osd/optimizer": "1.0.0",
     "comment-stripper": "^0.0.4",

diff --git a/packages/osd-ui-shared-deps/package.json b/packages/osd-ui-shared-deps/package.json
@@ -10,7 +10,7 @@
   },
   "dependencies": {
     "@elastic/charts": "31.1.0",
-    "@elastic/eui": "npm:@opensearch-project/oui@1.7.0",
+    "@elastic/eui": "npm:@opensearch-project/oui@1.8.0",
     "@elastic/numeral": "npm:@amoo-miki/[email protected]",
     "@opensearch/datemath": "5.0.3",
     "@osd/i18n": "1.0.0",

diff --git a/src/core/public/application/types.ts b/src/core/public/application/types.ts
@@ -539,6 +539,8 @@ export interface AppMountParameters<HistoryLocationState = unknown> {
    * Optional datasource id to pass while mounting app
    */
   dataSourceId?: string;
+
+  optionalRef?: Record<string, React.RefObject<HTMLDivElement>>;
 }
 
 /**
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		feat:
		- [Auto Suggest] OpenSearch SQL autosuggest with ANTLR ([#7336](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7336))
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		feat:
		- Bump OUI to 1.8.0 ([#7363](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7363))
-Original file line number
+Diff line change
@@ Expand Up @@
        * Optional datasource id to pass while mounting app
        */
       dataSourceId?: string;
+      optionalRef?: Record<string, React.RefObject<HTMLDivElement>>;
     }
     /**
@@ Expand Down @@