Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove opensearch.sql.query.analysis.* related settings #76

Merged
merged 11 commits into from
May 20, 2021
Merged

remove opensearch.sql.query.analysis.* related settings #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
080e2a1
remove opensearch.sql.engine.new.enabled setting
penghuo May 19, 2021
0eaf9ac
Remove unnecessary @Test
penghuo May 19, 2021
5f0e674
update IT and docs
penghuo May 19, 2021
318c5a2
fix IT
penghuo May 19, 2021
94f3037
remove opensearch.sql.query.analysis.enabled
penghuo May 19, 2021
5b4e3cc
fix IT
penghuo May 19, 2021
aaef2b2
fix IT
penghuo May 19, 2021
493e81c
Fix PluginIT
penghuo May 19, 2021
e253563
Fix PluginIT
penghuo May 19, 2021
0fd2840
change to opensearch
penghuo May 19, 2021
71a84b0
remove opensearch.sql.query.analysis.* related settings
penghuo May 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 0 additions & 158 deletions docs/user/admin/settings.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,164 +121,6 @@ Result set::
}
}

opensearch.sql.query.analysis.enabled
=====================================

Description
-----------

You can disable query analyzer to bypass strict syntactic and semantic analysis.

1. The default value is true.
2. This setting is node scope.
3. This setting can be updated dynamically.


Example
-------

You can update the setting with a new value like this.

SQL query::

>> curl -H 'Content-Type: application/json' -X PUT localhost:9200/_plugins/_sql/settings -d '{
"transient" : {
"opensearch.sql.query.analysis.enabled" : "false"
}
}'

Result set::

{
"acknowledged" : true,
"persistent" : { },
"transient" : {
"opensearch" : {
"sql" : {
"query" : {
"analysis" : {
"enabled" : "false"
}
}
}
}
}
}

opensearch.sql.query.analysis.semantic.suggestion
=================================================

Description
-----------

You can enable query analyzer to suggest correct field names for quick fix.

1. The default value is false.
2. This setting is node scope.
3. This setting can be updated dynamically.


Example 1
---------

You can update the setting with a new value like this.

SQL query::

>> curl -H 'Content-Type: application/json' -X PUT localhost:9200/_plugins/_sql/settings -d '{
"transient" : {
"opensearch.sql.query.analysis.semantic.suggestion" : "true"
}
}'

Result set::

{
"acknowledged" : true,
"persistent" : { },
"transient" : {
"opensearch" : {
"sql" : {
"query" : {
"analysis" : {
"semantic" : {
"suggestion" : "true"
}
}
}
}
}
}
}

Example 2
---------

Query result after the setting updated is like:

SQL query::

>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_plugins/_sql -d '{
"query" : "SELECT first FROM accounts"
}'

Result set::

{
"error" : {
"reason" : "Invalid SQL query",
"details" : "Field [first] cannot be found or used here. Did you mean [firstname]?",
"type" : "SemanticAnalysisException"
},
"status" : 400
}

opensearch.sql.query.analysis.semantic.threshold
================================================

Description
-----------

Because query analysis needs to build semantic context in memory, index with large number of field would be skipped. You can update it to apply analysis to smaller or larger index as needed.

1. The default value is 200.
2. This setting is node scope.
3. This setting can be updated dynamically.


Example
-------

You can update the setting with a new value like this.

SQL query::

>> curl -H 'Content-Type: application/json' -X PUT localhost:9200/_plugins/_sql/settings -d '{
"transient" : {
"opensearch.sql.query.analysis.semantic.threshold" : "50"
}
}'

Result set::

{
"acknowledged" : true,
"persistent" : { },
"transient" : {
"opensearch" : {
"sql" : {
"query" : {
"analysis" : {
"semantic" : {
"threshold" : "50"
}
}
}
}
}
}
}

opensearch.sql.query.response.format
====================================

Expand Down
32 changes: 0 additions & 32 deletions integ-test/src/test/java/org/opensearch/sql/doctest/admin/PluginSettingIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,6 @@
import static org.opensearch.sql.legacy.plugin.SqlSettings.CURSOR_ENABLED;
import static org.opensearch.sql.legacy.plugin.SqlSettings.CURSOR_FETCH_SIZE;
import static org.opensearch.sql.legacy.plugin.SqlSettings.CURSOR_KEEPALIVE;
import static org.opensearch.sql.legacy.plugin.SqlSettings.QUERY_ANALYSIS_ENABLED;
import static org.opensearch.sql.legacy.plugin.SqlSettings.QUERY_ANALYSIS_SEMANTIC_SUGGESTION;
import static org.opensearch.sql.legacy.plugin.SqlSettings.QUERY_ANALYSIS_SEMANTIC_THRESHOLD;
import static org.opensearch.sql.legacy.plugin.SqlSettings.QUERY_RESPONSE_FORMAT;
import static org.opensearch.sql.legacy.plugin.SqlSettings.QUERY_SLOWLOG;
import static org.opensearch.sql.legacy.plugin.SqlSettings.SQL_ENABLED;
Expand Down Expand Up @@ -87,35 +84,6 @@ public void slowLogSetting() {
);
}

@Section(3)
public void queryAnalysisEnabledSetting() {
docSetting(
QUERY_ANALYSIS_ENABLED,
"You can disable query analyzer to bypass strict syntactic and semantic analysis.",
false
);
}

@Section(4)
public void semanticSuggestionSetting() {
docSetting(
QUERY_ANALYSIS_SEMANTIC_SUGGESTION,
"You can enable query analyzer to suggest correct field names for quick fix.",
true,
"SELECT first FROM accounts"
);
}

@Section(5)
public void semanticAnalysisThresholdSetting() {
docSetting(
QUERY_ANALYSIS_SEMANTIC_THRESHOLD,
"Because query analysis needs to build semantic context in memory, index with large number of field " +
"would be skipped. You can update it to apply analysis to smaller or larger index as needed.",
50
);
}

@Section(6)
public void responseFormatSetting() {
docSetting(
Expand Down
12 changes: 6 additions & 6 deletions integ-test/src/test/java/org/opensearch/sql/legacy/AggregationExpressionIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,11 @@ public void noGroupKeySingleFuncOverAggWithoutAliasShouldPass() {
@Test
public void noGroupKeyMaxAddMinShouldPass() {
JSONObject response = executeJdbcRequest(String.format(
"SELECT MAX(age) + MIN(age) as add " +
"SELECT MAX(age) + MIN(age) as addValue " +
"FROM %s",
Index.ACCOUNT.getName()));

verifySchema(response, schema("add", "add", "long"));
verifySchema(response, schema("MAX(age) + MIN(age)", "addValue", "long"));
verifyDataRows(response, rows(60));
}

Expand Down Expand Up @@ -111,14 +111,14 @@ public void hasGroupKeyAvgOnIntegerShouldPass() {
@Test
public void hasGroupKeyMaxAddMinShouldPass() {
JSONObject response = executeJdbcRequest(String.format(
"SELECT gender, MAX(age) + MIN(age) as add " +
"SELECT gender, MAX(age) + MIN(age) as addValue " +
"FROM %s " +
"GROUP BY gender",
Index.ACCOUNT.getName()));

verifySchema(response,
schema("gender", null, "text"),
schema("add", "add", "long"));
schema("MAX(age) + MIN(age)", "addValue", "long"));
verifyDataRows(response,
rows("m", 60),
rows("f", 60));
Expand Down Expand Up @@ -156,14 +156,14 @@ public void noGroupKeyLogMaxAddMinShouldPass() {
@Test
public void hasGroupKeyLogMaxAddMinShouldPass() {
JSONObject response = executeJdbcRequest(String.format(
"SELECT gender, Log(MAX(age) + MIN(age)) as log " +
"SELECT gender, Log(MAX(age) + MIN(age)) as logValue " +
"FROM %s " +
"GROUP BY gender",
Index.ACCOUNT.getName()));

verifySchema(response,
schema("gender", null, "text"),
schema("log", "log", "double"));
schema("Log(MAX(age) + MIN(age))", "logValue", "double"));
verifyDataRows(response,
rows("m", 4.0943445622221d),
rows("f", 4.0943445622221d));
Expand Down
2 changes: 1 addition & 1 deletion integ-test/src/test/java/org/opensearch/sql/legacy/AggregationIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ public void countTest() throws IOException {
Assert.assertThat(getIntAggregationValue(result, "COUNT(*)", "value"), equalTo(1000));
}

@Test
@Ignore("The distinct is not supported in new engine")
public void countDistinctTest() {
JSONObject response = executeJdbcRequest(
String.format("SELECT COUNT(distinct gender) FROM %s", TEST_INDEX_ACCOUNT));
Expand Down
28 changes: 6 additions & 22 deletions integ-test/src/test/java/org/opensearch/sql/legacy/PluginIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,17 +237,16 @@ public void sqlPersistentOnlySettingTest() throws IOException {

/**
* Both transient and persistent settings are applied for same settings.
* This is similiar to _cluster/settings behavior
* This is similar to _cluster/settings behavior
*/
@Test
public void sqlCombinedSettingTest() throws IOException {
String settings = "{" +
" \"transient\": {" +
" \"opensearch.sql.metrics.rollingwindow\": \"3700\"," +
" \"opensearch.sql.query.analysis.semantic.suggestion\" : \"false\"" +
" \"opensearch.sql.metrics.rollingwindow\": \"3700\"" +
" }," +
" \"persistent\": {" +
" \"opensearch.sql.query.analysis.semantic.suggestion\" : \"true\"" +
" \"opensearch.sql.query.slowlog\" : \"2\"" +
" }" +
"}";
JSONObject actual = updateViaSQLSettingsAPI(settings);
Expand All @@ -257,11 +256,7 @@ public void sqlCombinedSettingTest() throws IOException {
" \"opensearch\" : {" +
" \"sql\" : {" +
" \"query\" : {" +
" \"analysis\" : {" +
" \"semantic\" : {" +
" \"suggestion\" : \"true\"" +
" }" +
" }" +
" \"slowlog\" : \"2\"" +
" }" +
" }" +
" }" +
Expand All @@ -271,13 +266,6 @@ public void sqlCombinedSettingTest() throws IOException {
" \"sql\" : {" +
" \"metrics\" : {" +
" \"rollingwindow\" : \"3700\"" +
" }," +
" \"query\" : {" +
" \"analysis\" : {" +
" \"semantic\" : {" +
" \"suggestion\" : \"false\"" +
" }" +
" }" +
" }" +
" }" +
" }" +
Expand All @@ -300,7 +288,7 @@ public void ignoreNonSQLSettingsTest() throws IOException {
" \"search.max_keep_alive\": \"24h\"" +
" }," +
" \"persistent\": {" +
" \"opensearch.sql.query.analysis.semantic.suggestion\": \"true\"," +
" \"opensearch.sql.query.slowlog\": \"2\"," +
" \"opensearch.alerting.metrics.rollingwindow\": \"3700\"," +
" \"thread_pool.analyze.queue_size\": \"16\"" +
" }" +
Expand All @@ -312,11 +300,7 @@ public void ignoreNonSQLSettingsTest() throws IOException {
" \"opensearch\" : {" +
" \"sql\" : {" +
" \"query\" : {" +
" \"analysis\" : {" +
" \"semantic\" : {" +
" \"suggestion\" : \"true\"" +
" }" +
" }" +
" \"slowlog\" : \"2\"" +
" }" +
" }" +
" }" +
Expand Down
Loading