Merge branch 'main' into add_concel_reason

Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]>

CHANGELOG.md

@@ -9,10 +9,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - [Remote Store] Rate limiter for remote store low priority uploads ([#14374](https://github.com/opensearch-project/OpenSearch/pull/14374/))
 - Apply the date histogram rewrite optimization to range aggregation ([#13865](https://github.com/opensearch-project/OpenSearch/pull/13865))
 - [Writable Warm] Add composite directory implementation and integrate it with FileCache ([12782](https://github.com/opensearch-project/OpenSearch/pull/12782))
+- [Workload Management] Add QueryGroup schema ([13669](https://github.com/opensearch-project/OpenSearch/pull/13669))
 - Add batching supported processor base type AbstractBatchingProcessor ([#14554](https://github.com/opensearch-project/OpenSearch/pull/14554))
 - Fix race condition while parsing derived fields from search definition ([14445](https://github.com/opensearch-project/OpenSearch/pull/14445))
 - Add allowlist setting for ingest-common and search-pipeline-common processors ([#14439](https://github.com/opensearch-project/OpenSearch/issues/14439))
-- Print reason why parent task was cancelled ([#14604](https://github.com/opensearch-project/OpenSearch/issues/14604))
+- Create SystemIndexRegistry with helper method matchesSystemIndex ([#14415](https://github.com/opensearch-project/OpenSearch/pull/14415))
+- Display reason why a parent task was cancelled ([#14604](https://github.com/opensearch-project/OpenSearch/issues/14604))
 
 ### Dependencies
 - Bump `org.gradle.test-retry` from 1.5.8 to 1.5.9 ([#13442](https://github.com/opensearch-project/OpenSearch/pull/13442))
@@ -25,10 +27,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `com.nimbusds:nimbus-jose-jwt` from 9.37.3 to 9.40 ([#14398](https://github.com/opensearch-project/OpenSearch/pull/14398))
 - Bump `org.apache.commons:commons-configuration2` from 2.10.1 to 2.11.0 ([#14399](https://github.com/opensearch-project/OpenSearch/pull/14399))
 - Bump `com.gradle.develocity` from 3.17.4 to 3.17.5 ([#14397](https://github.com/opensearch-project/OpenSearch/pull/14397))
-- Bump `opentelemetry` from 1.36.0 to 1.39.0 ([#14457](https://github.com/opensearch-project/OpenSearch/pull/14457))
-- Bump `azure-identity` from 1.11.4 to 1.13.0, Bump `msal4j` from 1.14.3 to 1.15.1, Bump `msal4j-persistence-extension` from 1.2.0 to 1.3.0 ([#14506](https://github.com/opensearch-project/OpenSearch/pull/14506))
+- Bump `opentelemetry` from 1.36.0 to 1.40.0 ([#14457](https://github.com/opensearch-project/OpenSearch/pull/14457), [#14674](https://github.com/opensearch-project/OpenSearch/pull/14674))
+- Bump `opentelemetry-semconv` from 1.25.0-alpha to 1.26.0-alpha ([#14674](https://github.com/opensearch-project/OpenSearch/pull/14674))
+- Bump `azure-identity` from 1.11.4 to 1.13.0, Bump `msal4j` from 1.14.3 to 1.15.1, Bump `msal4j-persistence-extension` from 1.2.0 to 1.3.0 ([#14506](https://github.com/opensearch-project/OpenSearch/pull/14673))
 - Bump `com.azure:azure-storage-common` from 12.21.2 to 12.25.1 ([#14517](https://github.com/opensearch-project/OpenSearch/pull/14517))
 - Bump `com.microsoft.azure:msal4j` from 1.15.1 to 1.16.0 ([#14610](https://github.com/opensearch-project/OpenSearch/pull/14610))
+- Bump `com.github.spullara.mustache.java:compiler` from 0.9.13 to 0.9.14 ([#14672](https://github.com/opensearch-project/OpenSearch/pull/14672))
+- Bump `net.minidev:accessors-smart` from 2.5.0 to 2.5.1 ([#14673](https://github.com/opensearch-project/OpenSearch/pull/14673))
+- Bump `jackson` from 2.17.1 to 2.17.2 ([#14687](https://github.com/opensearch-project/OpenSearch/pull/14687))
 
 ### Changed
 - [Tiered Caching] Move query recomputation logic outside write lock ([#14187](https://github.com/opensearch-project/OpenSearch/pull/14187))
@@ -47,6 +53,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix bug in SBP cancellation logic ([#13259](https://github.com/opensearch-project/OpenSearch/pull/13474))
 - Fix handling of Short and Byte data types in ScriptProcessor ingest pipeline ([#14379](https://github.com/opensearch-project/OpenSearch/issues/14379))
 - Switch to iterative version of WKT format parser ([#14086](https://github.com/opensearch-project/OpenSearch/pull/14086))
+- Fix match_phrase_prefix_query not working on text field with multiple values and index_prefixes ([#10959](https://github.com/opensearch-project/OpenSearch/pull/10959))
 - Fix the computed max shards of cluster to avoid int overflow ([#14155](https://github.com/opensearch-project/OpenSearch/pull/14155))
 - Fixed rest-high-level client searchTemplate & mtermVectors endpoints to have a leading slash ([#14465](https://github.com/opensearch-project/OpenSearch/pull/14465))
 - Write shard level metadata blob when snapshotting searchable snapshot indexes ([#13190](https://github.com/opensearch-project/OpenSearch/pull/13190))
@@ -57,6 +64,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix file cache initialization ([#14004](https://github.com/opensearch-project/OpenSearch/pull/14004))
 - Handle NPE in GetResult if "found" field is missing ([#14552](https://github.com/opensearch-project/OpenSearch/pull/14552))
 - Refactoring FilterPath.parse by using an iterative approach ([#14200](https://github.com/opensearch-project/OpenSearch/pull/14200))
+- Refactoring Grok.validatePatternBank by using an iterative approach ([#14206](https://github.com/opensearch-project/OpenSearch/pull/14206))
 
 ### Security
 

buildSrc/version.properties

@@ -7,8 +7,8 @@ bundled_jdk = 21.0.3+9
 # optional dependencies
 spatial4j         = 0.7
 jts               = 1.15.0
-jackson           = 2.17.1
-jackson_databind  = 2.17.1
+jackson           = 2.17.2
+jackson_databind  = 2.17.2
 snakeyaml         = 2.1
 icu4j             = 70.1
 supercsv          = 2.4.0
@@ -74,5 +74,5 @@ jzlib             = 1.1.3
 resteasy          = 6.2.4.Final
 
 # opentelemetry dependencies
-opentelemetry         = 1.39.0
-opentelemetrysemconv  = 1.25.0-alpha
+opentelemetry         = 1.40.0
+opentelemetrysemconv  = 1.26.0-alpha

client/sniffer/licenses/jackson-core-2.17.2.jar.sha1

@@ -0,0 +1 @@
+969a35cb35c86512acbadcdbbbfb044c877db814

distribution/tools/upgrade-cli/licenses/jackson-annotations-2.17.2.jar.sha1

@@ -0,0 +1 @@
+147b7b9412ffff24339f8aba080b292448e08698

distribution/tools/upgrade-cli/licenses/jackson-databind-2.17.2.jar.sha1

@@ -0,0 +1 @@
+e6deb029e5901e027c129341fac39e515066b68c

libs/core/licenses/jackson-core-2.17.2.jar.sha1

@@ -0,0 +1 @@
+969a35cb35c86512acbadcdbbbfb044c877db814

libs/grok/src/main/java/org/opensearch/grok/Grok.java

@@ -37,14 +37,18 @@
 import java.io.InputStream;
 import java.io.InputStreamReader;
 import java.nio.charset.StandardCharsets;
+import java.util.ArrayDeque;
 import java.util.ArrayList;
 import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Iterator;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
-import java.util.Stack;
+import java.util.Set;
 import java.util.function.Consumer;
 
 import org.jcodings.specific.UTF8Encoding;
@@ -86,6 +90,7 @@ public final class Grok {
         UTF8Encoding.INSTANCE,
         Syntax.DEFAULT
     );
+    private static final int MAX_PATTERN_DEPTH_SIZE = 500;
 
     private static final int MAX_TO_REGEX_ITERATIONS = 100_000; // sanity limit
 
@@ -128,7 +133,7 @@ private Grok(
             expressionBytes.length,
             Option.DEFAULT,
             UTF8Encoding.INSTANCE,
-            message -> logCallBack.accept(message)
+            logCallBack::accept
         );
 
         List<GrokCaptureConfig> captureConfig = new ArrayList<>();
@@ -144,7 +149,7 @@ private Grok(
      */
     private void validatePatternBank() {
         for (String patternName : patternBank.keySet()) {
-            validatePatternBank(patternName, new Stack<>());
+            validatePatternBank(patternName);
         }
     }
 
@@ -156,33 +161,84 @@ private void validatePatternBank() {
      * a reference to another named pattern. This method will navigate to all these named patterns and
      * check for a circular reference.
      */
-    private void validatePatternBank(String patternName, Stack<String> path) {
-        String pattern = patternBank.get(patternName);
-        boolean isSelfReference = pattern.contains("%{" + patternName + "}") || pattern.contains("%{" + patternName + ":");
-        if (isSelfReference) {
-            throwExceptionForCircularReference(patternName, pattern);
-        } else if (path.contains(patternName)) {
-            // current pattern name is already in the path, fetch its predecessor
-            String prevPatternName = path.pop();
-            String prevPattern = patternBank.get(prevPatternName);
-            throwExceptionForCircularReference(prevPatternName, prevPattern, patternName, path);
-        }
-        path.push(patternName);
-        for (int i = pattern.indexOf("%{"); i != -1; i = pattern.indexOf("%{", i + 1)) {
-            int begin = i + 2;
-            int syntaxEndIndex = pattern.indexOf('}', begin);
-            if (syntaxEndIndex == -1) {
-                throw new IllegalArgumentException("Malformed pattern [" + patternName + "][" + pattern + "]");
+    private void validatePatternBank(String initialPatternName) {
+        Deque<Frame> stack = new ArrayDeque<>();
+        Set<String> visitedPatterns = new HashSet<>();
+        Map<String, List<String>> pathMap = new HashMap<>();
+
+        List<String> initialPath = new ArrayList<>();
+        initialPath.add(initialPatternName);
+        pathMap.put(initialPatternName, initialPath);
+        stack.push(new Frame(initialPatternName, initialPath, 0));
+
+        while (!stack.isEmpty()) {
+            Frame frame = stack.peek();
+            String patternName = frame.patternName;
+            List<String> path = frame.path;
+            int startIndex = frame.startIndex;
+            String pattern = patternBank.get(patternName);
+
+            if (visitedPatterns.contains(patternName)) {
+                stack.pop();
+                continue;
+            }
+
+            visitedPatterns.add(patternName);
+            boolean foundDependency = false;
+
+            for (int i = startIndex; i < pattern.length(); i++) {
+                if (pattern.startsWith("%{", i)) {
+                    int begin = i + 2;
+                    int syntaxEndIndex = pattern.indexOf('}', begin);
+                    if (syntaxEndIndex == -1) {
+                        throw new IllegalArgumentException("Malformed pattern [" + patternName + "][" + pattern + "]");
+                    }
+
+                    int semanticNameIndex = pattern.indexOf(':', begin);
+                    int end = semanticNameIndex == -1 ? syntaxEndIndex : Math.min(syntaxEndIndex, semanticNameIndex);
+
+                    String dependsOnPattern = pattern.substring(begin, end);
+
+                    if (dependsOnPattern.equals(patternName)) {
+                        throwExceptionForCircularReference(patternName, pattern);
+                    }
+
+                    if (pathMap.containsKey(dependsOnPattern)) {
+                        throwExceptionForCircularReference(patternName, pattern, dependsOnPattern, path.subList(0, path.size() - 1));
+                    }
+
+                    List<String> newPath = new ArrayList<>(path);
+                    newPath.add(dependsOnPattern);
+                    pathMap.put(dependsOnPattern, newPath);
+
+                    stack.push(new Frame(dependsOnPattern, newPath, 0));
+                    frame.startIndex = i + 1;
+                    foundDependency = true;
+                    break;
+                }
             }
-            int semanticNameIndex = pattern.indexOf(':', begin);
-            int end = syntaxEndIndex;
-            if (semanticNameIndex != -1) {
-                end = Math.min(syntaxEndIndex, semanticNameIndex);
+
+            if (!foundDependency) {
+                pathMap.remove(patternName);
+                stack.pop();
+            }
+
+            if (stack.size() > MAX_PATTERN_DEPTH_SIZE) {
+                throw new IllegalArgumentException("Pattern references exceeded maximum depth of " + MAX_PATTERN_DEPTH_SIZE);
             }
-            String dependsOnPattern = pattern.substring(begin, end);
-            validatePatternBank(dependsOnPattern, path);
         }
-        path.pop();
+    }
+
+    private static class Frame {
+        String patternName;
+        List<String> path;
+        int startIndex;
+
+        Frame(String patternName, List<String> path, int startIndex) {
+            this.patternName = patternName;
+            this.path = path;
+            this.startIndex = startIndex;
+        }
     }
 
     private static void throwExceptionForCircularReference(String patternName, String pattern) {
@@ -192,13 +248,13 @@ private static void throwExceptionForCircularReference(String patternName, Strin
     private static void throwExceptionForCircularReference(
         String patternName,
         String pattern,
-        String originPatterName,
-        Stack<String> path
+        String originPatternName,
+        List<String> path
     ) {
         StringBuilder message = new StringBuilder("circular reference in pattern [");
         message.append(patternName).append("][").append(pattern).append("]");
-        if (originPatterName != null) {
-            message.append(" back to pattern [").append(originPatterName).append("]");
+        if (originPatternName != null) {
+            message.append(" back to pattern [").append(originPatternName).append("]");
         }
         if (path != null && path.size() > 1) {
             message.append(" via patterns [").append(String.join("=>", path)).append("]");
@@ -217,9 +273,7 @@ private String groupMatch(String name, Region region, String pattern) {
             int begin = region.getBeg(number);
             int end = region.getEnd(number);
             return new String(pattern.getBytes(StandardCharsets.UTF_8), begin, end - begin, StandardCharsets.UTF_8);
-        } catch (StringIndexOutOfBoundsException e) {
-            return null;
-        } catch (ValueException e) {
+        } catch (StringIndexOutOfBoundsException | ValueException e) {
             return null;
         }
     }

libs/grok/src/test/java/org/opensearch/grok/GrokTests.java

@@ -377,6 +377,16 @@ public void testCircularReference() {
             "circular reference in pattern [NAME5][!!!%{NAME1}!!!] back to pattern [NAME1] " + "via patterns [NAME1=>NAME2=>NAME3=>NAME4]",
             e.getMessage()
         );
+
+        e = expectThrows(IllegalArgumentException.class, () -> {
+            Map<String, String> bank = new TreeMap<>();
+            for (int i = 1; i <= 501; i++) {
+                bank.put("NAME" + i, "!!!%{NAME" + (i + 1) + "}!!!");
+            }
+            String pattern = "%{NAME1}";
+            new Grok(bank, pattern, false, logger::warn);
+        });
+        assertEquals("Pattern references exceeded maximum depth of 500", e.getMessage());
     }
 
     public void testMalformedPattern() {

libs/x-content/licenses/jackson-core-2.17.2.jar.sha1

@@ -0,0 +1 @@
+969a35cb35c86512acbadcdbbbfb044c877db814

libs/x-content/licenses/jackson-dataformat-cbor-2.17.2.jar.sha1

@@ -0,0 +1 @@
+57fa7c1b5104bbc4599278d13933a937ee058e68

libs/x-content/licenses/jackson-dataformat-smile-2.17.2.jar.sha1

@@ -0,0 +1 @@
+20e956b9b6f67138edd39fab7a506ded19638bcb

libs/x-content/licenses/jackson-dataformat-yaml-2.17.2.jar.sha1

@@ -0,0 +1 @@
+78d2c73dbec62044d7cf3b544b2e0d24a1a093b0

modules/ingest-geoip/licenses/jackson-annotations-2.17.2.jar.sha1

@@ -0,0 +1 @@
+147b7b9412ffff24339f8aba080b292448e08698

modules/ingest-geoip/licenses/jackson-databind-2.17.2.jar.sha1

@@ -0,0 +1 @@
+e6deb029e5901e027c129341fac39e515066b68c

modules/lang-mustache/build.gradle

@@ -38,7 +38,7 @@ opensearchplugin {
 }
 
 dependencies {
-  api "com.github.spullara.mustache.java:compiler:0.9.13"
+  api "com.github.spullara.mustache.java:compiler:0.9.14"
 }
 
 restResources {

modules/lang-mustache/licenses/compiler-0.9.14.jar.sha1

@@ -0,0 +1 @@
+e6df8b5aabb80d6eb6d8fef312a56d66b7659ba6

plugins/crypto-kms/licenses/jackson-annotations-2.17.2.jar.sha1

@@ -0,0 +1 @@
+147b7b9412ffff24339f8aba080b292448e08698

plugins/crypto-kms/licenses/jackson-databind-2.17.2.jar.sha1

@@ -0,0 +1 @@
+e6deb029e5901e027c129341fac39e515066b68c

plugins/discovery-ec2/licenses/jackson-annotations-2.17.2.jar.sha1

@@ -0,0 +1 @@
+147b7b9412ffff24339f8aba080b292448e08698

plugins/discovery-ec2/licenses/jackson-databind-2.17.2.jar.sha1

@@ -0,0 +1 @@
+e6deb029e5901e027c129341fac39e515066b68c

plugins/query-insights/src/main/java/org/opensearch/plugin/insights/core/service/TopQueriesService.java

@@ -138,17 +138,15 @@ public int getTopNSize() {
      * @param size the wanted top N size
      */
     public void validateTopNSize(final int size) {
-        if (size > QueryInsightsSettings.MAX_N_SIZE) {
+        if (size < 1 || size > QueryInsightsSettings.MAX_N_SIZE) {
             throw new IllegalArgumentException(
                 "Top N size setting for ["
                     + metricType
                     + "]"
-                    + " should be smaller than max top N size ["
+                    + " should be between 1 and "
                     + QueryInsightsSettings.MAX_N_SIZE
-                    + "was ("
+                    + ", was ("
                     + size
-                    + " > "
-                    + QueryInsightsSettings.MAX_N_SIZE
                     + ")"
             );
         }

plugins/query-insights/src/test/java/org/opensearch/plugin/insights/core/service/TopQueriesServiceTests.java

@@ -78,6 +78,10 @@ public void testValidateTopNSize() {
         assertThrows(IllegalArgumentException.class, () -> { topQueriesService.validateTopNSize(QueryInsightsSettings.MAX_N_SIZE + 1); });
     }
 
+    public void testValidateNegativeTopNSize() {
+        assertThrows(IllegalArgumentException.class, () -> { topQueriesService.validateTopNSize(-1); });
+    }
+
     public void testGetTopQueriesWhenNotEnabled() {
         topQueriesService.setEnabled(false);
         assertThrows(IllegalArgumentException.class, () -> { topQueriesService.getTopQueriesRecords(false); });