Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autodetermine heap settings based on node roles and total system memory #65905

Merged
merged 39 commits into from
Dec 16, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
0098632
Autodetermine heap settings based on node roles and total system memory
mark-vieira Dec 4, 2020
045ddea
Fix spotless
mark-vieira Dec 4, 2020
c57b8a9
Fix existing packaging tests
mark-vieira Dec 5, 2020
372d7e8
More packaging test fixes
mark-vieira Dec 5, 2020
6c49f48
More fixes
mark-vieira Dec 5, 2020
afe45b7
Set heap size when starting windows service in packaging tests
mark-vieira Dec 5, 2020
bf25202
Add some better exception handling for poorly formatted config files
mark-vieira Dec 5, 2020
6ba5c66
restore heap defaults in jvm.options
rjernst Dec 9, 2020
8e6966c
Reuse findFinalOptions from ergonomics
rjernst Dec 10, 2020
e3726e7
cleanup role responses
rjernst Dec 10, 2020
2efc239
address tests
rjernst Dec 10, 2020
5daf41c
docker test
rjernst Dec 10, 2020
c6f432a
Fix up Docker test
pugnascotia Dec 10, 2020
d44c784
More fixes
pugnascotia Dec 10, 2020
1c7565c
Merge branch 'master' into HEAD
rjernst Dec 11, 2020
8ca2053
checkstyle
rjernst Dec 11, 2020
f3d025b
spotless
rjernst Dec 11, 2020
ec827f0
null guard
rjernst Dec 12, 2020
75cb18b
Add fix for earlier jdks which used InitialHeap instead of MinHeap
mark-vieira Dec 14, 2020
c36892b
Merge branch 'master' into machine-dependent-heap
elasticmachine Dec 14, 2020
88a81a1
Remove Xmx/Xms from jvm.options
rjernst Dec 14, 2020
ca7e564
Set 1g heap for packaging tests by default
rjernst Dec 15, 2020
755cfac
guard for no installation yet when setting heap to 1g
rjernst Dec 15, 2020
1c3552e
fix format oops
rjernst Dec 15, 2020
472df8f
tweak tests overriding heap
rjernst Dec 15, 2020
759e76f
Use temp config directory when appropriate
mark-vieira Dec 15, 2020
7c7e71e
Merge branch 'master' into machine-dependent-heap
elasticmachine Dec 15, 2020
f407562
More packaging test fixes
mark-vieira Dec 15, 2020
8c45424
Even more packaging test fixes for platform-specific line separators
mark-vieira Dec 15, 2020
557af6a
Fix forbidden apis
mark-vieira Dec 16, 2020
47baa29
Only create parent when required
mark-vieira Dec 16, 2020
b6032e8
No need to set default args since we do it in PackagingTestCase setup
mark-vieira Dec 16, 2020
bce1e4f
Fix docker packaging tests
mark-vieira Dec 16, 2020
f6fed4b
Ensure we add a newline so we can safely append to this file
mark-vieira Dec 16, 2020
41f7d4d
Merge branch 'master' into machine-dependent-heap
elasticmachine Dec 16, 2020
7068cd0
Fix packaging test when heap is overriden
mark-vieira Dec 16, 2020
df4ba2e
Fix checkstyle
mark-vieira Dec 16, 2020
2a53668
remove heap options before removing rpm
rjernst Dec 16, 2020
9b1cf77
Fix rpm tests
mark-vieira Dec 16, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 7 additions & 11 deletions distribution/src/config/jvm.options
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,13 @@
## IMPORTANT: JVM heap size
################################################################
##
## You must always set the initial and maximum JVM heap size to
## the same value. For example, to set the heap to 4 GB, create
## a new file in the jvm.options.d directory containing these
## lines:
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
## -Xms4g
## -Xmx4g
Expand All @@ -33,13 +36,6 @@
##
################################################################

# Xms represents the initial size of the JVM heap
# Xmx represents the maximum size of the JVM heap

-Xms${heap.min}
-Xmx${heap.max}



################################################################
## Expert settings
Expand Down
3 changes: 2 additions & 1 deletion distribution/tools/launchers/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ apply plugin: 'elasticsearch.build'

dependencies {
compileOnly project(':distribution:tools:java-version-checker')
compileOnly "org.yaml:snakeyaml:${versions.snakeyaml}"
testImplementation "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${versions.randomizedrunner}"
testImplementation "junit:junit:${versions.junit}"
testImplementation "org.hamcrest:hamcrest:${versions.hamcrest}"
Expand All @@ -44,4 +45,4 @@ tasks.named("testingConventions").configure {

["javadoc", "loggerUsageCheck", "jarHell"].each { tsk ->
tasks.named(tsk).configure { enabled = false }
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.elasticsearch.tools.launchers;

import com.sun.management.OperatingSystemMXBean;
import org.elasticsearch.tools.java_version_checker.JavaVersion;
import org.elasticsearch.tools.java_version_checker.SuppressForbidden;

import java.lang.management.ManagementFactory;

/**
* A {@link SystemMemoryInfo} which delegates to {@link OperatingSystemMXBean}.
*
* <p>Prior to JDK 14 {@link OperatingSystemMXBean} did not take into consideration container memory limits when reporting total system
* memory. Therefore attempts to use this implementation on earlier JDKs will result in an {@link SystemMemoryInfoException}.
*/
@SuppressForbidden(reason = "Using com.sun internals is the only way to query total system memory")
public final class DefaultSystemMemoryInfo implements SystemMemoryInfo {
private final OperatingSystemMXBean operatingSystemMXBean;

public DefaultSystemMemoryInfo() {
this.operatingSystemMXBean = (OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
}

@Override
@SuppressWarnings("deprecation")
public long availableSystemMemory() throws SystemMemoryInfoException {
if (JavaVersion.majorVersion(JavaVersion.CURRENT) < 14) {
throw new SystemMemoryInfoException("The minimum required Java version is 14 to use " + this.getClass().getName());
}

return operatingSystemMXBean.getTotalPhysicalMemorySize();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -19,22 +19,13 @@

package org.elasticsearch.tools.launchers;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.Stream;

/**
* Tunes Elasticsearch JVM settings based on inspection of provided JVM options.
Expand All @@ -53,9 +44,9 @@ private JvmErgonomics() {
*/
static List<String> choose(final List<String> userDefinedJvmOptions) throws InterruptedException, IOException {
final List<String> ergonomicChoices = new ArrayList<>();
final Map<String, JvmOption> finalJvmOptions = finalJvmOptions(userDefinedJvmOptions);
final long heapSize = extractHeapSize(finalJvmOptions);
final long maxDirectMemorySize = extractMaxDirectMemorySize(finalJvmOptions);
final Map<String, JvmOption> finalJvmOptions = JvmOption.findFinalOptions(userDefinedJvmOptions);
final long heapSize = JvmOption.extractMaxHeapSize(finalJvmOptions);
final long maxDirectMemorySize = JvmOption.extractMaxDirectMemorySize(finalJvmOptions);
if (maxDirectMemorySize == 0) {
ergonomicChoices.add("-XX:MaxDirectMemorySize=" + heapSize / 2);
}
Expand All @@ -78,89 +69,6 @@ static List<String> choose(final List<String> userDefinedJvmOptions) throws Inte
return ergonomicChoices;
}

private static final Pattern OPTION = Pattern.compile(
"^\\s*\\S+\\s+(?<flag>\\S+)\\s+:?=\\s+(?<value>\\S+)?\\s+\\{[^}]+?\\}\\s+\\{(?<origin>[^}]+)}"
);

private static class JvmOption {
private final String value;
private final String origin;

JvmOption(String value, String origin) {
this.value = value;
this.origin = origin;
}

public Optional<String> getValue() {
return Optional.ofNullable(value);
}

public String getMandatoryValue() {
return value;
}

public boolean isCommandLineOrigin() {
return "command line".equals(this.origin);
}
}

static Map<String, JvmOption> finalJvmOptions(final List<String> userDefinedJvmOptions) throws InterruptedException, IOException {
return flagsFinal(userDefinedJvmOptions).stream()
.map(OPTION::matcher)
.filter(Matcher::matches)
.collect(Collectors.toUnmodifiableMap(m -> m.group("flag"), m -> new JvmOption(m.group("value"), m.group("origin"))));
}

private static List<String> flagsFinal(final List<String> userDefinedJvmOptions) throws InterruptedException, IOException {
/*
* To deduce the final set of JVM options that Elasticsearch is going to start with, we start a separate Java process with the JVM
* options that we would pass on the command line. For this Java process we will add two additional flags, -XX:+PrintFlagsFinal and
* -version. This causes the Java process that we start to parse the JVM options into their final values, display them on standard
* output, print the version to standard error, and then exit. The JVM itself never bootstraps, and therefore this process is
* lightweight. By doing this, we get the JVM options parsed exactly as the JVM that we are going to execute would parse them
* without having to implement our own JVM option parsing logic.
*/
final String java = Path.of(System.getProperty("java.home"), "bin", "java").toString();
final List<String> command = Stream.of(
Stream.of(java),
userDefinedJvmOptions.stream(),
Stream.of("-Xshare:off"),
Stream.of("-XX:+PrintFlagsFinal"),
Stream.of("-version")
).reduce(Stream::concat).get().collect(Collectors.toUnmodifiableList());
final Process process = new ProcessBuilder().command(command).start();
final List<String> output = readLinesFromInputStream(process.getInputStream());
final List<String> error = readLinesFromInputStream(process.getErrorStream());
final int status = process.waitFor();
if (status != 0) {
final String message = String.format(
Locale.ROOT,
"starting java failed with [%d]\noutput:\n%s\nerror:\n%s",
status,
String.join("\n", output),
String.join("\n", error)
);
throw new RuntimeException(message);
} else {
return output;
}
}

private static List<String> readLinesFromInputStream(final InputStream is) throws IOException {
try (InputStreamReader isr = new InputStreamReader(is, StandardCharsets.UTF_8); BufferedReader br = new BufferedReader(isr)) {
return br.lines().collect(Collectors.toUnmodifiableList());
}
}

// package private for testing
static Long extractHeapSize(final Map<String, JvmOption> finalJvmOptions) {
return Long.parseLong(finalJvmOptions.get("MaxHeapSize").getMandatoryValue());
}

static long extractMaxDirectMemorySize(final Map<String, JvmOption> finalJvmOptions) {
return Long.parseLong(finalJvmOptions.get("MaxDirectMemorySize").getMandatoryValue());
}

// Tune G1GC options for heaps < 8GB
static boolean tuneG1GCForSmallHeap(final long heapSize) {
return heapSize < 8L << 30;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.elasticsearch.tools.launchers;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.Stream;

class JvmOption {
private final String value;
private final String origin;

JvmOption(String value, String origin) {
this.value = value;
this.origin = origin;
}

public Optional<String> getValue() {
return Optional.ofNullable(value);
}

public String getMandatoryValue() {
return value;
}

public boolean isCommandLineOrigin() {
return "command line".equals(this.origin);
}

private static final Pattern OPTION = Pattern.compile(
"^\\s*\\S+\\s+(?<flag>\\S+)\\s+:?=\\s+(?<value>\\S+)?\\s+\\{[^}]+?\\}\\s+\\{(?<origin>[^}]+)}"
);

public static Long extractMaxHeapSize(final Map<String, JvmOption> finalJvmOptions) {
return Long.parseLong(finalJvmOptions.get("MaxHeapSize").getMandatoryValue());
}

public static boolean isMaxHeapSpecified(final Map<String, JvmOption> finalJvmOptions) {
JvmOption maxHeapSize = finalJvmOptions.get("MaxHeapSize");
return maxHeapSize != null && maxHeapSize.isCommandLineOrigin();
}

public static boolean isMinHeapSpecified(final Map<String, JvmOption> finalJvmOptions) {
JvmOption minHeapSize = finalJvmOptions.get("MinHeapSize");
return minHeapSize != null && minHeapSize.isCommandLineOrigin();
}

public static boolean isInitialHeapSpecified(final Map<String, JvmOption> finalJvmOptions) {
JvmOption initialHeapSize = finalJvmOptions.get("InitialHeapSize");
return initialHeapSize != null && initialHeapSize.isCommandLineOrigin();
}

public static long extractMaxDirectMemorySize(final Map<String, JvmOption> finalJvmOptions) {
return Long.parseLong(finalJvmOptions.get("MaxDirectMemorySize").getMandatoryValue());
}

/**
* Determine the options present when invoking a JVM with the given user defined options.
*/
public static Map<String, JvmOption> findFinalOptions(final List<String> userDefinedJvmOptions) throws InterruptedException,
IOException {
return flagsFinal(userDefinedJvmOptions).stream()
.map(OPTION::matcher)
.filter(Matcher::matches)
.collect(Collectors.toUnmodifiableMap(m -> m.group("flag"), m -> new JvmOption(m.group("value"), m.group("origin"))));
}

private static List<String> flagsFinal(final List<String> userDefinedJvmOptions) throws InterruptedException, IOException {
/*
* To deduce the final set of JVM options that Elasticsearch is going to start with, we start a separate Java process with the JVM
* options that we would pass on the command line. For this Java process we will add two additional flags, -XX:+PrintFlagsFinal and
* -version. This causes the Java process that we start to parse the JVM options into their final values, display them on standard
* output, print the version to standard error, and then exit. The JVM itself never bootstraps, and therefore this process is
* lightweight. By doing this, we get the JVM options parsed exactly as the JVM that we are going to execute would parse them
* without having to implement our own JVM option parsing logic.
*/
final String java = Path.of(System.getProperty("java.home"), "bin", "java").toString();
final List<String> command = Stream.of(
Stream.of(java),
userDefinedJvmOptions.stream(),
Stream.of("-Xshare:off"),
Stream.of("-XX:+PrintFlagsFinal"),
Stream.of("-version")
).reduce(Stream::concat).get().collect(Collectors.toUnmodifiableList());
final Process process = new ProcessBuilder().command(command).start();
final List<String> output = readLinesFromInputStream(process.getInputStream());
final List<String> error = readLinesFromInputStream(process.getErrorStream());
final int status = process.waitFor();
if (status != 0) {
final String message = String.format(
Locale.ROOT,
"starting java failed with [%d]\noutput:\n%s\nerror:\n%s",
status,
String.join("\n", output),
String.join("\n", error)
);
throw new RuntimeException(message);
} else {
return output;
}
}

private static List<String> readLinesFromInputStream(final InputStream is) throws IOException {
try (InputStreamReader isr = new InputStreamReader(is, StandardCharsets.UTF_8); BufferedReader br = new BufferedReader(isr)) {
return br.lines().collect(Collectors.toUnmodifiableList());
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ private List<String> jvmOptions(final Path config, Path plugins, final String es
throws InterruptedException, IOException, JvmOptionsFileParserException {

final List<String> jvmOptions = readJvmOptionsFiles(config);
final MachineDependentHeap machineDependentHeap = new MachineDependentHeap(new DefaultSystemMemoryInfo());

if (esJavaOpts != null) {
jvmOptions.addAll(
Expand All @@ -142,6 +143,7 @@ private List<String> jvmOptions(final Path config, Path plugins, final String es
}

final List<String> substitutedJvmOptions = substitutePlaceholders(jvmOptions, Collections.unmodifiableMap(substitutions));
substitutedJvmOptions.addAll(machineDependentHeap.determineHeapSettings(config, substitutedJvmOptions));
final List<String> ergonomicJvmOptions = JvmErgonomics.choose(substitutedJvmOptions);
final List<String> systemJvmOptions = SystemJvmOptions.systemJvmOptions();
final List<String> bootstrapOptions = BootstrapJvmOptions.bootstrapJvmOptions(plugins);
Expand Down
Loading