Override the JVM DNS cache policy (#36570)

When a security manager is present, the JVM will cache positive hostname
lookups indefinitely. This can be problematic, especially in the modern
world with cloud services where DNS addresses can change, or
environments using Docker containers where IP addresses could be
considered ephemeral. This behavior impacts cluster discovery,
cross-cluster replication and cross-cluster search, reindex from remote,
snapshot repositories, webhooks in Watcher, external authentication
mechanisms, and the Elastic Stack Monitoring Service. The experience of
watching a DNS lookup change yet not be reflected within Elasticsearch
is a poor experience for users. The reason the JVM has this is guard
against DNS cache posioning attacks. Yet, there is already a defense in
the modern world against such attacks: TLS. With proper certificate
validation, even if a resolver falls prey to a DNS cache poisoning
attack, using TLS would neuter the attack. Therefore we have a policy
with dubious security value that significantly impacts usability. As
such we make the usability/security tradeoff towards usability, since
the security risks are very low. This commit introduces new system
properties that Elasticsearch observes to override the JVM DNS cache
policy.

distribution/src/config/jvm.options

@@ -45,6 +45,15 @@
 # 10-:-XX:+UseG1GC
 # 10-:-XX:InitiatingHeapOccupancyPercent=75
 
+## DNS cache policy
+# cache ttl in seconds for positive DNS lookups noting that this overrides the
+# JDK security property networkaddress.cache.ttl; set to -1 to cache forever
+-Des.networkaddress.cache.ttl=60
+# cache ttl in seconds for negative DNS lookups noting that this overrides the
+# JDK security property networkaddress.cache.negative ttl; set to -1 to cache
+# forever
+-Des.networkaddress.cache.negative.ttl=10
+
 ## optimizations
 
 # pre-touch memory pages used by the JVM during initialization

docs/reference/setup/sysconfig/dns-cache.asciidoc

@@ -2,17 +2,18 @@
 === DNS cache settings
 
 Elasticsearch runs with a security manager in place. With a security manager in
-place, the JVM defaults to caching positive hostname resolutions
-indefinitely. If your Elasticsearch nodes rely on DNS in an environment where
-DNS resolutions vary with time (e.g., for node-to-node discovery) then you might
-want to modify the default JVM behavior.  This can be modified by adding
+place, the JVM defaults to caching positive hostname resolutions indefinitely
+and defaults to caching negative hostname resolutions for ten
+seconds. Elasticsearch overrides this behavior with default values to cache
+positive lookups for sixty seconds, and to cache negative lookups for ten
+seconds. These values should be suitable for most environments, including
+environments where DNS resolutions vary with time. If not, you can edit the
+values `es.networkaddress.cache.ttl` and `es.networkaddress.cache.negative.ttl`
+in the <<jvm-options,JVM options>>. Note that the values
 http://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html[`networkaddress.cache.ttl=<timeout>`]
-to your
-http://docs.oracle.com/javase/8/docs/technotes/guides/security/PolicyFiles.html[Java
-security policy]. Any hosts that fail to resolve will be logged. Note also that
-with the Java security manager in place, the JVM defaults to caching negative
-hostname resolutions for ten seconds. This can be modified by adding
+and
 http://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html[`networkaddress.cache.negative.ttl=<timeout>`]
-to your
+in the
 http://docs.oracle.com/javase/8/docs/technotes/guides/security/PolicyFiles.html[Java
-security policy].
+security policy] are ignored by Elasticsearch unless you remove the settings for
+`es.networkaddress.cache.ttl` and `es.networkaddress.cache.negative.ttl`.

server/src/main/java/org/elasticsearch/bootstrap/Elasticsearch.java

@@ -36,6 +36,7 @@
 import java.io.IOException;
 import java.nio.file.Path;
 import java.security.Permission;
+import java.security.Security;
 import java.util.Arrays;
 import java.util.Locale;
 
@@ -72,13 +73,19 @@ class Elasticsearch extends EnvironmentAwareCommand {
      * Main entry point for starting elasticsearch
      */
     public static void main(final String[] args) throws Exception {
-        // we want the JVM to think there is a security manager installed so that if internal policy decisions that would be based on the
-        // presence of a security manager or lack thereof act as if there is a security manager present (e.g., DNS cache policy)
+        overrideDnsCachePolicyProperties();
+        /*
+         * We want the JVM to think there is a security manager installed so that if internal policy decisions that would be based on the
+         * presence of a security manager or lack thereof act as if there is a security manager present (e.g., DNS cache policy). This
+         * forces such policies to take effect immediately.
+         */
         System.setSecurityManager(new SecurityManager() {
+
             @Override
             public void checkPermission(Permission perm) {
                 // grant all permissions so that we can later set the security manager to the one that we want
             }
+
         });
         LogConfigurator.registerErrorListener();
         final Elasticsearch elasticsearch = new Elasticsearch();
@@ -88,6 +95,22 @@ public void checkPermission(Permission perm) {
         }
     }
 
+    private static void overrideDnsCachePolicyProperties() {
+        for (final String property : new String[] {"networkaddress.cache.ttl", "networkaddress.cache.negative.ttl" }) {
+            final String overrideProperty = "es." + property;
+            final String overrideValue = System.getProperty(overrideProperty);
+            if (overrideValue != null) {
+                try {
+                    // round-trip the property to an integer and back to a string to ensure that it parses properly
+                    Security.setProperty(property, Integer.toString(Integer.valueOf(overrideValue)));
+                } catch (final NumberFormatException e) {
+                    throw new IllegalArgumentException(
+                            "failed to parse [" + overrideProperty + "] with value [" + overrideValue + "]", e);
+                }
+            }
+        }
+    }
+
     static int main(final String[] args, final Elasticsearch elasticsearch, final Terminal terminal) throws Exception {
         return elasticsearch.main(args, terminal);
     }