[DOC] Clarify Interaction of JVM Memory Settings via OPENSEARCH_JAVA_OPTS with Default Values

[c-neto]

📝 Issue Description

• Request a change to existing documentation
• Add new documentation

Add a note to clarify how JVM memory allocation settings specified via the OPENSEARCH_JAVA_OPTS environment variable interact with JVM default values in OpenSearch.

🔴 Problem Statement

When administrators configure JVM memory allocation using percentage notation (-XX:InitialRAMPercentage, -XX:MinRAMPercentage, -XX:MaxRAMPercentage) through OPENSEARCH_JAVA_OPTS, predefined default values (-Xms1g -Xmx1g) take precedence over these percentage settings. This behavior is crucial for administrators to understand as it can lead to unintended configuration errors.

For instance, if OPENSEARCH_JAVA_OPTS=-XX:MinRAMPercentage=30 -XX:MaxRAMPercentage=70 is defined, the default values for absolute memory allocation -Xms1g -Xmx1g will take precedence, rendering them ineffective.

🟢 Purpose

This PR aims to add a clear note on the important-settings page to inform administrators about this behavior preemptively, preventing potential configuration issues.

🔍 Evidences:

[2024-06-28T01:11:21,925][INFO ][o.o.n.Node               ] [4558dee99f38] JVM arguments [
- -Xms1g -Xmx1g ...
+ -Dopensearch.cgroups.hierarchy.override=/, -XX:InitialRAMPercentage=50, -XX:MinRAMPercentage=30, -XX:MaxRAMPercentage=70 ...
]

🟥 Default values.
🟩 Parameters given by OPENSEARCH_JAVA_OPTS.

Click here to see the log in original format...

$ docker run -it -e OPENSEARCH_JAVA_OPTS="-XX:InitialRAMPercentage=50 -XX:MinRAMPercentage=30 -XX:MaxRAMPercentage=70" -e OPENSEARCH_INITIAL_ADMIN_PASSWORD="0penS3@rch123654" opensearchproject/opensearch:latest

...
[2024-06-28T01:11:21,925][INFO ][o.o.n.Node               ] [4558dee99f38] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-06-28T01:11:21,925][INFO ][o.o.n.Node               ] [4558dee99f38] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5075506150552013227, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -XX:InitialRAMPercentage=50, -XX:MinRAMPercentage=30, -XX:MaxRAMPercentage=70, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-06-28T01:11:23,223][INFO ][o.o.s.s.t.SSLConfig      ] [4558dee99f38] SSL dual mode is disabled
...

[hdhalter]

Thanks, @c-neto !