Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SOLR-15960 Unified use of system properties and environment variables #1935

Merged
merged 43 commits into from
Jan 5, 2024
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
1b420f5
SOLR-15960: Unified use of system properties and environment variables
janhoy Sep 17, 2023
a26cf6d
SOLR-15960: Simplify env or sysprop code in ModuleUtils and SolrCli
janhoy Sep 17, 2023
2d12513
SOLR-15960: Put mapping in external file
janhoy Sep 17, 2023
3d94a14
SOLR-15960: Precommit fix forbidden API
janhoy Sep 17, 2023
f6630e2
SOLR-15960: Simplify some methods
janhoy Sep 17, 2023
e3ed389
SOLR-15960: Tidy, shellcheck
janhoy Sep 17, 2023
11bd546
SOLR-15960: Fix env list test
janhoy Sep 17, 2023
3b33541
SOLR-15960: Disable SSL env.vars
janhoy Sep 17, 2023
b598c85
Update solr/core/src/java/org/apache/solr/util/EnvUtils.java
janhoy Sep 19, 2023
ef91e63
SOLR-15960: Use StrUtils.parseBool
janhoy Sep 19, 2023
2082a41
SOLR-15960: Remove unneeded stream/trim in ModuleUtils
janhoy Sep 19, 2023
087a0a3
SOLR-15960: Typo in javadoc
janhoy Sep 19, 2023
ee5765f
SOLR-15960: More mappings and exclusions
janhoy Sep 19, 2023
2584aa3
Add more env mappings for solr.xml and solrconfig.xml
janhoy Sep 19, 2023
2f4818a
Make sure EnvUtils is initialized before allowing property getters
janhoy Sep 19, 2023
f08a6c7
Replace some static property initialization with EnvUtils calls
janhoy Sep 19, 2023
5d1dd98
SOLR-15960: Sort
janhoy Sep 22, 2023
3cf0430
SOLR-15960: Add back SSL mappings
janhoy Sep 22, 2023
8f5030c
SOLR-15960: Ref guide about variable to property option
janhoy Sep 22, 2023
7482222
SOLR-15960: Example in solr.in.sh
janhoy Sep 22, 2023
98c4eb2
SOLR-15960: Add more mappings, export some irregular vars
janhoy Sep 22, 2023
ea976b3
SOLR-15960: Always convert vars in mappings file
janhoy Sep 22, 2023
dca2ae4
SOLR-15960: Support fooBar equal to foo.bar for getProp
janhoy Sep 22, 2023
4d54119
SOLR-15960: setProp("camelCase") will set both 'camelCase' and 'camel…
janhoy Sep 22, 2023
d224aa0
Merge branch 'main' into SOLR-15960-env-sysprop
janhoy Sep 23, 2023
1d122aa
Merge branch 'main' into SOLR-15960-env-sysprop
janhoy Dec 9, 2023
8b3a017
SOLR-15960: Enumerate entrySet instead of getters
janhoy Dec 9, 2023
589bf34
SOLR-15960: Make init() non public and update javadoc
janhoy Dec 9, 2023
92745ca
SOLR-15960: Use Set.copyOf()
janhoy Dec 9, 2023
3ce6c82
SOLR-15960: Add test for splitting a backslash string.
janhoy Dec 9, 2023
2c74052
SOLR-15960: Cache of camelCase -> dots
janhoy Dec 9, 2023
28400bb
SOLR-15960: Remove explicit var conversion of global circuit breaker …
janhoy Dec 9, 2023
552cd68
SOLR-15960: Prevent deadlock
janhoy Dec 9, 2023
d5a4f55
SOLR-15960: Separate non-blocking method for internal use. All public…
janhoy Dec 11, 2023
6fb6bc4
SOLR-15960: Document that string splitting respects escaped charts
janhoy Dec 11, 2023
1a4f080
SOLR-15960: Fix javadoc
janhoy Dec 11, 2023
c16004d
SOLR-15960: Fix javadoc
janhoy Dec 11, 2023
c229b9d
SOLR-15960: Exclude SOLR_SSL_* from property mappings
janhoy Dec 23, 2023
e992b28
Merge branch 'main' into SOLR-15960-env-sysprop
janhoy Dec 25, 2023
0089ff3
SOLR-15960: Experiment: Disable locking
janhoy Dec 25, 2023
7022e85
SOLR-15960: Remove init synchronization
janhoy Dec 26, 2023
f2932e0
Merge branch 'main' into SOLR-15960-env-sysprop
janhoy Jan 4, 2024
45071a3
SOLR-15960: Changes and major-changes
janhoy Jan 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 8 additions & 12 deletions solr/bin/solr
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,13 @@ elif [ -r "$SOLR_INCLUDE" ]; then
. "$SOLR_INCLUDE"
fi

# Export variables we want to make visible to Solr sub-process
for var in $(compgen -e); do
janhoy marked this conversation as resolved.
Show resolved Hide resolved
if [[ "$var" =~ ^(SOLR_.*|DEFAULT_CONFDIR|ZK_.*|GCS_BUCKET|GCS_.*|S3_.*|OTEL_.*|AWS_.*)$ ]]; then
janhoy marked this conversation as resolved.
Show resolved Hide resolved
export "${var?}"
fi
done

# if pid dir is unset, default to $solr_tip/bin
: "${SOLR_PID_DIR:=$SOLR_TIP/bin}"

Expand Down Expand Up @@ -1345,12 +1352,10 @@ if [ $# -gt 0 ]; then
;;
-v)
SOLR_LOG_LEVEL=DEBUG
PASS_TO_RUN_EXAMPLE+=("-Dsolr.log.level=$SOLR_LOG_LEVEL")
janhoy marked this conversation as resolved.
Show resolved Hide resolved
shift
;;
-q)
SOLR_LOG_LEVEL=WARN
PASS_TO_RUN_EXAMPLE+=("-Dsolr.log.level=$SOLR_LOG_LEVEL")
shift
;;
-all)
Expand Down Expand Up @@ -1384,15 +1389,6 @@ if [ $# -gt 0 ]; then
done
fi

if [[ -n ${SOLR_LOG_LEVEL:-} ]] ; then
SOLR_LOG_LEVEL_OPT="-Dsolr.log.level=$SOLR_LOG_LEVEL"
fi

# Solr modules option
if [[ -n "${SOLR_MODULES:-}" ]] ; then
SCRIPT_SOLR_OPTS+=("-Dsolr.modules=$SOLR_MODULES")
fi

# Default placement plugin
if [[ -n "${SOLR_PLACEMENTPLUGIN_DEFAULT:-}" ]] ; then
SCRIPT_SOLR_OPTS+=("-Dsolr.placementplugin.default=$SOLR_PLACEMENTPLUGIN_DEFAULT")
Expand Down Expand Up @@ -1883,7 +1879,7 @@ function start_solr() {
fi

SOLR_START_OPTS=('-server' "${JAVA_MEM_OPTS[@]}" "${GC_TUNE_ARR[@]}" "${GC_LOG_OPTS[@]}" "${IP_ACL_OPTS[@]}" \
"${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" ${SOLR_LOG_LEVEL_OPT:-} -Dsolr.log.dir="$SOLR_LOGS_DIR" \
"${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" -Dsolr.log.dir="$SOLR_LOGS_DIR" \
"-Djetty.port=$SOLR_PORT" "-DSTOP.PORT=$stop_port" "-DSTOP.KEY=$STOP_KEY" \
# '-OmitStackTraceInFastThrow' ensures stack traces in errors,
# users who don't care about useful error msgs can override in SOLR_OPTS with +OmitStackTraceInFastThrow
Expand Down
8 changes: 0 additions & 8 deletions solr/bin/solr.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -559,13 +559,11 @@ goto parse_args

:set_debug
set SOLR_LOG_LEVEL=DEBUG
set "PASS_TO_RUN_EXAMPLE=!PASS_TO_RUN_EXAMPLE! -Dsolr.log.level=%SOLR_LOG_LEVEL%"
SHIFT
goto parse_args

:set_warn
set SOLR_LOG_LEVEL=WARN
set "PASS_TO_RUN_EXAMPLE=!PASS_TO_RUN_EXAMPLE! -Dsolr.log.level=%SOLR_LOG_LEVEL%"
SHIFT
goto parse_args

Expand Down Expand Up @@ -807,11 +805,6 @@ IF NOT "%SOLR_HOST%"=="" (

set SCRIPT_SOLR_OPTS=

REM Solr modules option
IF DEFINED SOLR_MODULES (
set "SCRIPT_SOLR_OPTS=%SCRIPT_SOLR_OPTS% -Dsolr.modules=%SOLR_MODULES%"
)

REM Default placement plugin
IF DEFINED SOLR_PLACEMENTPLUGIN_DEFAULT (
set "SCRIPT_SOLR_OPTS=%SCRIPT_SOLR_OPTS% -Dsolr.placementplugin.default=%SOLR_PLACEMENTPLUGIN_DEFAULT%"
Expand Down Expand Up @@ -1220,7 +1213,6 @@ IF "%SOLR_SSL_ENABLED%"=="true" (
set "SSL_PORT_PROP=-Dsolr.jetty.https.port=%SOLR_PORT%"
set "START_OPTS=%START_OPTS% %SOLR_SSL_OPTS% !SSL_PORT_PROP!"
)
IF NOT "%SOLR_LOG_LEVEL%"=="" set "START_OPTS=%START_OPTS% -Dsolr.log.level=%SOLR_LOG_LEVEL%"

set SOLR_LOGS_DIR_QUOTED="%SOLR_LOGS_DIR%"
set SOLR_DATA_HOME_QUOTED="%SOLR_DATA_HOME%"
Expand Down
5 changes: 4 additions & 1 deletion solr/bin/solr.in.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,10 @@ REM start command line as-is, in ADDITION to other options. If you specify the
REM -a option on start script, those options will be appended as well. Examples:
REM set SOLR_OPTS=%SOLR_OPTS% -Dsolr.autoSoftCommit.maxTime=3000
REM set SOLR_OPTS=%SOLR_OPTS% -Dsolr.autoCommit.maxTime=60000
REM set SOLR_OPTS=%SOLR_OPTS% -Dsolr.clustering.enabled=true

REM Most properties have an environment variable equivalent.
REM A naming convention is that SOLR_FOO_BAR maps to solr.foo.bar
REM SOLR_CLUSTERING_ENABLED=true

REM Path to a directory for Solr to store cores and their data. By default, Solr will use server\solr
REM If solr.xml is not stored in ZooKeeper, this directory needs to contain solr.xml
Expand Down
5 changes: 4 additions & 1 deletion solr/bin/solr.in.sh
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,10 @@
# -a option on start script, those options will be appended as well. Examples:
#SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000"
#SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000"
#SOLR_OPTS="$SOLR_OPTS -Dsolr.clustering.enabled=true"

# Most properties have an environment variable equivalent.
# A naming convention is that SOLR_FOO_BAR maps to solr.foo.bar
#SOLR_CLUSTERING_ENABLED=true

# Location where the bin/solr script will save PID files for running instances
# If not set, the script will create PID files in $SOLR_TIP/bin
Expand Down
16 changes: 4 additions & 12 deletions solr/core/src/java/org/apache/solr/cli/SolrCLI.java
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@
import org.apache.solr.common.params.CommonParams;
import org.apache.solr.common.util.ContentStreamBase;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.util.EnvUtils;
import org.apache.solr.util.StartupLoggingUtils;
import org.apache.solr.util.configuration.SSLConfigurationsFactory;
import org.slf4j.Logger;
Expand Down Expand Up @@ -187,18 +188,9 @@ public static CommandLine parseCmdLine(String toolName, String[] args, List<Opti
}

public static String getDefaultSolrUrl() {
String scheme = System.getenv("SOLR_URL_SCHEME");
if (scheme == null) {
scheme = "http";
}
String host = System.getenv("SOLR_TOOL_HOST");
if (host == null) {
host = "localhost";
}
String port = System.getenv("SOLR_PORT");
if (port == null) {
port = "8983";
}
String scheme = EnvUtils.getEnv("SOLR_URL_SCHEME", "http");
String host = EnvUtils.getEnv("SOLR_TOOL_HOST", "localhost");
String port = EnvUtils.getEnv("SOLR_PORT", "8983");
return String.format(Locale.ROOT, "%s://%s:%s", scheme.toLowerCase(Locale.ROOT), host, port);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import java.util.Map;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.util.EnvUtils;
import org.apache.solr.util.plugin.NamedListInitializedPlugin;
import org.apache.solr.util.tracing.SimplePropagator;
import org.apache.solr.util.tracing.TraceUtils;
Expand All @@ -33,10 +34,10 @@
public abstract class TracerConfigurator implements NamedListInitializedPlugin {

public static final boolean TRACE_ID_GEN_ENABLED =
Boolean.parseBoolean(System.getProperty("solr.alwaysOnTraceId", "true"));
Boolean.parseBoolean(EnvUtils.getProp("solr.alwaysOnTraceId", "true"));

private static final String DEFAULT_CLASS_NAME =
System.getProperty(
EnvUtils.getProp(
"solr.otelDefaultConfigurator", "org.apache.solr.opentelemetry.OtelTracerConfigurator");

private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
Expand Down
4 changes: 2 additions & 2 deletions solr/core/src/java/org/apache/solr/pkg/PackageAPI.java
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
import org.apache.solr.filestore.PackageStoreAPI;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.util.EnvUtils;
import org.apache.solr.util.SolrJacksonAnnotationInspector;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
Expand All @@ -57,8 +58,7 @@

/** This implements the public end points (/api/cluster/package) of package API. */
public class PackageAPI {
public final boolean enablePackages =
Boolean.parseBoolean(System.getProperty("enable.packages", "false"));
public final boolean enablePackages = EnvUtils.getPropAsBool("enable.packages", false);
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());

public static final String ERR_MSG =
Expand Down
Loading