Upgrade Solr to 9.4.1 (#10636)

* As discussed in sec.92, getting rid of the conf/solr/9.3.0 directory, in favor of keeping the config files in conf/solr/ going forward. * Removed the hard-coded solr version number from the installer makefile * removed the hard-coded 9.3.0 from the metadata customization guide. * changed to reflect 9.4.1 as the currently required version * updated the "classic" dev. install instruction, to reflect the new solr version * a release note. * #92sec move solr conf files into versioned subdirectory * Revert "#92sec move solr conf files into versioned subdirectory" This reverts commit eaa9c7c. * getting rid of 9.3.0 in this shellspec script #10636 * updated solr.version in the parent pom file (it is used in making docker images, apparently). #10636 * removed the remaining instances of the old solr version here and there #10636 * Removed the solr insallation instructions from the "classic dev. install" guide, replaced them with "follow the instructions provided in the main Installation/prerequisites gujide". No need to duplicate these in both places. #10636. * checking in @stevenwinship's workaround for the failing shellcheck test. #10636 --------- Co-authored-by: Don Sizemore <[email protected]>
IQSS · Jun 20, 2024 · 0f0c3b6 · 0f0c3b6
1 parent c141941
commit 0f0c3b6
Show file tree

Hide file tree

Showing 14 changed files with 51 additions and 68 deletions.
diff --git a/conf/solr/9.3.0/schema.xml → conf/solr/schema.xml b/conf/solr/9.3.0/schema.xml → conf/solr/schema.xml
diff --git a/conf/solr/9.3.0/solrconfig.xml → conf/solr/solrconfig.xml b/conf/solr/9.3.0/solrconfig.xml → conf/solr/solrconfig.xml
diff --git a/conf/solr/9.3.0/update-fields.sh → conf/solr/update-fields.sh b/conf/solr/9.3.0/update-fields.sh → conf/solr/update-fields.sh
diff --git a/doc/release-notes/solr-9.4.1.md b/doc/release-notes/solr-9.4.1.md
@@ -0,0 +1,14 @@
+Solr 9.4.1 is now the version recommended in our installation guides and used with automated testing. There is a known security issue in the previously recommended version 9.3.0: https://nvd.nist.gov/vuln/detail/CVE-2023-36478. While the risk of an exploit should not be significant unless the Solr instance is accessible from the outside networks (which we have always recommended against), existing Dataverse installations should consider upgrading.
+
+For the upgrade instructions section:
+
+[note that 6.3 will contain other solr-related changes, so the instructions may need to contain information merged from multiple release notes!]
+
+If you are upgrading Solr:
+ - Install solr-9.4.1 following the instructions from the Installation guide.
+ - Run a full reindex to populate the search catalog.
+ - Note that it may be possible to skip the reindexing step by simply moving the existing `.../server/solr/collection1/` under the new `solr-9.4.1` installation directory. This however has not been thoroughly tested and is not officially supported.
+
+
+
+
diff --git a/doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr b/doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr
@@ -5,7 +5,7 @@
 # chkconfig: 35 92 08
 # description: Starts and stops Apache Solr
 
-SOLR_DIR="/usr/local/solr/solr-9.3.0"
+SOLR_DIR="/usr/local/solr/solr-9.4.1"
 SOLR_COMMAND="bin/solr"
 SOLR_ARGS="-m 1g"
 SOLR_USER=solr

diff --git a/doc/sphinx-guides/source/_static/installation/files/etc/systemd/solr.service b/doc/sphinx-guides/source/_static/installation/files/etc/systemd/solr.service
@@ -5,9 +5,9 @@ After = syslog.target network.target remote-fs.target nss-lookup.target
 [Service]
 User = solr
 Type = forking
-WorkingDirectory = /usr/local/solr/solr-9.3.0
-ExecStart = /usr/local/solr/solr-9.3.0/bin/solr start -m 1g
-ExecStop = /usr/local/solr/solr-9.3.0/bin/solr stop
+WorkingDirectory = /usr/local/solr/solr-9.4.1
+ExecStart = /usr/local/solr/solr-9.4.1/bin/solr start -m 1g
+ExecStop = /usr/local/solr/solr-9.4.1/bin/solr stop
 LimitNOFILE=65000
 LimitNPROC=65000
 Restart=on-failure

diff --git a/doc/sphinx-guides/source/admin/metadatacustomization.rst b/doc/sphinx-guides/source/admin/metadatacustomization.rst
@@ -513,7 +513,7 @@ the Solr schema configuration, including any enabled metadata schemas:
 
 ``curl "http://localhost:8080/api/admin/index/solr/schema"``
 
-You can use :download:`update-fields.sh <../../../../conf/solr/9.3.0/update-fields.sh>` to easily add these to the
+You can use :download:`update-fields.sh <../../../../conf/solr/update-fields.sh>` to easily add these to the
 Solr schema you installed for your Dataverse installation.
 
 The script needs a target XML file containing your Solr schema. (See the :doc:`/installation/prerequisites/` section of
@@ -537,7 +537,7 @@ from some place else than your Dataverse installation).
 Please note that reconfigurations of your Solr index might require a re-index. Usually release notes indicate
 a necessary re-index, but for your custom metadata you will need to keep track on your own.
 
-Please note also that if you are going to make a pull request updating ``conf/solr/9.3.0/schema.xml`` with fields you have
+Please note also that if you are going to make a pull request updating ``conf/solr/schema.xml`` with fields you have
 added, you should first load all the custom metadata blocks in ``scripts/api/data/metadatablocks`` (including ones you
 don't care about) to create a complete list of fields. (This might change in the future.)
 

diff --git a/doc/sphinx-guides/source/developers/classic-dev-env.rst b/doc/sphinx-guides/source/developers/classic-dev-env.rst
@@ -136,41 +136,9 @@ On Linux, you should just install PostgreSQL using your favorite package manager
 Install Solr
 ^^^^^^^^^^^^
 
-`Solr <https://lucene.apache.org/solr/>`_ 9.3.0 is required.
+`Solr <https://lucene.apache.org/solr/>`_ 9.4.1 is required.
 
-To install Solr, execute the following commands:
-
-``sudo mkdir /usr/local/solr``
-
-``sudo chown $USER /usr/local/solr``
-
-``cd /usr/local/solr``
-
-``curl -O https://archive.apache.org/dist/solr/solr/9.3.0/solr-9.3.0.tgz``
-
-``tar xvfz solr-9.3.0.tgz``
-
-``cd solr-9.3.0/server/solr``
-
-``cp -r configsets/_default collection1``
-
-``curl -O https://raw.githubusercontent.com/IQSS/dataverse/develop/conf/solr/9.3.0/schema.xml``
-
-``curl -O https://raw.githubusercontent.com/IQSS/dataverse/develop/conf/solr/9.3.0/schema_dv_mdb_fields.xml``
-
-``mv schema*.xml collection1/conf``
-
-``curl -O https://raw.githubusercontent.com/IQSS/dataverse/develop/conf/solr/9.3.0/solrconfig.xml``
-
-``mv solrconfig.xml collection1/conf/solrconfig.xml``
-
-``cd /usr/local/solr/solr-9.3.0``
-
-(Please note that the extra jetty argument below is a security measure to limit connections to Solr to only your computer. For extra security, run a firewall.)
-
-``bin/solr start -j "-Djetty.host=127.0.0.1"``
-
-``bin/solr create_core -c collection1 -d server/solr/collection1/conf``
+Follow the instructions in the "Installing Solr" section of :doc:`/installation/prerequisites` in the main Installation guide.
 
 Install Service Dependencies Using Docker Compose
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst
@@ -163,7 +163,7 @@ The Dataverse software search index is powered by Solr.
 Supported Versions
 ==================
 
-The Dataverse software has been tested with Solr version 9.3.0. Future releases in the 9.x series are likely to be compatible. Please get in touch (:ref:`support`) if you are having trouble with a newer version.
+The Dataverse software has been tested with Solr version 9.4.1. Future releases in the 9.x series are likely to be compatible. Please get in touch (:ref:`support`) if you are having trouble with a newer version.
 
 Installing Solr
 ===============
@@ -178,19 +178,19 @@ Become the ``solr`` user and then download and configure Solr::
 
         su - solr
         cd /usr/local/solr
-        wget https://archive.apache.org/dist/solr/solr/9.3.0/solr-9.3.0.tgz
-        tar xvzf solr-9.3.0.tgz
-        cd solr-9.3.0
+        wget https://archive.apache.org/dist/solr/solr/9.4.1/solr-9.4.1.tgz
+        tar xvzf solr-9.4.1.tgz
+        cd solr-9.4.1
         cp -r server/solr/configsets/_default server/solr/collection1
 
 You should already have a "dvinstall.zip" file that you downloaded from https://github.com/IQSS/dataverse/releases . Unzip it into ``/tmp``. Then copy the files into place::
 
-        cp /tmp/dvinstall/schema*.xml /usr/local/solr/solr-9.3.0/server/solr/collection1/conf
-        cp /tmp/dvinstall/solrconfig.xml /usr/local/solr/solr-9.3.0/server/solr/collection1/conf
+        cp /tmp/dvinstall/schema*.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf
+        cp /tmp/dvinstall/solrconfig.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf
 
 Note: The Dataverse Project team has customized Solr to boost results that come from certain indexed elements inside the Dataverse installation, for example prioritizing results from Dataverse collections over Datasets. If you would like to remove this, edit your ``solrconfig.xml`` and remove the ``<str name="qf">`` element and its contents. If you have ideas about how this boosting could be improved, feel free to contact us through our Google Group https://groups.google.com/forum/#!forum/dataverse-dev .
 
-A Dataverse installation requires a change to the ``jetty.xml`` file that ships with Solr. Edit ``/usr/local/solr/solr-9.3.0/server/etc/jetty.xml`` , increasing ``requestHeaderSize`` from ``8192`` to ``102400``
+A Dataverse installation requires a change to the ``jetty.xml`` file that ships with Solr. Edit ``/usr/local/solr/solr-9.4.1/server/etc/jetty.xml`` , increasing ``requestHeaderSize`` from ``8192`` to ``102400``
 
 Solr will warn about needing to increase the number of file descriptors and max processes in a production environment but will still run with defaults. We have increased these values to the recommended levels by adding ulimit -n 65000 to the init script, and the following to ``/etc/security/limits.conf``::
 
@@ -209,7 +209,7 @@ Solr launches asynchronously and attempts to use the ``lsof`` binary to watch fo
 
 Finally, you need to tell Solr to create the core "collection1" on startup::
 
-        echo "name=collection1" > /usr/local/solr/solr-9.3.0/server/solr/collection1/core.properties
+        echo "name=collection1" > /usr/local/solr/solr-9.4.1/server/solr/collection1/core.properties
 
 Dataverse collection ("dataverse") page uses Solr very heavily. On a busy instance this may cause the search engine to become the performance bottleneck, making these pages take increasingly longer to load, potentially affecting the overall performance of the application and/or causing Solr itself to crash. If this is observed on your instance, we recommend uncommenting the following lines in the ``<circuitBreaker ...>`` section of the ``solrconfig.xml`` file::
 

diff --git a/docker/compose/demo/compose.yml b/docker/compose/demo/compose.yml
@@ -103,7 +103,7 @@ services:
   solr:
     container_name: "solr"
     hostname: "solr"
-    image: solr:9.3.0
+    image: solr:9.4.1
     depends_on:
       - solr_initializer
     restart: on-failure

diff --git a/modules/container-configbaker/assembly.xml b/modules/container-configbaker/assembly.xml
@@ -8,7 +8,7 @@
         </fileSet>
         <!-- Get our custom Solr files -->
         <fileSet>
-            <directory>conf/solr/9.3.0</directory>
+            <directory>conf/solr</directory>
             <outputDirectory>solr</outputDirectory>
         </fileSet>
         <!-- Get the setup scripts from the installer (selected choice only) -->

diff --git a/modules/dataverse-parent/pom.xml b/modules/dataverse-parent/pom.xml
@@ -150,7 +150,7 @@
         <!-- Major system components and dependencies -->
         <payara.version>6.2023.8</payara.version>
         <postgresql.version>42.7.2</postgresql.version>
-        <solr.version>9.3.0</solr.version>
+        <solr.version>9.4.1</solr.version>
         <aws.version>1.12.290</aws.version>
         <google.library.version>26.30.0</google.library.version>
 

diff --git a/scripts/installer/Makefile b/scripts/installer/Makefile
@@ -55,13 +55,13 @@ ${JHOVE_SCHEMA}:	../../conf/jhove/jhoveConfig.xsd ${INSTALLER_ZIP_DIR}
 	@echo copying jhove schema file
 	/bin/cp ../../conf/jhove/jhoveConfig.xsd ${INSTALLER_ZIP_DIR}
 
-${SOLR_SCHEMA}:		../../conf/solr/9.3.0/schema.xml ../../conf/solr/9.3.0/update-fields.sh ${INSTALLER_ZIP_DIR}
+${SOLR_SCHEMA}:		../../conf/solr/schema.xml ../../conf/solr/update-fields.sh ${INSTALLER_ZIP_DIR}
 	@echo copying Solr schema file
-	/bin/cp ../../conf/solr/9.3.0/schema.xml ../../conf/solr/9.3.0/update-fields.sh ${INSTALLER_ZIP_DIR}
+	/bin/cp ../../conf/solr/schema.xml ../../conf/solr/update-fields.sh ${INSTALLER_ZIP_DIR}
 
-${SOLR_CONFIG}:		../../conf/solr/9.3.0/solrconfig.xml ${INSTALLER_ZIP_DIR}
+${SOLR_CONFIG}:		../../conf/solr/solrconfig.xml ${INSTALLER_ZIP_DIR}
 	@echo copying Solr config file
-	/bin/cp ../../conf/solr/9.3.0/solrconfig.xml ${INSTALLER_ZIP_DIR}
+	/bin/cp ../../conf/solr/solrconfig.xml ${INSTALLER_ZIP_DIR}
 
 ${PYTHON_FILES}: README_python.txt install.py installConfig.py installAppServer.py installUtils.py requirements.txt default.config interactive.config ${INSTALLER_ZIP_DIR}
 	@echo copying Python installer files

diff --git a/tests/shell/spec/update_fields_spec.sh b/tests/shell/spec/update_fields_spec.sh
@@ -1,16 +1,17 @@
 #shellcheck shell=sh
+#shellcheck disable=SC2154
 
 update_fields() {
-  ../../conf/solr/9.3.0/update-fields.sh "$@"
+  ../../conf/solr/update-fields.sh "$@"
 }
 
 Describe "Update fields command"
 
   Describe "can operate on upstream data"
-    copyUpstreamSchema() { cp ../../conf/solr/9.3.0/schema.xml data/solr/upstream-schema.xml; }
+    copyUpstreamSchema() { cp ../../conf/solr/schema.xml data/solr/upstream-schema.xml; }
     AfterAll 'copyUpstreamSchema'
 
-    Path schema-xml="../../conf/solr/9.3.0/schema.xml"
+    Path schema-xml="../../conf/solr/schema.xml"
     It "needs upstream schema.xml"
       The path schema-xml should be exist
     End
@@ -115,16 +116,16 @@ Describe "Update fields command"
     End
 
     Describe "succeeds because"
-      setup() { cp data/solr/minimal-schema.xml data/solr/minimal-schema-work.xml; }
-      cleanup() { rm data/solr/minimal-schema-work.xml; }
-      BeforeEach 'setup'
-      AfterEach 'cleanup'
+      setup1() { cp data/solr/minimal-schema.xml data/solr/minimal-schema-work.xml; }
+      cleanup1() { rm data/solr/minimal-schema-work.xml; }
+      BeforeEach 'setup1'
+      AfterEach 'cleanup1'
 
       deleteUpstreamSchema() { rm data/solr/upstream-schema.xml; }
       AfterAll 'deleteUpstreamSchema'
 
-      match_content() {
-        grep -q "$@" "${match_content}"
+      match_content1() {
+        grep -q "$@" "${match_content1}"
       }
 
       It "prints nothing when editing minimal schema"
@@ -133,8 +134,8 @@ Describe "Update fields command"
         The status should equal 0
         The output should equal ""
         The path data/solr/minimal-schema-work.xml should be file
-        The path data/solr/minimal-schema-work.xml should satisfy match_content "<field name=\"test\""
-        The path data/solr/minimal-schema-work.xml should satisfy match_content "<copyField source=\"test\""
+        The path data/solr/minimal-schema-work.xml should satisfy match_content1 "<field name=\"test\""
+        The path data/solr/minimal-schema-work.xml should satisfy match_content1 "<copyField source=\"test\""
       End
 
       It "prints nothing when editing upstream schema"
@@ -143,8 +144,8 @@ Describe "Update fields command"
         The status should equal 0
         The output should equal ""
         The path data/solr/upstream-schema.xml should be file
-        The path data/solr/upstream-schema.xml should satisfy match_content "<field name=\"test\""
-        The path data/solr/upstream-schema.xml should satisfy match_content "<copyField source=\"test\""
+        The path data/solr/upstream-schema.xml should satisfy match_content1 "<field name=\"test\""
+        The path data/solr/upstream-schema.xml should satisfy match_content1 "<copyField source=\"test\""
       End
     End
   End