[EPIC] Simplify Search

[pgwillia]

Epic / Very High Level User Story

Step 1: Describe The Problem We're Here to Solve

We have 9 servers (SolrCloud nodes, Zookeeper, etc) supporting a end of life version of Solr (6.6) for just ERA now. It would be a big win to be able to simplify this. Coming up with a plan and estimates for that work would be a good first step.

New information enforcing this as a priority is that yesterday we had a power supply failure for one of the Solr servers.
Henry had the foresight to order extras the last time they were replaced so Neil could quickly resolve the issue. Because Solr is redundant there was no impact to ERA. We may have more difficulty finding replacement power supplies in the future. Wendy will see if she can find another spare.

Obviously, we're good in the short term, we can borrow parts from staging servers or let nodes go dark. Longer term we should simplify this and retire the ageing hardware.

Additional information:

#2406

• Simplifying the stack will make the decommission of ECHA datacenter easier, because we have several bare metal servers to support this.
• Simplifying the stack will make the OS migration project easier, because we have many systems to migrate as part of this supporting SolrCloud
• The Metadata Strategies Unit uses Solr directly to get reports/data from ERA. Will need to consider their needs.

Step 2: The Elevator Pitch & Refinement

Elevator Pitch

(more information: https://agilewarrior.wordpress.com/2010/11/06/the-agile-inception-deck/)

A feature-oriented proposal for a solution in user-understandable language (and related artifacts like mockups, if those might be helpful), generated by Developers & Metadata specialists based on the problem description, for
approval by Product Owner & relevant stakeholders. Should address the following:

• What are we doing?

• What are we NOT doing?

• What resources will be needed to implement this?

Refinement

(as much or as little as makes sense for the pitch)

Metadata

What metadata is involved? What is changing in terms of metadata tracked for specific entities? Links to documentation etc.

Backend

Things like whiteboard sketches of architectural changes, Rails model component interactions, pictures of the outcome of event storm sessions, etc

Frontend

Things like wireframes, sketches of potential UI approaches, etc

Step 3: Tickets

Links to specific, achievable, estimable tickets with clear "Done" criteria, describing the work needed to implement the solution outlined above. By the time these are added to this document, they should be in a state ready for estimation.

[murny]

The plan here for a short term win is to simply upgrade Solr to a version that is supported and not EOL.

So this will require slowly upgrading Solr until version 8 or 9. We will go to 7 first, then up to 8/9 next.

I expect lots of issues with our configuration. For example simply bumping our docker container to 7 gets us:

Can't load schema /opt/solr/server/solr/mycores/development/schema.xml: Setting default operator in schema (solrQueryParser/@defaultOperator) not supported

[murny]

Hyrax is on 8.11 now, so hopefully we can see how they upgraded theirs and follow suit

[pgwillia]

Right sizing the project will be important because it's sounding like we'll be moving to DSpace sometime in 2025 (I've heard December, July and Q1 in the last few weeks). I think the driving requirements of the project are

• Solr Cloud is supported by several bare metal servers that are end of life and in the ECHA datacenter that will be decommissioned in the next year
• Solr Cloud is supported by several servers with an OS that is end of life in June 2024
• Solr 6.6 is end of life many years ago
• Metadata Strategies uses Solr in their workflow for generating reports and auditing
  So another option might be to upgrade Solr and/or simplify the back-end architecture supporting it.