Create and test Hadoop 020/3 Solr indexer

[anjackson]

We want to start automatically indexing FC WARCs for full-text search.

• See e.g. website/scripts/run-solr-indexer.sh for the basic operation.
• The Solr indexer uses a SurtPrefixSet for the Open Access list, so that is expected to be SURTs. This should be provided by the OA Surts file generated by w3act_export.
• The Solr indexer uses a StaticMapExclusionFilterFactory for exclusions, like Open Wayback, so this can be a mixture of URLs and SURTs. The PyWB block files are manually-managed files from the internal GitLab repo.

Some of this will be done in ukwa-manage rather than here, but we'll need an Airflow runner.

Create a Solr indexer that:

• Runs like the CDX Indexer, tracking progress in TrackDB.
• Use http://solr.api.wa.bl.uk/solr/admin/collections?action=LIST&wt=json to determine which collection to use, and throw an error if there's no match.
• Add option to only index items marked as indexed in CDX.
• Verify that the the full-text block file should be the union of both of the PyWB block files.
• Create Airflow DAGs to run the Solr indexer like we do for the CDX indexer.
• Should commit at the end?
• If possible, verifies it worked.