Support running across two Hadoop clusters

[anjackson]

So we can upgrade/migrate as needed, we need to be able to run across two Hadoop clusters.

• Airflow Add support for scanning TrackDB from either Hadoop service. #50
• Support multiple HDFS backends ukwa-warc-server#8
• Create and test Hadoop 020/3 CDX indexer ukwa-manage#80
• Create Hadoop 020/3 Document Harvester Log Analysis task ukwa-manage#82
• Add Airflow DAGs to run the above tools on both Hadoop services

[anjackson]

A DAG for CDX indexing has been created. Tested and running on h020, expected to work on h3 but awaiting tests.

[anjackson]

Pleasingly, the same indexer code now runs smoothly over h020 and h3 with little fuss.

[anjackson]

Okay, so we are now running over two clusters so I'm going to call this done. There are some minor issues outstanding but they'll be dealt with in other tickets.