exp/lighthorizon: Unify single-process and map-reduce index builders.

[Shaptic]

What

This unifies the index building across the map-reduce and single-process versions:

• builder.go:BuildIndices does the work (so we get txmeta indexing "for free")
• the redundant participantsForOperations and getLedgerKeyParticipants are gone
• the new ProcessAccountsWithoutBackend module does the backend-less index building
• the shouldAccountBeProcessed and shouldTransactionBeProcessed helpers are pulled out versions of the batching code

It also cleans up and abstracts away the environmental variables for AWS Batch, which should make it simpler to support other platforms or even local runs.

Why

Less code, more sharing. Related to #4403.

Known limitations

I haven't tested this on AWS Batch yet.

[Shaptic]

Here's a bash script I used to test this locally, for posterity:

# Performs a local map-reduce indexing.
#
# Obviously, this does not offer many performance benefits over running the
# indexer using the single-process version. However, it does allow us to test it
# locally without needing to set up AWS Batch jobs.

MONOREPO="$GOPATH/src/github.com/stellar/go"
MAPREDUCE="$MONOREPO/exp/lighthorizon/index/cmd/batch"
CC_TOML="/etc/default/stellar-captive-core.toml"

# We assume the ledger exporter has already been run.
TXMETA_PATH="$HOME/workspace/txmeta-archive-bigly"
echo "Searching $TXMETA_PATH..."

LATEST_LEDGER=$(ls $TXMETA_PATH/ledgers | tail -n1)
BASE=$(ls $TXMETA_PATH/ledgers | head -n1)
START_LEDGER=$((((($BASE / 64) + 1) * 64) - 1))
COUNT=$(($LATEST_LEDGER - $START_LEDGER))

echo "Determined ledger range: [$START_LEDGER, $LATEST_LEDGER] ($COUNT ledgers)"

INDICES_PATH="$HOME/workspace/map-reduce/indices-dump"
echo "Recreating target directory: $INDICES_PATH"

# rm -rf $INDICES_PATH/G*/ $INDICES_PATH/accounts

rm -rf $INDICES_PATH
mkdir -p $INDICES_PATH

cd $MAPREDUCE/map
go build . || exit
for i in {0..3}
do
    echo "Creating map job $i"

    AWS_BATCH_JOB_ARRAY_INDEX=$i \
    BATCH_SIZE=128 \
    FIRST_CHECKPOINT=$START_LEDGER \
    TXMETA_SOURCE=file://$TXMETA_PATH \
    INDEX_TARGET=file://$INDICES_PATH/job_$i/ \
        ./map &
done

while :
do
    COUNT=$(ps aux | grep -E './map$' | wc -l)
    if [[ $COUNT -eq "0" ]]; then
        break
    fi
    sleep 1
done

echo "Map jobs are complete"

cd $MAPREDUCE/reduce
go build . || exit
for i in {0..0}
do
    echo "Creating reduce job $i"
    AWS_BATCH_JOB_ARRAY_INDEX=$i \
    REDUCE_JOB_COUNT=1 \
    MAP_JOB_COUNT=2 \
    WORKER_COUNT=2 \
    INDEX_SOURCE_ROOT=file://$INDICES_PATH/ \
    INDEX_TARGET=file://$INDICES_PATH/ \
        ./reduce &
done

while :
do
    COUNT=$(ps aux | grep -E './reduce$' | wc -l)
    if [[ $COUNT -eq "0" ]]; then
        break
    fi
    sleep 1
done

echo "Reduce jobs are complete"

We should probably turn this into an integration test of sorts at some point.