Horizon db reingest command has duplicate error when run multiple times with same ledger range

[sreuland]

What version are you using?

v2.11.0

What did you do?

I ran this ingest command on an empty db
horizon db reingest range 962300 963300 --db-url <any_url_here> --history-archive-urls https://history.stellar.org/prd/core-testnet/core_testnet_001,https://history.stellar.org/prd/core-testnet/core_testnet_002 --network-passphrase "Test SDF Network ; September 2015"

then I ran the same command a second time against the same db, noticed reingest reported errors on the second time, stopped re-ingesting.

What did you expect to see?

the reingestion completes to final ledger without errors on console

What did you see instead?

ERRO[2021-11-30T12:22:56.255-08:00] Error in ingestion state machine
current_state="reingestHistoryRange(fromLedger=962300, toLedger=963300, force=false)" 
error="error processing ledger sequence=962976: Error commiting changes from processor: error in 
*processors.LiquidityPoolsTransactionProcessor.Commit: could not flush transaction liquidity 
pools to db: error adding values while inserting to history_transaction_liquidity_pools: 
exec failed: pq: duplicate key value violates unique 
constraint \"index_history_transaction_liquidity_pools_on_ids\"" next_state=stop pid=20551 service=ingest

INFO[2021-11-30T12:22:56.255-08:00] Shut down pid=20551 service=ingest
error processing ledger sequence=962976: Error commiting changes from processor: error in
*processors.LiquidityPoolsTransactionProcessor.Commit: could not flush 
transaction liquidity pools to db: error adding values while inserting to 
history_transaction_liquidity_pools: exec failed: pq: duplicate key value violates unique constraint
"index_history_transaction_liquidity_pools_on_ids"

[sreuland]

there is possible related history of same pattern just on different table, and the approach taken:

#3661

[bartekn]

Looks like quite easy to fix and also critical (reingestion doesn't work). @tamirms do you think we can include this in 2.12.0 (we can assign somebody to fix)?

[tamirms]

yeah, we can include this in the 2.12.0 release

[bartekn]

Done but not closed. Doing it now.