horizon/ingest: automatic best effort removal of captive core cursors during init

[sreuland]

What version are you using?

2.28.0

What did you do?

upgraded horizon from any prior release to 2.28.0, and the instance is ingest enabled via INGEST=true, which will run a captive core instance.

What did you expect to see?

horizon ingestion no longer sets/advances a captive cores cursor via setcursor and no additional disk space usage results from this.

What did you see instead?

horizon 2.28.0 ingestion stops advancing cursor per #5158 but core uses more diskspace as it doesn't purge captive core's bucket data on storage-path disk past the last known cursor that was set which becomes progressively aged more.

the horizon release notes recommend a manual work-around to resolve this in 2.28.0 breaking changes

however, this requires users to fully read the notes, recognize this and perform the actions which does impose extra management time on their behalf. To resolve this automatically, horizon could make a best effort to run an idempotent cleanup routine in ingestion captive core init state after captive core has started to remove any left-over cursors on a captive core instance by checking core's http service for cursors:
curl http://localhost:11626/getcursor

this will return an array:

{
   "cursors" : [
      {
         "cursor" : 49996882,
         "id" : "HORIZON"
      }
   ]
}

if it's empty, do nothing, otherwise for each cursor in the array, drop it with additional HTTP GET to core's http service:
http://127.0.0.1:11626/dropcursor?id=<id>

Alternatives?

manually doing the same, horizon release 2.28.0 notes mention how to do this same cursor removal manually, but this relies on users reading and doing it - https://github.com/stellar/go/releases/tag/horizon-v2.28.0

[sreuland]

another alternative solution is to wipe the captive core storage path directory, using core's new-db command, this achieves same end result of removing any prior cursors, but would also remove any prior ledger state, next run of the captive core would have to perform a full catch-up.

[mollykarcher]

Given we only experienced this on a single staging instance, we believe a new-db resolves it (and most fresh deploys/upgrades will have this), and most partners have already upgraded to 2.28.1, we've chosen to close this as we don't think it's worth the effort.