ci(portal loop): Create exports from portal loop

[sw360cab]

Introduces a CI workflow that backup deltas of new transactions from portal loop and perform the required extractions from there

[moul]

can you create a new PR with only the logic and without the result? (feel free to keep this one open as a way to preview the result).

[moul]

I don't believe a CI workflow is the best approach. The portal loop isn't a stable network, so you can easily miss transactions by not calling this workflow at the right moment, which will likely happen about 99.9% of the time.

The portal loop already has a concept of local logs that are created on the machine precisely when needed, specifically just before an upgrade.

This is likely what is already backed up on S3, which isn't what we want. We need a way to patch the logs, which is why we require this repository instead of s3.

---

Edit: I understand now; you're combining the newly fetched genesis file with the new transactions.

I believe this approach can work. We should verify its consistency, such as what happens when we run the script twice or when we run it twice with two different instances.

Otherwise, looks good to me.

[sw360cab]

I can do the PR, I just lost track of this since it is not clear to me what is the final expected outcome.

The way it was thought for the portal loop is that we are saving only the deltas from last / most recent backup.
So there will a very first save of the whole transactions, following stored files (monotonic), will only store the deltas from the previous backup.
In this sense:

• using timestamps and sort guarantees non overlapping data
• the moment of saving is not so important since the delta stored will go back to the latest save (sort + diff)

A way to make this more effective is to connect the job to the merge into master of the monorepo so that the save occur before portal loop is looped.
Or we should embed all this stuff into the "looping" process of the portal-loop itself. To me the latter is way more complicated.

[moul]

Can you do the PR, please?

[sw360cab]

The PR has been updated leaving only the relevant script (used into the workflow).

We reviewed the flow to make it simple and clear:

• there is only one backup reference
  • a file for txs
  • a file for balances
• the script checks every hour the status of PL
• the first time the file will just save the backup files as nothing else was found
• following tasks dispatched (cron) will compare the current txs / balances export with the prevoius one
  • if no differences are found nothing happens
• otherwise the incoming exported txs / balances files will replace existing ones

Since any updated file should triggers a commit in the repo, we should actually achieve out of-the-box history and versioning of the backup reference.

[zivkovicmilos]

Looks good 😎

Pinging @moul for a final review 🙏

[zivkovicmilos]

Just a note, it's fine to replace both the balances and txs, because balances can't be changed without transactions

[zivkovicmilos]

@moul

The idea behind this approach is to always have a git history of the portal loop transactions (that are equal to the ones on the machine), so we can change the history, and easily "replay" the portal loop with the modified genesis balances / txs