postgres table sharding POC

[InoMurko]

https://pgdash.io/blog/postgres-11-sharding.html

https://www.highgo.ca/2020/03/12/horizontal-scalability-sharding-in-postgresql-core-missing-pieces-of-the-puzzle/

https://wiki.postgresql.org/wiki/Built-in_Sharding

[InoMurko]

So the goal of this exercise is to see if PG internal sharding (partitioning) is helpful in splitting up the transaction table.
We would like to se:

1. local docker compose setup
2. demonstrable test with increased performance over the current
3. estimation of any necessary code changes and it's implications

[InoMurko]

The table that gets hit most and we would like to see it being more performant is Engine.DB.Output and perhaps also Engine.DB.Transaction. But at least for the Engine.DB.Transaction we have plans to prune it if it's going to be doable.

So these two tables should be your focus!

cc @pgebal @mederic-p if you have any more inputs since you've been doing some performance measurements, please leave them here for @DanielaIvanova !