Upgrade the DB client to support batching of PSQL queries #18

cortze · 2023-05-26T09:59:32Z

Task descriptions

This PR includes

Upgrades on the DB client to support batching of queries (significantly improves the performance of writing to the DB)
Adds Dockerfile to support faster deployments and testing
Removes the HTTP API to communicate with a remote OP publisher. Why?
1. OP is part of the KAD-DHT now - so there is no need to update it
2. as OP will be part of the DHT, the hoarder will support an OP publisher in the future (probably on the next PR)
3. It increased the complexity of the publisher when it is essentially pretty simple
Updates logic of the Hoarder to make it compatible with a long-running measurement
support single publisher to have accurate publishing times

closes Persisting into DB becoming very slow #16 as we now support batch querying and aggregate similar queries into a single header
closes Issues with pinging rounds #17 as it now pings by time and not by rounds

… (generate privKeys at the host)

… logs)

…ents)

cortze and others added 19 commits January 16, 2023 13:01

update the SQL tables to add the time since publication column

a53ae03

update the models to track and expose the time since publication

1e31a77

update code to suport new models

3274a4d

update bugged publisher typo

bc00f3d

update in publisher and configs to support long-run hoards

52ba8e7

remove cid-source and keep only cid-generator

306cb2e

remove http API as OP will be part of the Hoarder itself

00e0f81

update the host to receive blacklisted peers and options on arguments…

b9057d7

… (generate privKeys at the host)

refactor/clean the hoarder service + make names clearer

d40bb61

update the models to support the MAddrs study on the DB (removes ugly…

e344f45

… logs)

clean up the DB service + update schemas

06f69a1

add docker image

9ce40d3

update dht fork reference commit

b95d283

update the hoarder to the kad-dht v0.23

91d6659

update DB client to support SQL query batching

ac90735

improve logic and closure of the hoarder (support contineous measurem…

479207b

…ents)

update main struct of the cid to keep track of time since publication

b594fa1

update dependencies

fcf4294

update kad-dht ref

fb633d6

cortze mentioned this pull request May 26, 2023

HTTP server fixes #15

Closed

cortze merged commit c60c575 into master May 26, 2023