store: answer history queries from sqlite DB

[kaiserd]

Background

While nwaku features a sqlite3 database for persisting messages,
history queries are only answered from the in-memory store.
When starting a nwaku node, --store-capacity messages are loaded from the sqlite3 DB into the in-memory database.

We also limited the capacity limit for the persistent storage in #900, to avoid overly long starting times for nwaku nodes.

Goal / Motivation

For some applications, storing more messages is desired. For instance, keeping a history of one month of messages.
To achieve this, a first simple solution is introducing a (large capacity) sqlite3-only store.

A follow up issue will track a feasibility test of the sqlite3-only store.
(Performance might not be sufficient.)

Acceptance Criteria

• sqlite3-only store implemented
• tests added

Future Work

• generic store interface

[jakubgs]

We seriously need this done. Right now it takes ~17 minutes for a node with 684365 messages in the DB to start:

INF 2022-05-23 17:37:46.236+00:00 attempting to load messages from persistent storage topics="wakustore" tid=1 file=waku_store.nim:370
INF 2022-05-23 17:53:53.236+00:00 successfully loaded from store             topics="wakustore" tid=1 file=waku_store.nim:377

And it appears it's fully utilizing the available disk I/O while doing it:

And we can see the container memory usage growing as it goes along:

[jakubgs]

I see the issue, it's because it's SWAPing due to container memory limits, which bumps the disk usage further:

This is not sustainable for any kind of large history.

[kaiserd]

Yes! I was busy with other pressing issues that took precedence, but now back on this.