Optimize SQL queries in pull_missing_blocks

[albrow]

Description

This PR optimizes the query for finding missing blocks. The old implementation is bogging down our SQL server and occasionally causing queries to take over one hour! In a full-ish database with many events and blocks the new implementation is at least a 1000x performance increase.

Testing instructions

Types of changes

Checklist:

• Prefix PR title with [WIP] if necessary.
• Add tests to cover changes as needed.
• Update documentation as needed.
• Add new entries to the relevant CHANGELOG.jsons.

[albrow]

I also increased the maximum number of concurrent requests. I've been monitoring memory and CPU usage and this should speed up the script without taxing resources too much. We can adjust further in the future.

[coveralls]

Coverage increased (+0.02%) to 85.312% when pulling 1d25925 on fix/missing-blocks-query-optimization into b8f3fa9 on development.

[xianny]

Nice! Getting rid of the WHERE.. NOT IN clause should speed things up a lot.

Couple questions:

• do we expect insignificant amount of overlap in block numbers across different events?
• if not, did you also try the original UNION query with the new JOIN... NOT NULL clause and find that removing the UNION query sped things up significantly? I would expect the WHERE... NOT IN clause to be the primary slowdown, not the UNION, but don't know offhand.

If yes to one or both, looks good to merge for me 👍

[albrow]

@xianny

do we expect insignificant amount of overlap in block numbers across different events?

We expect overlap between block numbers across different events. However, since we save any newly found blocks before checking for missing events in the next table in the list, this doesn't result in any duplicated work.

In other words, if the same block is missing in both raw.exchange_fill_events and raw.exchange_cancel_events, the block number will only be returned once from any query. Data for that block will also only be retrieved and saved once.

if not, did you also try the original UNION query with the new JOIN... NOT NULL clause and find that removing the UNION query sped things up significantly? I would expect the WHERE... NOT IN clause to be the primary slowdown, not the UNION, but don't know offhand.

I just tested locally and adding UNION back into the query (and keeping the JOIN.. NOT NULL) results in a 100-200ms increase (about 40-80%). I vote that we keep the query the way it is now.

[xianny]

Makes sense to me :)