feat(elastic): improve indexing performance

[hendrik-advantitge]

As promised, a PR with some performance upgrades for the ElasticSearch plugin.

• Returned FacetValues are now filtered based on Channel since those became ChannelAware. I fixed this by making the findByIds() function ChannelAware, which makes this function not consistent with the others. I'm open to suggestions here.
• We expanded the connectivity check on startup to retry a configurable amount of times. Reason for this is that hosting platforms perform maintenance tasks that often involve restarting nodes. ElasticSearch is much slower in startup than Vendure, and hence the worker would start without subscribing to the indexing jobs.
• During indexing, the Product and its variants are queried separately. This greatly reduces memory usage of the worker.
• Indexing itself is simplified a lot. Every indexing job now just performs a delete and insert on all Channels.
• During reindexing, if the dropIndices parameter is false, the products are deleted just before they are inserted again. This allows admins to reindex while the shop is live. Before, a reindex would clear the index entirely before inserting everything.
• Batching for reindexing is removed.

I'm aware this could be further cleaned up, but if that is okay with you I'm leaving that to you for now.

[michaelbromley]

Great, thanks! Looks like you need to include @types/lodash in devDependencies - that seems to be what is making the CI runs fail.

I'll review properly later today 👍

[hendrik-advantitge]

I see some tests are failing for search. Many of them seem to be concerning the order of results, which is just hard coded? Others are regarding the count of FacetValues. Probably the tests are not updated for the ChannelAware FacetValues? I'm not sure.

[michaelbromley]

I'll investigate when I review the changes later. Perhaps the ordering is affected by the new method of indexing?

[hendrik-advantitge]

That is very likely. To be determined if the order in these tests is important or not?

[michaelbromley]

The order is only important if there is an explicit sort order given in the search query.

[michaelbromley]

I believe this change to the way facetValues are counted is responsible for some of the test failures. This change does not replicate the old behaviour: previously, it would return the facetValues counts for the entire result set. Now it is only returning the count for the current page of results.

[hendrik-advantitge]

I see. We are running this revised plugin for quite some time now but hadn't noticed as we only use this for the items in the current result set. Now I also understand why there were two calls to the Elastic instance before. Should be relatively straightforward to put the old system back in place.

[michaelbromley]

Alright, I'll revert that part 👍

[michaelbromley]

This is now tidied up and merged. Thank you for your work on this!