Improving cell detection code

[dstansby]

This is a meta-issue to collect tasks to do to work towards improving the cell detection code. Feel free to edit this and add things!

Currently the code (after #84 is merged) has the following downsides:

• The parallelism is done in a custom and low level way
• The Cython code is not particularly easy to maintain, and requires the production of pre-compiled wheels
• Running a small volume of data takes a long time, due to overheads in the multiprocessing

To explore better ways of structuring the code we can start with the following tasks:

• Create pure Python versions of the currently used Cython code
• Write a benchmark for the 2D filtering
• Write a benchmark for the 3D filtering
• Write a benchmakrk for the whole cell detection stage
• Investigate if it's possible to run a unit test that errors if too much memory is used

Benchmarking results

Benchmarking code will be merge into the benchmarks branch. This branch is not intended to be merged into main, but will contain different implementations and benchmarking code. The best implementation and the benchmarking code will be merged into main eventually.

• 2D benchmarking results are documented in Benchmark 2D filter #88
• 3D benchmarking results are documented in Benchmark 3D filter #89

Useful resources:
https://examples.dask.org/applications/stencils-with-numba.html

[dstansby]

Since we merged our Cython > numba work and added some benchmarks as part of that, I'm going to close this as completed.