This runs incredibly slow on my data. How to speed it up?

[loomcode]

I'm trying to track about 5000 peaks (2D) per frame for 10 frames. Using trackpy it takes about 10-15 seconds to run but btrack is still running after 20 minutes.. I've tried reducing theta_time and theta_dist in the model ("cellmodel.json") and that doesn't seem to have an effect. What optimization parameters can I change to speed this up? Additionally I'm running on a 20 core server. Can I take advantage of multiprocessing without writing my own implementation?

[quantumjot]

Hi @loomcode - can you post the log output so I can see where it is slow?

[quantumjot]

Quick follow up, it can take a long time for two reasons: (i) you're assessing all combinations of track linkages, or (ii) the optimizer is not well configured for your data. The log would help to discern which to optimize.

If you're tracking lots of objects, you can also use the APPROXIMATE updates, which should speed up the first step dramatically: see https://github.com/quantumjot/BayesianTracker#dealing-with-very-large-datasets

[loomcode]

Thanks for the "APPROXIMATE" updates suggestion. Here's the output when I run it with approximate updates enabled and max_search_radius = 5.

[INFO][2020/12/14 04:22:36 PM] Loading motion model: b'cell_motion'
[INFO][2020/12/14 04:22:36 PM] Setting Bayesian update method to: BayesianUpdates.APPROXIMATE
[INFO][2020/12/14 04:22:36 PM] Setting max XYZ search radius to: 10
[INFO][2020/12/14 04:22:36 PM] Starting tracking...
[INFO][2020/12/14 04:22:37 PM] Tracking objects in frames 0 to 11 (of 11)...
[INFO][2020/12/14 04:22:42 PM] - Timing (Bayesian updates: 325.71ms, Linking: 104.25ms)
[INFO][2020/12/14 04:22:42 PM] - Probabilities (Link: 0.99856, Lost: 0.09855)
[INFO][2020/12/14 04:22:42 PM] SUCCESS.
[INFO][2020/12/14 04:22:42 PM] - Found 8739 tracks in 11 frames (in 0.0s)
[INFO][2020/12/14 04:22:42 PM] - Inserted 987 dummy objects to fill tracking gaps
[INFO][2020/12/14 04:22:42 PM] Loading hypothesis model: cell_hypothesis
[INFO][2020/12/14 04:22:42 PM] Calculating hypotheses (relax: True)...
[INFO][2020/12/14 04:22:43 PM] Setting up constraints matrix for global optimisation..
[INFO][2020/12/14 04:30:39 PM] Optimizing...
GLPK Integer Optimizer, v4.65
35592 rows, 493661 columns, 1379167 non-zeros
493661 integer variables, all of which are binary
Preprocessing...
17796 rows, 493661 columns, 1379167 non-zeros
493661 integer variables, all of which are binary
Scaling...
A: min|aij| = 1.000e+00 max|aij| = 1.000e+00 ratio = 1.000e+00
Problem data seem to be well scaled
Constructing initial basis...
Size of triangular part is 17796
Solving LP relaxation...
GLPK Simplex Optimizer, v4.65
17796 rows, 493661 columns, 1379167 non-zeros

• 0: obj =   1.726683430e+04 inf =   0.000e+00 (437158)
  

Perturbing LP to avoid stalling [308]...

• 393: obj = 1.673753430e+04 inf = 0.000e+00 (417025)
• 778: obj = 1.627406764e+04 inf = 0.000e+00 (397086)
• 1161: obj = 1.583956764e+04 inf = 0.000e+00 (379940)

It continues like this for a long time. I'll leave it overnight to see if it finishes but my project requires an implementation that will finish in a matter of seconds.

[quantumjot]

It can take a long time to optimize if you have posed the problem poorly, i.e. you probably need to change some parameters in the config to better to describe what you are trying to track and what you care about.

Do you actually need to run the optimizer with only 10 frames of data? Do you care about identifying cell divisions? Seems like you could just comment that line out, and get decent results:

# tracker.optimize()

Are those tracks good enough? If not, then you'll need to optimize the config.