Speed of Fitting in Different Versions of SasView

[smk78]

I thought it might be worthwhile to capture in a Discussion a recent thread from the Developers List.

INTRODUCTION
The thread started when I happened to ask an academic user of SasView why they were still using SasView 4.1.0 when we had released 6 versions of SasView 5.x. Their reply was:

...when the size polydispersity is incorporated into the fitting, the fitting process is relatively fast with Version 4. Sometimes, the fitting process with Version 5 is very slow. More importantly, the parameters extracted from model fitting show little difference between the two Versions, with respect to our SANS data.

The second part of that statement is, of course, good news. But the first part sufficiently intrigued me that I made some crude measurements and posted them to the Developers List. Responses advanced some plausible explanations (which we'll come to later) but @pkienzle also suggested a 'more refined' way to time my measurements. I've now had an opportunity to put that into practice.

PLATFORM
Dell desktop running W10 Enterprise (Build 19044)
Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz, 3701 Mhz, 4 Core(s), 8 Logical Processor(s)
16Gb RAM

METHODOLOGY
Run a version of SasView
Load \test\1d_data\cyl_400_40.txt (NB: for the record, these data do not contain dQ information)
Send to Fitting
Select Cylinder : cylinder
Check scale, background, radius
(as required) Check Polydispersity and enter PD=0.1 for Distribution of radius
(as required) Check Distribution of radius
Choose an optimizer (Fitting > Fit Algorithms/Options)
Disable OpenCL (Fitting > GPU Options/OpenCL Options > No OpenCL)
(5.x only) Start the iPython Console (Tool > Python Shell/Editor) and paste in the following (with thanks to @pkienzle):

import time
import sas.sascalc.fit.BumpsFitting as BF
import logging

old_run = BF.run_bumps
def run_bumps(problem, handler, curr_thread):
    N = 100
    t0 = time.time()
    for k in range(N):
        problem.model_update()
        problem.nllf()
    [logging.info](http://logging.info/)("model speed = %.3f ms/eval", 1000*(time.time()-t0)/N)
    return old_run(problem, handler, curr_thread)
BF.run_bumps = run_bumps

For everything else, use the SasView defaults!
Click Fit
When the fit has converged:

• (4.x) Click the Console button (bottom right), scroll to the bottom of the popup, and note the 'Duration time'
• (5.x) In the Log Explorer note the 'INFO: model speed' and the bottom of the GUI panel note the 'Fitting completed successfully in'

Close SasView
Rinse and repeat

RESULTS
The following SasView versions have been evaluated:

• 4.2.2 (built with SasModels 0.99 & Bumps 0.7.11)
• 5.0.3 (built with SasModels 1.0.3 & Bumps 0.7.16)
• 5.0.5 (built with SasModels 1.0.6 & Bumps 0.9.0)

Key:

• L-M = Levenberg-Marquadt
• N-M = Nelder-Mead Simplex

Fit Duration: No Polydispersity

SasView	L-M	N-M	DREAM
4.2.2	0.08 s	0.74 s	39.1 s
5.0.3	0.16 s	0.93 s	39.4 s
5.0.5	0.39 s	0.61 s	39.4 s

Fit Duration: With Polydispersity but PD Fixed

SasView	L-M	N-M	DREAM
4.2.2	1.18 s	22.7 s	367 s
5.0.3	2.86 s	21.4 s	308 s
5.0.5	4.91 s	22.0 s	308 s

Fit Duration: With Polydispersity and PD Optimising

SasView	L-M	N-M	DREAM
4.2.2	4.73 s	35.4 s	280 s
5.0.3	5.38 s	65.4 s	261 s
5.0.5	18.2 s	68.0 s	270 s

Time per evaluation: No Polydispersity

SasView	L-M	N-M	DREAM
5.0.3	1.13 ms	0.96 ms	1.02 ms
5.0.5	0.86 ms	0.91 ms	0.94 ms

Time per evaluation: With Polydispersity but PD Fixed

SasView	L-M	N-M	DREAM
5.0.3	18.7 ms	19.2 ms	19.4 ms
5.0.5	20.9 ms	18.5 ms	18.4 ms

Time per evaluation: With Polydispersity and PD Optimising

SasView	L-M	N-M	DREAM
5.0.3	18.5 ms	18.9 ms	19.3 ms
5.0.5	19.6 ms	19.8 ms	18.8 ms

DISCUSSION
On the face of it, the L-M optimizer in 4.2.2/5.0.3 is indeed faster than the L-M optimizer in 5.0.5. But of course they are not the same optimizer! SasView Users had pointed out that the L-M optimizer in 5.0.4 and earlier did not respect preset parameter bounds; ie, if you set min and max values for a parameter, say the radius, these would be ignored during the fitting! So for 5.0.5 the L-M optimizer used by SasView was changed from the one in SciPy to the MPfit one. But notice that the time per evaluation for both versions of the L-M optimizer are actually very similar for a given problem!

The rest of the timings are logical: more robust optimisers naturally take longer to converge, and fitting with polydispersity takes longer than fitting without polydispersity!

Perhaps the one surprise is that DREAM converged faster when the radius distribution was allowed to optimize compared to when it was not. But this is presumably a reflection of it just searching a more realistic parameter space?

[smk78]

But what are the reasons for the difference in speed between the L-M optimizers?

In the Developer thread @pkienzle suggests that the MPfit optimizer is making more function evaluations (which would make sense given that the time per evaluation is now shown to be ~same for both optimizers) and suggests a couple of reasons for why this might be:

• that the MPfit stopping condition is more exacting^, and/or
• that the amount of work per step may be different, for example, if MPfit is using the central difference rather than the forward difference when computing the derivative (meaning 2x evaluations per derivative).

^ Even when not using GPU MPfit is still setting tolerances and step sizes to 2ε for double precision ε rather than single precision so you may need more iterations. A 3x increase in the number of steps runs counter to theoretical convergence expectation. Newton style steppers should more or less double the number of digits of precision with each step near the minimum so a tiny ε should only add a couple of steps. For L-M it will be somewhat more depending on the adaptive rate parameter built into the algorithm, but it should not be 3x.

[smk78]

Also see SasView/sasmodels#518