Program stop mid running without any outup

[alexmars123]

Hello! I tried to run the code several time and sometimes (but not always), it stops in the middle (not always in the same place). This is some output from one of the runs where it stopped (the last lines of the terminal output - Powershell on Windows 10):

INFO     cost 32.265573116854235 +/- 0.0
INFO     Run: 70 (machine learner)
INFO     params [-500.         -965.9951292  -600.97861861 -300.         -352.33433901]

INFO     cost 1000.0 +/- 0.0
INFO     Run: 71 (trainer)
INFO     params [-1298.35679689  -816.87250569  -615.15777157  -426.3981931
  -499.14163221]

INFO     cost 1000.0 +/- 0.0
INFO     Run: 72 (machine learner)
INFO     params [-1265.03281961 -1137.39862994  -657.10271614  -719.20824797
  -377.11512437]

INFO     cost 1000.0 +/- 0.0
INFO     Run: 73 (machine learner)
INFO     params [-1096.93149896 -1076.23267012  -614.1589204   -706.57107924
  -351.21274201]

INFO     cost 49.006086364989656 +/- 0.0
INFO     Run: 74 (machine learner)
INFO     params [-1500.         -1313.5352554   -765.02461555  -796.65924047
  -369.23886747]

INFO     cost 107.2868748223299 +/- 0.0
INFO     Run: 75 (trainer)
INFO     params [-726.25157754 -628.76582468 -604.00469597 -403.95902915 -397.53860198]

Sometimes it stops at a very early run (<10) sometimes it goes few hundred runs, sometimes it finishes, without changing anything in the python script, just re-running it. This is the get_next_cost_dict:

def get_next_cost_dict(self,params_dict):
    
    params = params_dict['params']
    V1 = params[0]
    V2 = params[1]
    V3 = params[2]
    V4 = params[3]
    V5 = params[4]

    filename = "out_test.txt"

    try:
        os.remove(filename)
    except:
        print("")

    subprocess.call(r"powershell.exe & '.\SIMION 8.1.lnk' --nogui fastadj .\electrode.PA0 " + "1=" + str(V1) + ",2=" + str(V2) + ",3=" + str(V3) + ",4=" + str(V4) + ",5=" + str(V5))
    subprocess.call(r"powershell.exe & '.\SIMION 8.1.lnk' --nogui fly --restore-potentials=0 --recording-output=out_test.txt .\electrode.iob")
    while True:
        if os.path.isfile(".\out_test.txt"):
            data = np.loadtxt(filename,skiprows=1,usecols=0)
            if len(data)==500:
                break
            else: 
                time.sleep(0.2)
        else:
            time.sleep(0.2)

    data_simion = np.loadtxt(filename,skiprows=1)

    idx = np.where(data_simion[:,0]==50)
    data_simion = data_simion[idx]
    pos_y = data_simion[:,1]
    pos_z = data_simion[:,2]
    radius = np.sqrt((pos_y)**2+(pos_z)**2)
    if len(data_simion)<400:
        resolution = 1000
    else:
        resolution = np.std(radius)/np.mean(radius)*200

    new_func_value = resolution
    os.remove(filename)

    cost = np.sum(new_func_value)
    uncer = 0
    bad = False
    
    cost_dict = {'cost':cost, 'uncer':uncer, 'bad':bad}
    return cost_dict

and this is the main function:

def main():
    filename = "learner_archive_" + str(strftime("%Y-%m-%d_%H-%M")) + ".txt"
    
    interface = CustomInterface()
    controller = mlc.create_controller(interface, 
                                       controller_type='neural_net',
                                       #controller_type='gaussian_process',
                                       max_num_runs = 500,
                                       num_params = 5, 
                                       min_boundary = [-1500,-1500,-1000,-1000,-500],
                                       max_boundary = [-500,-500,-300,-300,-100],
                                       first_params = [-1000,-1000,-500,-500,-300],
                                       training_type = "differential_evolution",
                                       num_training_runs = 50)
    controller.optimize()
    
    print('Best parameters found:')
    best_params = controller.best_params
    best_cost = controller.best_cost
    print(best_params)
    print(best_cost)

I get this behavior both for NN and Gaussian process.

[zakv]

Hi Alex,

From the log output it looks to me like M-LOOP has called get_next_cost_dict() with a set of parameters but hasn't gotten the result back, so I'm suspicious that there is an issue in your get_next_cost_dict(). My best guess is that one of your .\SIMION subprocess calls occasionally fails, possibly erroring for certain values (which vary run-to-run and iteration-to-iteration, and so could cause intermittent failures) or for some other reasons.

I'd suggest adding some logging or print() calls before/after each subprocess.call() and after the while True loop. That would help you nail down exactly where the hang occurs. Additionally you could try calling your get_next_cost_dict() function with the last set of values from the log output - it would be helpful to know if there are particular sets of values which reliably cause the hang you see.

Cheers,
Zak