segment fault

[fengjiaxin]

hi，excuse me
i meet a new issue,when i train the model
i meet another issue
segment fault core dump
would you update the new code,i have no idea to solve the problem

and more:
i think GLN/gln/mods/mol_gnn/gnn_family/utils.py can update by replace cuda() to to(DEVICE)
thanks a lot

[Hanjun-Dai]

could you please provide more details for the segfault?

[fengjiaxin]

./run_mf.sh: 行 60: 9301 段错误 (吐核)python ../main.py -gm $gm -fp_degree 2 -neg_sample $neg_sample -att_type $att_type -gnn_out $gnn_out -tpl_enc $tpl_enc -subg_enc $subg_enc -latent_dim $msg_dim -bn $bn -gen_method $gen -retro_during_train $retro -neg_num $neg_size -embed_dim $embed_dim -readout_agg_type $graph_agg -act_func $act -act_last True -max_lv $lv -dropbox $dropbox -data_name $data_name -save_dir $save_dir -tpl_name $tpl_name -f_atoms $dropbox/cooked_$data_name/atom_list.txt -iters_per_val 3000 -gpu 1 -topk 50 -beam_size 50 -num_parts 1

no other information, i think its not environment issue

[Hanjun-Dai]

are you able to run the test with existing model dumps?

[Hanjun-Dai]

and did you modify the script?

I use -gpu 0 in the script. Please try with the vanilla code and see if that works

[fengjiaxin]

get another issue gpu cuda error
are ckpt file saved by gpu?

[fengjiaxin]

i use -gpu 1 ,and did you save the model by gpu 0, i run test script by error as follows:

Traceback (most recent call last):
File "main_test.py", line 139, in
model = RetroGLN(cmd_args.dropbox, local_args.model_for_test)
File "/home/fengjiaxin/GLN/gln/test/model_inference.py", line 43, in init
self.gln.load_state_dict(torch.load(model_file))
File "/home/fengjiaxin/.conda/envs/my-rdkit-env/lib/python3.6/site-packages/torch/serialization.py", line 426, in load
return _load(f, map_location, pickle_module, **pickle_load_args)
File "/home/fengjiaxin/.conda/envs/my-rdkit-env/lib/python3.6/site-packages/torch/serialization.py", line 613, in _load
result = unpickler.load()
File "/home/fengjiaxin/.conda/envs/my-rdkit-env/lib/python3.6/site-packages/torch/serialization.py", line 576, in persistent_load
deserialized_objects[root_key] = restore_location(obj, location)
File "/home/fengjiaxin/.conda/envs/my-rdkit-env/lib/python3.6/site-packages/torch/serialization.py", line 155, in default_restore_location
result = fn(storage, location)
File "/home/fengjiaxin/.conda/envs/my-rdkit-env/lib/python3.6/site-packages/torch/serialization.py", line 135, in _cuda_deserialize
return storage_type(obj.size())
File "/home/fengjiaxin/.conda/envs/my-rdkit-env/lib/python3.6/site-packages/torch/cuda/init.py", line 634, in _lazy_new
return super(_CudaBase, cls).new(cls, *args, **kwargs)
RuntimeError: CUDA error: out of memory

[Hanjun-Dai]

yes it uses gpu by default. Please always use -gpu 0 in your script.
If you want to change GPU, please use CUDA_VISIBLE_DEVICES instead

[fengjiaxin]

hi , i debug the code ,some error at GLN/gln/graph_logic/soft_logic.py line 29
jagged_forward graph_embed = graph_enc(list)
no other information
can you introduce your code in brief
i can not find the error
thanks

[fengjiaxin]

can you give a docker image? i think it will be useful

[Hanjun-Dai]

graph_enc is from another sub package in this repo.

Can you first try without GPU? Please take a look at this:
https://discuss.pytorch.org/t/on-a-cpu-device-how-to-load-checkpoint-saved-on-gpu-device/349

to see how to load a gpu dump into cpu

[fengjiaxin]

hi, i debug the traing file and test file
got the same error ,not cuda error
would you introduce your code in brief ,thanks

[Hanjun-Dai]

If the error is happening in that line, you may double check the
https://github.com/Hanjun-Dai/GLN/blob/master/gln/mods/mol_gnn/gnn_family/utils.py#L64

note that different graph nn implementation will override this function.