KeyError: 'logPath' while running remote experiment

[apatsekin]

While trying to run remote training and then get some info about the status, I get an error from nnicli:

root# nnictl log trial
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.6.8/bin/nnictl", line 11, in <module>
    sys.exit(parse_args())
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/nni_cmd/nnictl.py", line 217, in parse_args
    args.func(args)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/nni_cmd/nnictl_utils.py", line 366, in log_trial
    trial_id_path_dict[trial['id']] = trial['logPath']
KeyError: 'logPath'

Brief what process you are following:

Running an experiment as described in MNIST remote example. It got stuck in a "waiting" status, so I'm trying to request a log - nnictl log trial

How to reproduce it:

nni Environment:

• nni version: 1.0
• nni mode(local|pai|remote): remote
• OS: Ubuntu 16.04.6 LTS
• python version: 3.6.8
• is conda or virtualenv used?: no
• is running in docker?: yes

need to update document(yes/no): no

Anything else we need to know:
This happens when I try to run experiment using remote mode, but from the same machine (trial and ctl are on the same machine). I see it starts to pack scripts dir with tar and then nothing happens. /tmp contains empty tar files (zero size). In Web UI status of trial keeps being waiting.

[SparkSnail]

Thanks to @apatsekin for raising the issue, this is a bug of NNI. We will fix this issue in #1550, please feel free to give feedback if you have any questions.