Snapshot on signal

[shelhamer]

Caffe should attempt to snapshot on SIGINT. Suggested by @longjon.

[seanbell]

I like this idea, though I think it would be helpful to have a way of easily disabling this behavior if it's the default (e.g. command-line flag). Otherwise, management scripts that call caffe now have to worry about extra disk-space consequences of stopping training early.

I think it's also worth letting users configure which signal code will trigger a snapshot (e.g. in solver.prototxt). The default could be SIGINT (2), but other codes such as SIGHUP (1) would let you signal a snapshot without interrupting training.

[shelhamer]

@seanbell agreed. All good points.

[beniz]

Is there a ticket to focus all thoughts on error handling within Caffe already ? If not, maybe it could make sense to create one with bullet points for sub-tasks. I would happily work on this after it's all been discussed and agreed upon so that no effort is wasted.

[jyegerlehner]

Ref PR 2253.

[shelhamer]

Solved by #2253