Exception handling in PyNEURON

[Helveg]

Currently NEURON frequently writes the exception message into stderr and throws the same RuntimeError: hoc error everywhere. It would be an added value if PyNEURON would handle this gracefully, using diverse Python exceptions instead with the error message encapsulated inside of it.

This allows for better and easier exception handling and can contribute to stability in environments where asking for permission is worse than asking for forgiveness (parallel computing e.g.).

[nrnhines]

This improvement would involve a two level approach. First, every (most? there are 518 of them) occurrence of
fprintf(stderr,... should be replaced by Fprintf(stderr,... as the latter redirects to the python stderr if python is being used. This, for example, means that in a jupyter notebook, the messages appear in the notebook. Second, I guess all (most? currently there are 94 of them) of the Fprintf(stderr should get a new name with an extra enum arg mapping to the relevant python exception. The second aspect may require some modification in approach to actually implement the desired behavior. A mere fprintf(stderr,is not itself an exception but often just some extra info
that was not convenient to fit into the subsequent hoc_execerror.

[nrnhines]

I should also mention that handleable exceptions that occur in the hoc world typically can only be handled when one enters
the hoc world from the python world via the nrn/src/nrnpyton/nrnpy_hoc.cpp:: fcall function which allows the hoc_execerror to return to fcall and raise the "hoc error". hoc_execerror that occurs when having entered the hoc world through any other path (eg, accessing section information), has to have
special guards around the hoc_execerror so that instead of hoc_execerror the system returns into the python world at the statement where it entered the hoc world but with some kind of return value or checkable hoc exception state which allows the python exception to be raised there. An example of
this is the classical nrn/src/nrnoc/cabcode.c:: Section* chk_access(void) which contains two hoc_execerror failure modes. From the python world I arrange that
Section* nrn_noerr_access(void) /* return 0 if no accessed section */ is called instead.

[Helveg]

When I redirect stderr and stdout most of the errors are captured, but sometimes parts of them leak through. Any idea how the message oc_restore_code tobj_count=1 should be 0 still reaches the console in this setup?

from neuron import h
t = h.Vector()
with redirect_stderr_and_stdout_to_dev_null():
  t.record(h._ref_t)

A similer one is obtained when you try h.NetCon(5,12): the NEURON: ... part of the error message is captured but bad stack access: expecting (double *); really (double) is still visible in the console.

Are they transmitted differently?

[nrnhines]

I don't know precisely why stdout/stdin are not the same in hoc and python if python changes them.
I suppose they start out the same and thereafter are independent.
What is your implementation of redirect_stderr_and_stdout_to_dev_null()
That is a printf (nrn/src/oc/code.c line 404). If you replace by Printf see if that fixes the problem.

[Helveg]

If called without arguments it writes to devnull, and by passing the stream argument a io.StringIO object I write the errors to a string.

import os, sys
from contextlib import contextmanager

@contextmanager
def _suppress_nrn(stream=None, close=False):
    if stream is None:
        stream = open(os.devnull, "w")
        close = True
    old_stdout = sys.stdout
    old_stderr = sys.stderr
    sys.stdout = stream
    sys.stderr = stream
    try:
        yield
    finally:
        if close:
            stream.close()
        sys.stdout = old_stdout
        sys.stderr = old_stderr

That is a printf (nrn/src/oc/code.c line 404). If you replace by Printf see if that fixes the problem.

I'll try to check that soon.

[Helveg]

That is a printf (nrn/src/oc/code.c line 404). If you replace by Printf see if that fixes the problem.

If I change printf to Printf I get the following warning:

/home/robin/nrn/src/nrnmpi/nrnmpi.c: In function ‘nrnmpi_init’:
/home/robin/nrn/src/nrnmpi/nrnmpi.c:164:9: warning: implicit declaration of function ‘Printf’; did you mean ‘printf’? [-Wimplicit-function-declaration]
         Printf("numprocs=%d\n", nrnmpi_numprocs_world);
         ^~~~~~
         printf

and finally an error:

../../lib/libnrniv.so: undefined reference to `Printf'

Is there something else I have to include?

Edit: I copy pasted some includes from another .c file that used Printf and that worked but I'm probably including too many things:

#include <math.h>
#include <nrnpython_config.h>
#include "section.h"
#include "nrniv_mf.h"
#include "membfunc.h"
#include "parse.h"
#include "hocparse.h"
#include "membdef.h"

So switching printf to Printf does remediate the problem! I can make a PR where I replace all these occurences. Atleast that way everything goes to the same stdout pipe.

For the exceptions I think I'll need a hands on example, it seems to be more complex than replacing some occurences ;p

[nrnhines]

Printf`` is defined in src/oc/hocdec.h as nrnpy_pr```. Turns out that hocdec.h only rarely is not included indirectly as it is included by oc2iv.h, section.h, oc_ansi.h, and hoc.h. And one or more of those are included by many other .h files.

[Helveg]

Generally the Printf switch seems to do the trick, but under MPI it does not resolve the issue.