Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opaque Numba compilation error messages #36

Open
pamvc opened this issue Jan 15, 2021 · 4 comments
Open

Opaque Numba compilation error messages #36

pamvc opened this issue Jan 15, 2021 · 4 comments

Comments

@pamvc
Copy link

pamvc commented Jan 15, 2021

Hello my name is Pamela Vazquez, a current wintern designing graph algorithms in order to get a graphs' average neighbor degree and its degree connectivity.

During my time of testing, I have run into error messages that left me confused just because they were long and convoluted.
For example, when I tried to test my code with this order of parameters:

def sum_neighbor_degree**(graph: PropertyGraph, nid, result_array, deg_array, weight)**:
    # fill in map that will hold the results key = node id and value = avg_neighbor_deg
    sum_neighbor_degree = 0
    #for edge connected to curr node: 
    for edge in graph.edges(nid):

I received this error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/lonestar/analytics/average_neighbor_degree.py", line 68, in average_neighbor_degree
    result_array = helper(graph, deg_array, weight)
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/lonestar/analytics/average_neighbor_degree.py", line 45, in helper
    do_all(
  File "_loops.pyx", line 2373, in galois._loops.do_all
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 258, in instantiate
    return self._builder.bind(self._args, unbound_argument_types)
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 216, in bind
    inst = self._generate(arg_types, unbound_argument_types)
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 207, in _generate
    inst = _ClosureInstance(
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 74, in __init__
    wrapper = self._build_wrapper(func, load_struct, return_type, bound_args, unbound_args)
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 118, in _build_wrapper
    exec(src, exec_glbls)
  File "<string>", line 3, in <module>
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/galois_compiler.py", line 82, in wrapper
    res.compile()
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/ccallback.py", line 67, in compile
    cres = self._compile_uncached()
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/ccallback.py", line 81, in _compile_uncached
    return self._compiler.compile(sig.args, sig.return_type)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/dispatcher.py", line 81, in compile
    raise retval
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/dispatcher.py", line 91, in _compile_cached
    retval = self._compile_core(args, return_type)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/dispatcher.py", line 104, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler.py", line 602, in compile_extra
    return pipeline.compile_extra(func)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler.py", line 352, in compile_extra
    return self._compile_bytecode()
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler.py", line 414, in _compile_bytecode
    return self._compile_core()
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler.py", line 394, in _compile_core
    raise e
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler.py", line 385, in _compile_core
    pm.run(self.state)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 339, in run
    raise patched_exception
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 330, in run
    self._runPass(idx, pass_inst, state)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 289, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 262, in check
    mangled = func(compiler_state)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/typed_passes.py", line 94, in run_pass
    typemap, return_type, calltypes = type_inference_stage(
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/typed_passes.py", line 72, in type_inference_stage
    infer.propagate(raise_errors=raise_errors)
  File "/home/pamvazquez/miniconda3/envs/galois-dev/lib/python3.8/site-packages/numba/core/typeinfer.py", line 1071, in propagate
    raise errors[0]
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
- Resolution failure for literal arguments:
No implementation of function Function(<function overload_PropertyGraph_edges at 0x7ef49574daf0>) found for signature:

 >>> overload_PropertyGraph_edges(PropertyGraph, reflected list(int64)<iv=None>)

**------> This was the indicator in the error message saying that my parameters were in the wrong order**

There are 2 candidate implementations:
  - Of which 2 did not match due to:
  Overload of function 'overload_PropertyGraph_edges': File: ../../../../../../katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/galois.py: Line 31.
    With argument(s): '(PropertyGraph, reflected list(int64)<iv=None>)':
   No match.

- Resolution failure for non-literal arguments:
None

During: resolving callee type: BoundFunction((<class 'galois.property_graph.PropertyGraph<numba type>'>, 'edges') for PropertyGraph)
During: typing of call at /home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/lonestar/analytics/average_neighbor_degree.py (20)


File "_skbuild/linux-x86_64-3.8/cmake-install/python/galois/lonestar/analytics/average_neighbor_degree.py", line 20:
def sum_neighbor_degree(graph: PropertyGraph, nid, result_array, deg_array, weight):
    <source elided>
    #for edge connected to curr node: 
    for edge in graph.edges(nid):
    ^

During: resolving callee type: type(CPUDispatcher(<function sum_neighbor_degree at 0x7ef495644dc0>))
During: typing of call at <string> (5)

During: resolving callee type: type(CPUDispatcher(<function sum_neighbor_degree at 0x7ef495644dc0>))
During: typing of call at <string> (5)


File "<string>", line 5:
<source missing, REPL/exec in use?>

As a newbie in python, I was confused and asked for help from Arthur. Which resulted in recognizing that the error was simply from having my parameters in the wrong order, but it was hard to tell from the opaque error message.

Thank you for your time!

@ddn0
Copy link
Contributor

ddn0 commented Jan 15, 2021

I formatted post a little bit to make it easier to read. Hint: you can use triple-backtick like so:

```
stuff I want fixed format text
```

to mark blocks of code or computer output.

I do agree that it can be confusing to read a long error message. It doesn't make it easier when some tools put the most important stuff at the beginning and others put the most important information towards the end.

@pamvc
Copy link
Author

pamvc commented Jan 15, 2021 via email

@arthurp
Copy link
Contributor

arthurp commented Jan 16, 2021

I do agree that it can be confusing to read a long error message. It doesn't make it easier when some tools put the most important stuff at the beginning and others put the most important information towards the end.

And to make things worse the most important part here is overload_PropertyGraph_edges(PropertyGraph, reflected list(int64)<iv=None>) and it's in the middle. Numba has the worst error messages.

I think I might need to push some changes up stream to allow better error messages. As far as I know there isn't really a way to make type errors produce simple messages for user defined functions (like we add). I guess this leads to a question:

@pamvc Could you have debugged this yourself if it had said the following?

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/lonestar/analytics/average_neighbor_degree.py", line 68, in average_neighbor_degree
    result_array = helper(graph, deg_array, weight)
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/lonestar/analytics/average_neighbor_degree.py", line 45, in helper
    do_all(
  File "_loops.pyx", line 2373, in galois._loops.do_all
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 258, in instantiate
    return self._builder.bind(self._args, unbound_argument_types)
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 216, in bind
    inst = self._generate(arg_types, unbound_argument_types)
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 207, in _generate
    inst = _ClosureInstance(
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 74, in __init__
    wrapper = self._build_wrapper(func, load_struct, return_type, bound_args, unbound_args)
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/closure.py", line 118, in _build_wrapper
    exec(src, exec_glbls)
  File "<string>", line 3, in <module>
  File "/home/pamvazquez/katana/_skbuild/linux-x86_64-3.8/cmake-install/python/galois/numba_support/galois_compiler.py", line 82, in wrapper
    res.compile()
numba.core.errors.TypingError:: Argument 1 of PropertyGraph.edges must be int. Was reflected list(int64). At:
File "_skbuild/linux-x86_64-3.8/cmake-install/python/galois/lonestar/analytics/average_neighbor_degree.py", line 20:
def sum_neighbor_degree(graph: PropertyGraph, nid, result_array, deg_array, weight):
    <source elided>
    #for edge connected to curr node: 
    for edge in graph.edges(nid):
    ^

Making this happen should be possible with the available information. The changes needed would be:

  • Track methods, so numba can report the method instead of the underlying function that implements it.
  • Prune the stack to remove python calls inside the numba compiler.
  • Simplify the error message to include less information about the internals of the compiler. (All the information currently included is still needed for debugging the compiler and plugins to it, but it can be disabled by default.)

@pamvc
Copy link
Author

pamvc commented Jan 17, 2021 via email

@arthurp arthurp changed the title Opaque Error Messages Opaque Numba compilation error messages Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants