Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

netcdf4-python debug installs #943

Open
astrojuanlu opened this issue Jun 30, 2019 · 4 comments
Open

netcdf4-python debug installs #943

astrojuanlu opened this issue Jun 30, 2019 · 4 comments

Comments

@astrojuanlu
Copy link
Contributor

I am trying to debug #814 and for that I would like to cythonize the sources with debugging information as suggested in https://cython.readthedocs.io/en/latest/src/userguide/debugging.html#debugging-your-cython-program. However, since afaf0fa#diff-2eeaed663bd0d25b7e608891384b7298, cythonize is not used and I don't see where to pass the option. Trying to reintroduce it in a naïve way:

diff --git a/setup.py b/setup.py
index c2a76da..c0b3869 100644
--- a/setup.py
+++ b/setup.py
@@ -2,6 +2,7 @@ import os, sys, subprocess
 import os.path as osp
 from setuptools import setup, Extension
 from distutils.dist import Distribution
+from Cython.Build import cythonize
 
 setuptools_extra_kwargs = {
     "install_requires": ["numpy>=1.7","cftime"],
@@ -609,5 +610,5 @@ setup(name="netCDF4",
                    "Topic :: System :: Archiving :: Compression",
                    "Operating System :: OS Independent"],
       packages=['netCDF4'],
-      ext_modules=ext_modules,
+      ext_modules=cythonize(ext_modules, gdb_debug=True),
       **setuptools_extra_kwargs)

results in an error:

Traceback (most recent call last):
  File "setup.py", line 613, in <module>
    ext_modules=cythonize(ext_modules, gdb_debug=True),
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 966, in cythonize
    aliases=aliases)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 824, in create_extension_list
    kwds = deps.distutils_info(file, aliases, base).values
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 679, in distutils_info
    return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 689, in transitive_merge
    node, extract, merge, seen, {}, self.cimported_files)[0]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 694, in transitive_merge_helper
    deps = extract(node)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 659, in distutils_info0
    cimports, externs, incdirs = self.cimports_externs_incdirs(filename)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Utils.py", line 54, in wrapper
    res = cache[args] = f(self, *args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 547, in cimports_externs_incdirs
    for include in self.included_files(filename):
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Utils.py", line 54, in wrapper
    res = cache[args] = f(self, *args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 529, in included_files
    include_path = self.context.find_include_file(include, None)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 285, in find_include_file
    error(pos, "'%s' not found" % filename)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Errors.py", line 178, in error
    raise InternalError(message)
Cython.Compiler.Errors.InternalError: Internal compiler error: 'constants.pyx' not found
@astrojuanlu
Copy link
Contributor Author

I saw that this was introduced in #673, which links to "issue #666", but I guess it's in some old issue tracker because it's a different thing.

If setup_requires was problematic, I wonder if using pyproject.toml as per PEP 518 would be a good replacement:

https://www.python.org/dev/peps/pep-0518/#build-system-table

@astrojuanlu
Copy link
Contributor Author

Turns out I was one step away:

diff --git a/setup.py b/setup.py
index c2a76da..e2aec1f 100644
--- a/setup.py
+++ b/setup.py
@@ -2,6 +2,7 @@ import os, sys, subprocess
 import os.path as osp
 from setuptools import setup, Extension
 from distutils.dist import Distribution
+from Cython.Build import cythonize
 
 setuptools_extra_kwargs = {
     "install_requires": ["numpy>=1.7","cftime"],
@@ -609,5 +610,5 @@ setup(name="netCDF4",
                    "Topic :: System :: Archiving :: Compression",
                    "Operating System :: OS Independent"],
       packages=['netCDF4'],
-      ext_modules=ext_modules,
+      ext_modules=cythonize(ext_modules, include_path=['include'], gdb_debug=True),
       **setuptools_extra_kwargs)

@astrojuanlu
Copy link
Contributor Author

As a last note, for some reason I was getting a really long compilation error when the cftime dependency was being resolved (similar to cython/cython#2730):

Searching for cftime
Reading https://pypi.org/simple/cftime/
Downloading https://files.pythonhosted.org/packages/7a/83/a61141ec141ceb0617468e04cc163dbdb9007b958191043618d1dc950b8f/cftime-1.0.3.4.tar.gz#sha256=dd74d0d470baf1c50e31335215793a5e78436903e34b4f151fa9ccbf3a6cc20c
Best match: cftime 1.0.3.4
Processing cftime-1.0.3.4.tar.gz
Writing /tmp/easy_install-jtfol7od/cftime-1.0.3.4/setup.cfg
Running cftime-1.0.3.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-jtfol7od/cftime-1.0.3.4/egg-dist-tmp-iuz36oic
/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py:752: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  elif isinstance(patterns, basestring) or not isinstance(patterns, collections.Iterable):
Compiling /tmp/easy_install-jtfol7od/cftime-1.0.3.4/cftime/_cftime.pyx because it changed.
[1/1] Cythonizing /tmp/easy_install-jtfol7od/cftime-1.0.3.4/cftime/_cftime.pyx
Traceback (most recent call last):
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2215, in visit_FuncDefNode
    node.body = node.body.analyse_expressions(node.local_scope)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in analyse_expressions
    for stat in self.stats]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in <listcomp>
    for stat in self.stats]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 6665, in analyse_expressions
    self.target = self.target.analyse_target_types(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 7480, in analyse_target_types
    coerced_unpacked_item = unpacked_item.coerce_to(arg.type, env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 952, in coerce_to
    src = PyTypeTestNode(src, dst_type, env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 12999, in __init__
    assert dst_type.is_extension_type or dst_type.is_builtin_type, "PyTypeTest on non extension type"
AssertionError: PyTypeTest on non extension type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-jtfol7od/cftime-1.0.3.4/setup.py", line 96, in <module>
    elif line.startswith('#define NC_HAS_PNETCDF'):
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
    cythonize_one(*args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1202, in cythonize_one
    result = compile_single(pyx_file, options, full_module_name=full_module_name)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 727, in compile_single
    return run_pipeline(source, options, full_module_name)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 515, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 355, in run_pipeline
    data = run(phase, data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 335, in run
    return phase(data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1597, in __call__
    return super_class.__call__(root)
  File "Cython/Compiler/Visitor.py", line 355, in Cython.Compiler.Visitor.EnvTransform.__call__
  File "Cython/Compiler/Visitor.py", line 302, in Cython.Compiler.Visitor.CythonTransform.__call__
  File "Cython/Compiler/Visitor.py", line 284, in Cython.Compiler.Visitor.VisitorTransform.__call__
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1608, in visit_ModuleNode
    self.visitchildren(node)
  File "Cython/Compiler/Visitor.py", line 247, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 251, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1903, in visit_DefNode
    node = self.visit_FuncDefNode(node)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1895, in visit_FuncDefNode
    node.body.analyse_declarations(lenv)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 431, in analyse_declarations
    stat.analyse_declarations(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 1329, in analyse_declarations
    base_type = self.base_type.analyse(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 1084, in analyse
    axes_specs = MemoryView.get_axes_specs(env, self.axes)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/MemoryView.py", line 533, in get_axes_specs
    cythonscope.load_cythonscope()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/CythonScope.py", line 128, in load_cythonscope
    whitelist=MemoryView.view_utility_whitelist)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 208, in declare_in_scope
    tree = self.get_tree(entries_only=True, cython_scope=cython_scope)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 185, in get_tree
    (err, tree) = Pipeline.run_pipeline(pipeline, tree, printtree=False)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 355, in run_pipeline
    data = run(phase, data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 335, in run
    return phase(data)
  File "Cython/Compiler/Visitor.py", line 302, in Cython.Compiler.Visitor.CythonTransform.__call__
  File "Cython/Compiler/Visitor.py", line 284, in Cython.Compiler.Visitor.VisitorTransform.__call__
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2210, in visit_ModuleNode
    self.visitchildren(node)
  File "Cython/Compiler/Visitor.py", line 247, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 251, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 188, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 148, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error
Cython.Compiler.Errors.CompilerCrash: 
Error compiling Cython file:
------------------------------------------------------------
...
                            __pyx_memoryview_thread_locks[__pyx_memoryview_thread_locks_used], __pyx_memoryview_thread_locks[i])
                    break
            else:
                PyThread_free_lock(self.lock)

    cdef char *get_item_pointer(memoryview self, object index) except NULL:
        ^
------------------------------------------------------------

View.MemoryView:389:9: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(View.MemoryView:5:0)
StatListNode.stats[18] = StatListNode(View.MemoryView:330:5)
StatListNode.stats[0] = CnameDecoratorNode(View.MemoryView:330:5,
    cname = '__pyx_memoryview')
CnameDecoratorNode.node = CClassDefNode(View.MemoryView:330:5,
    as_name = 'memoryview',
    class_name = 'memoryview',
    decorators = [...]/0,
    module_name = '',
    visibility = 'private')
CClassDefNode.body = StatListNode(View.MemoryView:332:4)
StatListNode.stats[2] = CFuncDefNode(View.MemoryView:389:9,
    args = [...]/2,
    is_c_class_method = 1,
    modifiers = [...]/0,
    visibility = 'private')

Compiler crash traceback from this point on:
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2215, in visit_FuncDefNode
    node.body = node.body.analyse_expressions(node.local_scope)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in analyse_expressions
    for stat in self.stats]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in <listcomp>
    for stat in self.stats]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 6665, in analyse_expressions
    self.target = self.target.analyse_target_types(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 7480, in analyse_target_types
    coerced_unpacked_item = unpacked_item.coerce_to(arg.type, env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 952, in coerce_to
    src = PyTypeTestNode(src, dst_type, env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 12999, in __init__
    assert dst_type.is_extension_type or dst_type.is_builtin_type, "PyTypeTest on non extension type"
AssertionError: PyTypeTest on non extension type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 613, in <module>
    **setuptools_extra_kwargs)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 418, in run
    self.easy_install(spec, not self.no_deps)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 660, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 707, in install_item
    self.process_distribution(spec, dist, deps)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 752, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
    replace_conflicting=replace_conflicting
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
    run_setup(setup_script, args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-jtfol7od/cftime-1.0.3.4/setup.py", line 96, in <module>
    elif line.startswith('#define NC_HAS_PNETCDF'):
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
    cythonize_one(*args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1202, in cythonize_one
    result = compile_single(pyx_file, options, full_module_name=full_module_name)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 727, in compile_single
    return run_pipeline(source, options, full_module_name)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 515, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 355, in run_pipeline
    data = run(phase, data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 335, in run
    return phase(data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1597, in __call__
    return super_class.__call__(root)
  File "Cython/Compiler/Visitor.py", line 355, in Cython.Compiler.Visitor.EnvTransform.__call__
  File "Cython/Compiler/Visitor.py", line 302, in Cython.Compiler.Visitor.CythonTransform.__call__
  File "Cython/Compiler/Visitor.py", line 284, in Cython.Compiler.Visitor.VisitorTransform.__call__
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1608, in visit_ModuleNode
    self.visitchildren(node)
  File "Cython/Compiler/Visitor.py", line 247, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 251, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1903, in visit_DefNode
    node = self.visit_FuncDefNode(node)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1895, in visit_FuncDefNode
    node.body.analyse_declarations(lenv)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 431, in analyse_declarations
    stat.analyse_declarations(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 1329, in analyse_declarations
    base_type = self.base_type.analyse(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 1084, in analyse
    axes_specs = MemoryView.get_axes_specs(env, self.axes)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/MemoryView.py", line 533, in get_axes_specs
    cythonscope.load_cythonscope()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/CythonScope.py", line 128, in load_cythonscope
    whitelist=MemoryView.view_utility_whitelist)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 208, in declare_in_scope
    tree = self.get_tree(entries_only=True, cython_scope=cython_scope)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 185, in get_tree
    (err, tree) = Pipeline.run_pipeline(pipeline, tree, printtree=False)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 355, in run_pipeline
    data = run(phase, data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 335, in run
    return phase(data)
  File "Cython/Compiler/Visitor.py", line 302, in Cython.Compiler.Visitor.CythonTransform.__call__
  File "Cython/Compiler/Visitor.py", line 284, in Cython.Compiler.Visitor.VisitorTransform.__call__
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2210, in visit_ModuleNode
    self.visitchildren(node)
  File "Cython/Compiler/Visitor.py", line 247, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 251, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 188, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 148, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error
setuptools.sandbox.UnpickleableException: CompilerCrash((<StringSourceDescriptor:View.MemoryView>, 389, 9), 'AnalyseExpressionsTransform', 'Compiler crash in AnalyseExpressionsTransform\n\nModuleNode.body = StatListNode(View.MemoryView:5:0)\nStatListNode.stats[18] = StatListNode(View.MemoryView:330:5)\nStatListNode.stats[0] = CnameDecoratorNode(View.MemoryView:330:5,\n    cname = \'__pyx_memoryview\')\nCnameDecoratorNode.node = CClassDefNode(View.MemoryView:330:5,\n    as_name = \'memoryview\',\n    class_name = \'memoryview\',\n    decorators = [...]/0,\n    module_name = \'\',\n    visibility = \'private\')\nCClassDefNode.body = StatListNode(View.MemoryView:332:4)\nStatListNode.stats[2] = CFuncDefNode(View.MemoryView:389:9,\n    args = [...]/2,\n    is_c_class_method = 1,\n    modifiers = [...]/0,\n    visibility = \'private\')\n\nCompiler crash traceback from this point on:\n  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2215, in visit_FuncDefNode\n    node.body = node.body.analyse_expressions(node.local_scope)\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in analyse_expressions\n    for stat in self.stats]\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in <listcomp>\n    for stat in self.stats]\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 6665, in analyse_expressions\n    self.target = self.target.analyse_target_types(env)\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 7480, in analyse_target_types\n    coerced_unpacked_item = unpacked_item.coerce_to(arg.type, env)\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 952, in coerce_to\n    src = PyTypeTestNode(src, dst_type, env)\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 12999, in __init__\n    assert dst_type.is_extension_type or dst_type.is_builtin_type, "PyTypeTest on non extension type"\nAssertionError: PyTypeTest on non extension type', AssertionError('PyTypeTest on non extension type'), <traceback object at 0x7fd872cd8d78>)

(perhaps because I'm using Python with debugging symbols enabled and I had to install numpy and cython from source?)

This was fixed by removing the setup_requires line and doing pip install cftime right before python setup.py install.

@ghost
Copy link

ghost commented Oct 7, 2019

@Juanlu001 this is the setup_requires sandbox vs cython clash most likely (also reported here: pypa/setuptools#1784 ) which happens when your own package uses Cython and has a setup_requires, rather than install_requires, dependency on another Cython-using package. One way to work around this is to get cftime package maintainers to call cythonize as subprocess.check_output([sys.executale, "-m", "Cython.Build.cythonize", ...]) in their setup.py/build_ext rather than importing it directly. Another way is to use a pyproject.toml with build-system.build_requires instead of setup_requires, even though that could cause issues if you ever plan an Android build for other reasons (it's complicated 😬 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant