Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot import cell2location: Failed in nopython mode pipeline (step: native lowering) #142

Closed
2 tasks done
98xy opened this issue Apr 8, 2022 · 7 comments
Closed
2 tasks done

Cannot import cell2location: Failed in nopython mode pipeline (step: native lowering) #142

98xy opened this issue Apr 8, 2022 · 7 comments
Labels
bug Something isn't working

Comments

@98xy
Copy link

98xy commented Apr 8, 2022 •
edited
Loading

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Minimal code sample (that we can run without your data, using public data)

This issue came up quite suddenly and it worked normally before. No library-related changes have been made.

# Your code here
import cell2location
[Paste the error output produced by the above code here]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/errors.py:823, in new_error_context(fmt_, *args, **kwargs)
    822 try:
--> 823     yield
    824 except NumbaError as e:

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:293, in BaseLower.lower_block(self, block)
    291     with new_error_context('lowering "{inst}" at {loc}', inst=inst,
    292                            loc=self.loc, errcls_=defaulterrcls):
--> 293         self.lower_inst(inst)
    294 self.post_block(block)

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:438, in Lower.lower_inst(self, inst)
    437 ty = self.typeof(inst.target.name)
--> 438 val = self.lower_assign(ty, inst)
    439 argidx = None

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:624, in Lower.lower_assign(self, ty, inst)
    623 elif isinstance(value, ir.Expr):
--> 624     return self.lower_expr(ty, value)
    626 elif isinstance(value, ir.Var):

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:1159, in Lower.lower_expr(self, resty, expr)
   1158 elif expr.op == 'call':
-> 1159     res = self.lower_call(resty, expr)
   1160     return res

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:889, in Lower.lower_call(self, resty, expr)
    888 else:
--> 889     res = self._lower_call_normal(fnty, expr, signature)
    891 # If lowering the call returned None, interpret that as returning dummy
    892 # value if the return type of the function is void, otherwise there is
    893 # a problem

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:1130, in Lower._lower_call_normal(self, fnty, expr, signature)
   1128     argvals = [the_self] + list(argvals)
-> 1130 res = impl(self.builder, argvals, self.loc)
   1131 return res

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/base.py:1201, in _wrap_impl.__call__(self, builder, args, loc)
   1200 def __call__(self, builder, args, loc=None):
-> 1201     res = self._imp(self._context, builder, self._sig, args, loc=loc)
   1202     self._context.add_linking_libs(getattr(self, 'libs', ()))

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/base.py:1231, in _wrap_missing_loc.__call__.<locals>.wrapper(*args, **kwargs)
   1230 kwargs.pop('loc')     # drop unused loc
-> 1231 return fn(*args, **kwargs)

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/cpython/rangeobj.py:40, in make_range_impl.<locals>.range1_impl(context, builder, sig, args)
     39 state.start = context.get_constant(int_type, 0)
---> 40 state.stop = stop
     41 state.step = context.get_constant(int_type, 1)

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/cgutils.py:164, in _StructProxy.__setattr__(self, field, value)
    163     return super(_StructProxy, self).__setattr__(field, value)
--> 164 self[self._datamodel.get_field_position(field)] = value

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/cgutils.py:188, in _StructProxy.__setitem__(self, index, value)
    187     else:
--> 188         raise TypeError("Invalid store of {value.type} to "
    189                         "{ptr.type.pointee} in "
    190                         "{self._datamodel} "
    191                         "(trying to write member #{index})"
    192                         .format(value=value, ptr=ptr, self=self,
    193                                 index=index))
    194 self._builder.store(value, ptr)

TypeError: Invalid store of i64 to i32 in <numba.core.datamodel.models.RangeModel object at 0x7fd85018ed30> (trying to write member #1)

During handling of the above exception, another exception occurred:

LoweringError                             Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 import cell2location

File /data/xy/anaconda3/lib/python3.8/site-packages/cell2location/__init__.py:5, in <module>
      2 from pyro.distributions.transforms import SoftplusTransform
      3 from torch.distributions import biject_to, transform_to
----> 5 from .run_c2l import run_cell2location
      6 from .run_colocation import run_colocation
      7 from .run_regression import run_regression

File /data/xy/anaconda3/lib/python3.8/site-packages/cell2location/run_c2l.py:18, in <module>
     15 import scipy
     16 import theano
---> 18 import cell2location.models as models
     19 import cell2location.plt as c2lpl
     20 from cell2location.cluster_averages import compute_cluster_averages, select_features

File /data/xy/anaconda3/lib/python3.8/site-packages/cell2location/models/__init__.py:1, in <module>
----> 1 from ._cell2location_model import Cell2location
      2 from ._cell2location_module import (
      3     LocationModelLinearDependentWMultiExperimentLocationBackgroundNormLevelGeneAlphaPyroModel,
      4 )
      5 from .downstream import CoLocatedGroupsSklearnNMF

File /data/xy/anaconda3/lib/python3.8/site-packages/cell2location/models/_cell2location_model.py:21, in <module>
     19 from cell2location.models.base._pyro_base_loc_module import Cell2locationBaseModule
     20 from cell2location.models.base._pyro_mixin import PltExportMixin, QuantileMixin
---> 21 from cell2location.utils import select_slide
     24 class Cell2location(QuantileMixin, PyroSampleMixin, PyroSviTrainMixin, PltExportMixin, BaseModelClass):
     25     r"""
     26     Cell2location model. User-end model class. See Module class for description of the model (incl. math).
     27 
   (...)
     42     >>>
     43     """

File /data/xy/anaconda3/lib/python3.8/site-packages/cell2location/utils/__init__.py:5, in <module>
      1 import sys
      3 import numpy as np
----> 5 from ._spatial_knn import spatial_neighbours, sum_neighbours
      6 from .filtering import filter_genes
      9 def select_slide(adata, s, batch_key="sample"):

File /data/xy/anaconda3/lib/python3.8/site-packages/cell2location/utils/_spatial_knn.py:5, in <module>
      3 from scipy.spatial import cKDTree
      4 from sklearn.neighbors import KDTree
----> 5 from umap.umap_ import fuzzy_simplicial_set
      8 def get_sparse_matrix_from_indices_distances_umap(knn_indices, knn_dists, n_obs, n_neighbors):
      9     """
     10     Copied out of scanpy.neighbors
     11     """

File /data/xy/anaconda3/lib/python3.8/site-packages/umap/__init__.py:1, in <module>
----> 1 from .umap_ import UMAP
      3 # Workaround: https://github.com/numba/numba/issues/3341
      4 import numba

File /data/xy/anaconda3/lib/python3.8/site-packages/umap/umap_.py:54, in <module>
     52 from umap.spectral import spectral_layout
     53 from umap.utils import deheap_sort, submatrix
---> 54 from umap.layouts import (
     55     optimize_layout_euclidean,
     56     optimize_layout_generic,
     57     optimize_layout_inverse,
     58 )
     60 try:
     61     # Use pynndescent, if installed (python 3 only)
     62     from pynndescent import NNDescent

File /data/xy/anaconda3/lib/python3.8/site-packages/umap/layouts.py:39, in <module>
     25     else:
     26         return val
     29 @numba.njit(
     30     "f4(f4[::1],f4[::1])",
     31     fastmath=True,
     32     cache=True,
     33     locals={
     34         "result": numba.types.float32,
     35         "diff": numba.types.float32,
     36         "dim": numba.types.int32,
     37     },
     38 )
---> 39 def rdist(x, y):
     40     """Reduced Euclidean distance.
     41 
     42     Parameters
   (...)
     49     The squared euclidean distance between x and y
     50     """
     51     result = 0.0

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/decorators.py:219, in _jit.<locals>.wrapper(func)
    217     with typeinfer.register_dispatcher(disp):
    218         for sig in sigs:
--> 219             disp.compile(sig)
    220         disp.disable_compile()
    221 return disp

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/dispatcher.py:965, in Dispatcher.compile(self, sig)
    963 with ev.trigger_event("numba:compile", data=ev_details):
    964     try:
--> 965         cres = self._compiler.compile(args, return_type)
    966     except errors.ForceLiteralArg as e:
    967         def folded(args, kws):

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/dispatcher.py:125, in _FunctionCompiler.compile(self, args, return_type)
    124 def compile(self, args, return_type):
--> 125     status, retval = self._compile_cached(args, return_type)
    126     if status:
    127         return retval

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/dispatcher.py:139, in _FunctionCompiler._compile_cached(self, args, return_type)
    136     pass
    138 try:
--> 139     retval = self._compile_core(args, return_type)
    140 except errors.TypingError as e:
    141     self._failed_cache[key] = e

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/dispatcher.py:152, in _FunctionCompiler._compile_core(self, args, return_type)
    149 flags = self._customize_flags(flags)
    151 impl = self._get_implementation(args, {})
--> 152 cres = compiler.compile_extra(self.targetdescr.typing_context,
    153                               self.targetdescr.target_context,
    154                               impl,
    155                               args=args, return_type=return_type,
    156                               flags=flags, locals=self.locals,
    157                               pipeline_class=self.pipeline_class)
    158 # Check typing error if object mode is used
    159 if cres.typing_error is not None and not flags.enable_pyobject:

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler.py:693, in compile_extra(typingctx, targetctx, func, args, return_type, flags, locals, library, pipeline_class)
    669 """Compiler entry point
    670 
    671 Parameter
   (...)
    689     compiler pipeline
    690 """
    691 pipeline = pipeline_class(typingctx, targetctx, library,
    692                           args, return_type, flags, locals)
--> 693 return pipeline.compile_extra(func)

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler.py:429, in CompilerBase.compile_extra(self, func)
    427 self.state.lifted = ()
    428 self.state.lifted_from = None
--> 429 return self._compile_bytecode()

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler.py:497, in CompilerBase._compile_bytecode(self)
    493 """
    494 Populate and run pipeline for bytecode input
    495 """
    496 assert self.state.func_ir is None
--> 497 return self._compile_core()

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler.py:476, in CompilerBase._compile_core(self)
    474         self.state.status.fail_reason = e
    475         if is_final_pipeline:
--> 476             raise e
    477 else:
    478     raise CompilerError("All available pipelines exhausted")

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler.py:463, in CompilerBase._compile_core(self)
    461 res = None
    462 try:
--> 463     pm.run(self.state)
    464     if self.state.cr is not None:
    465         break

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler_machinery.py:353, in PassManager.run(self, state)
    350 msg = "Failed in %s mode pipeline (step: %s)" % \
    351     (self.pipeline_name, pass_desc)
    352 patched_exception = self._patch_error(msg, e)
--> 353 raise patched_exception

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler_machinery.py:341, in PassManager.run(self, state)
    339 pass_inst = _pass_registry.get(pss).pass_inst
    340 if isinstance(pass_inst, CompilerPass):
--> 341     self._runPass(idx, pass_inst, state)
    342 else:
    343     raise BaseException("Legacy pass in use")

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler_lock.py:35, in _CompilerLock.__call__.<locals>._acquire_compile_lock(*args, **kwargs)
     32 @functools.wraps(func)
     33 def _acquire_compile_lock(*args, **kwargs):
     34     with self:
---> 35         return func(*args, **kwargs)

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler_machinery.py:296, in PassManager._runPass(self, index, pss, internal_state)
    294     mutated |= check(pss.run_initialization, internal_state)
    295 with SimpleTimer() as pass_time:
--> 296     mutated |= check(pss.run_pass, internal_state)
    297 with SimpleTimer() as finalize_time:
    298     mutated |= check(pss.run_finalizer, internal_state)

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/compiler_machinery.py:269, in PassManager._runPass.<locals>.check(func, compiler_state)
    268 def check(func, compiler_state):
--> 269     mangled = func(compiler_state)
    270     if mangled not in (True, False):
    271         msg = ("CompilerPass implementations should return True/False. "
    272                "CompilerPass with name '%s' did not.")

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/typed_passes.py:394, in NativeLowering.run_pass(self, state)
    391 with targetctx.push_code_library(library):
    392     lower = lowering.Lower(targetctx, library, fndesc, interp,
    393                            metadata=metadata)
--> 394     lower.lower()
    395     if not flags.no_cpython_wrapper:
    396         lower.create_cpython_wrapper(flags.release_gil)

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:196, in BaseLower.lower(self)
    194 if self.generator_info is None:
    195     self.genlower = None
--> 196     self.lower_normal_function(self.fndesc)
    197 else:
    198     self.genlower = self.GeneratorLower(self)

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:250, in BaseLower.lower_normal_function(self, fndesc)
    248 # Init argument values
    249 self.extract_function_arguments()
--> 250 entry_block_tail = self.lower_function_body()
    252 # Close tail of entry block, do not emit debug metadata else the
    253 # unconditional jump gets associated with the metadata from the function
    254 # body end.
    255 with debuginfo.suspend_emission(self.builder):

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:279, in BaseLower.lower_function_body(self)
    277     bb = self.blkmap[offset]
    278     self.builder.position_at_end(bb)
--> 279     self.lower_block(block)
    280 self.post_lower()
    281 return entry_block_tail

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/lowering.py:293, in BaseLower.lower_block(self, block)
    290     defaulterrcls = partial(LoweringError, loc=self.loc)
    291     with new_error_context('lowering "{inst}" at {loc}', inst=inst,
    292                            loc=self.loc, errcls_=defaulterrcls):
--> 293         self.lower_inst(inst)
    294 self.post_block(block)

File /data/xy/anaconda3/lib/python3.8/contextlib.py:131, in _GeneratorContextManager.__exit__(self, type, value, traceback)
    129     value = type()
    130 try:
--> 131     self.gen.throw(type, value, traceback)
    132 except StopIteration as exc:
    133     # Suppress StopIteration *unless* it's the same exception that
    134     # was passed to throw().  This prevents a StopIteration
    135     # raised inside the "with" statement from being suppressed.
    136     return exc is not value

File /data/xy/anaconda3/lib/python3.8/site-packages/numba/core/errors.py:837, in new_error_context(fmt_, *args, **kwargs)
    835     else:
    836         tb = None
--> 837     raise newerr.with_traceback(tb)
    838 elif use_new_style_errors():
    839     raise e

LoweringError: Failed in nopython mode pipeline (step: native lowering)
Invalid store of i64 to i32 in <numba.core.datamodel.models.RangeModel object at 0x7fd85018ed30> (trying to write member #1)

File "../../../../anaconda3/lib/python3.8/site-packages/umap/layouts.py", line 53:
def rdist(x, y):
    <source elided>
    dim = x.shape[0]
    for i in range(dim):
    ^

During: lowering "$20call_function.7 = call $16load_global.5(dim, func=$16load_global.5, args=[Var(dim, layouts.py:52)], kws=(), vararg=None, target=None)" at /data/xy/anaconda3/lib/python3.8/site-packages/umap/layouts.py (53)
@98xy 98xy added the bug Something isn't working label Apr 8, 2022
@vitkl
Copy link
Contributor

vitkl commented Apr 13, 2022 •
edited
Loading

Hi @98xy

We have just updated cell2location to the latest scvi-tools version (0.15.3+). Please try re-creating a clean conda environment. The error you observe looks like umap installation version issues. Given that cell2location passes GitHub tests, the problem with UMAP can be specific to your platform.

@98xy
Copy link
Author

98xy commented Apr 17, 2022

Hi @98xy

We have just updated cell2location to the latest scvi-tools version (0.15.3+). Please try re-creating a clean conda environment. The error you observe looks like umap installation version issues. Given that cell2location passes GitHub tests, the problem with UMAP can be specific to your platform.

Hi @vitkl
Thanks for your help! I have re-created a clean conda environment but the error is still exist.

@vitkl
Copy link
Contributor

vitkl commented Apr 17, 2022 via email •
edited
Loading

@98xy
Copy link
Author

98xy commented Apr 17, 2022 via email

This was referenced Apr 17, 2022
Closed
Closed
@vitkl
Copy link
Contributor

vitkl commented Apr 17, 2022 •
edited
Loading

This is a temporary error with JAX (scvi-tools dependency) - as discussed here: #144

I think you can solve this by installing older jax and jaxlib:

pip install jax>=0.3,<=0.3.6 jaxlib>=0.1.65,<=0.3.5

@98xy
Copy link
Author

98xy commented Apr 17, 2022 via email

@MertDemirdizen MertDemirdizen mentioned this issue Apr 17, 2022
Closed
@vitkl
Copy link
Contributor

vitkl commented May 4, 2022 •
edited
Loading

This does not seem to be a problem anymore (JAX fixed this). If you see similar issues with JAX in the future - could be good to try installing the previous version or wait for a few days until the bug is fixed.

@vitkl vitkl closed this as completed May 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vitkl @98xy