Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new skan.summarize(separator="_") #986

Closed
ns-rse opened this issue Oct 29, 2024 · 4 comments · Fixed by #1052
Closed

Use new skan.summarize(separator="_") #986

ns-rse opened this issue Oct 29, 2024 · 4 comments · Fixed by #1052
Assignees
Labels
DNATracing Issues pertaining to the DNATracing class Medium Priority v2.3.0
Milestone

Comments

@ns-rse
Copy link
Collaborator

ns-rse commented Oct 29, 2024

We added a line in disrordered_tracing() to substitute the - in column names of dataframes produced by Skan.

A new version of skan has been released (19hrs ago) which will include the minor contribution I made of converting all variable names from - to _ (see release notes and linked pull request).

We should, until _ becomes the default, set skan.summarize(..., separator="_").

@ns-rse ns-rse added the DNATracing Issues pertaining to the DNATracing class label Oct 29, 2024
@ns-rse ns-rse self-assigned this Oct 29, 2024
@ns-rse
Copy link
Collaborator Author

ns-rse commented Oct 29, 2024

Discovered tests were failing on #985 with this new skan-0.12.0 though...

tests/tracing/test_disordered_tracing.py:1265:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
topostats/tracing/disordered_tracing.py:660: in disordered_trace_grain
    "branch_types": get_skan_image(
topostats/tracing/disordered_tracing.py:697: in get_skan_image
    skan_skeleton = skan.Skeleton(skeleton_image, spacing=1e-9, value_is_height=True)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/skan/csr.py:532: in __init__
    self.nbgraph = csr_to_nbgraph(graph, self.pixel_values)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/skan/csr.py:205: in csr_to_nbgraph
    return NBGraph(
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/experimental/jitclass/base.py:124: in __call__
    return cls._ctor(*bind.args[1:], **bind.kwargs)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:442: in _compile_for_args
    raise e
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:375: in _compile_for_args
    return_val = self.compile(tuple(argtypes))
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:905: in compile
    cres = self._compiler.compile(args, return_type)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:80: in compile
    status, retval = self._compile_cached(args, return_type)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:94: in _compile_cached
    retval = self._compile_core(args, return_type)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:107: in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:744: in compile_extra
    return pipeline.compile_extra(func)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:438: in compile_extra
    return self._compile_bytecode()
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:506: in _compile_bytecode
    return self._compile_core()
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:481: in _compile_core
    raise e
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:472: in _compile_core
    pm.run(self.state)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_machinery.py:364: in run
    raise e
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_machinery.py:356: in run
    self._runPass(idx, pass_inst, state)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_lock.py:35: in _acquire_compile_lock
    return func(*args, **kwargs)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_machinery.py:311: in _runPass
    mutated |= check(pss.run_pass, internal_state)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_machinery.py:273: in check
    mangled = func(compiler_state)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typed_passes.py:112: in run_pass
    typemap, return_type, calltypes, errs = type_inference_stage(
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typed_passes.py:93: in type_inference_stage
    errs = infer.propagate(raise_errors=raise_errors)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typeinfer.py:1083: in propagate
    errors = self.constraints.propagate(self)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typeinfer.py:182: in propagate
    raise e
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typeinfer.py:160: in propagate
    constraint(typeinfer)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typeinfer.py:583: in __call__
    self.resolve(typeinfer, typevars, fnty)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typeinfer.py:606: in resolve
    sig = typeinfer.resolve_call(fnty, pos_args, kw_args)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typeinfer.py:1577: in resolve_call
    return self.context.resolve_function_type(fnty, pos_args, kw_args)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typing/context.py:196: in resolve_function_type
    res = self._resolve_user_function_type(func, args, kws)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typing/context.py:248: in _resolve_user_function_type
    return func.get_call_type(self, args, kws)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/types/misc.py:441: in get_call_type
    return self.ctor_template(context).apply(args, kws)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typing/templates.py:350: in apply
    sig = generic(args, kws)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/experimental/jitclass/base.py:269: in generic
    sig = disp_type.get_call_type(self.context, boundargs, kws)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/types/functions.py:541: in get_call_type
    self.dispatcher.get_call_template(args, kws)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:318: in get_call_template
    self.compile(tuple(args))
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:905: in compile
    cres = self._compiler.compile(args, return_type)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:80: in compile
    status, retval = self._compile_cached(args, return_type)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:94: in _compile_cached
    retval = self._compile_core(args, return_type)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/dispatcher.py:107: in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:744: in compile_extra
    return pipeline.compile_extra(func)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:438: in compile_extra
    return self._compile_bytecode()
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:506: in _compile_bytecode
    return self._compile_core()
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:481: in _compile_core
    raise e
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler.py:472: in _compile_core
    pm.run(self.state)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_machinery.py:364: in run
    raise e
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_machinery.py:356: in run
    self._runPass(idx, pass_inst, state)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_lock.py:35: in _acquire_compile_lock
    return func(*args, **kwargs)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_machinery.py:311: in _runPass
    mutated |= check(pss.run_pass, internal_state)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/compiler_machinery.py:273: in check
    mangled = func(compiler_state)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/typed_passes.py:468: in run_pass
    lower.lower()
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/lowering.py:187: in lower
    self.lower_normal_function(self.fndesc)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/lowering.py:226: in lower_normal_function
    entry_block_tail = self.lower_function_body()
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/lowering.py:256: in lower_function_body
    self.lower_block(block)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/lowering.py:270: in lower_block
    self.lower_inst(inst)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/lowering.py:564: in lower_inst
    return impl(self.builder, (target, value))
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/base.py:627: in wrapped
    return impl(self, builder, sig, args, attr)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/imputils.py:166: in res
    return real_impl(context, builder, sig, args, attr)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/experimental/jitclass/base.py:505: in set_attr_impl
    setattr(data, _mangle_attr(attr), val)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/cgutils.py:164: in __setattr__
    self[self._datamodel.get_field_position(field)] = value
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/cgutils.py:178: in __setitem__
    value = self._cast_member_from_value(index, value)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/cgutils.py:251: in _cast_member_from_value
    return model.as_data(self._builder, val)
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/numba/core/datamodel/models.py:586: in as_data
    struct = builder.insert_value(struct, el, [i])
/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/llvmlite/ir/builder.py:983: in insert_value
    instr = instructions.InsertValue(self.block, agg, value, idx, name=name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <[AttributeError("'InsertValue' object has no attribute '_name'") raised in repr()] InsertValue object at 0x7ccfbd4f6b90>, parent = <ir.Block 'B0' of type 'label'>
agg = <ir.InsertValue '.258' of type '{i8*, i8*, i64, i64, double*, [1 x i64], [1 x i64]}', opname 'insertvalue', operands [...value=<llvmlite.ir.values._Undefined object at 0x7cd0142cda50>>, <ir.Argument 'arg.node_props.6.0' of type i64>]>]>]>]>
elem = <ir.ExtractValue 'extracted.data.22' of type 'float*', opname 'extractvalue', operands [<ir.InsertValue 'inserted.stri...' value=<llvmlite.ir.values._Undefined object at 0x7cd0142cda50>>, <ir.Argument 'arg.node_props.6.0' of type i64>]>]>]>
indices = [4], name = ''

    def __init__(self, parent, agg, elem, indices, name=''):
        typ = agg.type
        try:
            for i in indices:
                typ = typ.elements[i]
        except (AttributeError, IndexError):
            raise TypeError("Can't index at %r in %s"
                            % (list(indices), agg.type))
        if elem.type != typ:
>           raise TypeError("Can only insert %s at %r in %s: got %s"
                            % (typ, list(indices), agg.type, elem.type))
E           TypeError: Can only insert double* at [4] in {i8*, i8*, i64, i64, double*, [1 x i64], [1 x i64]}: got float*

/home/neil/.virtualenvs/topostats-311b/lib/python3.11/site-packages/llvmlite/ir/instructions.py:702: TypeError
__________________________________________________________________________________________ tests/test_run_topostats.py ___________________________________________________________________________________________

Its related to Numba and type errors. Will see if I can fix upstream before making these changes locally.

@ns-rse
Copy link
Collaborator Author

ns-rse commented Oct 30, 2024

This should await the release of skan-0.12.1 which hopefully won't be too long as I found the source of the problem and have submitted a PR to fix it (see related issue report).

@ns-rse
Copy link
Collaborator Author

ns-rse commented Dec 11, 2024

skan-0.12.1 has been released but does not include the proposed fix as its awaiting me extracting the data that caused the error and writing a test ensure there is no regression in the future. Might try doing that next week as we can then unpin the version we depend on and remove the substitution that is required to be done manually.

@ns-rse ns-rse added the v2.3.0 label Dec 16, 2024
@ns-rse
Copy link
Collaborator Author

ns-rse commented Dec 16, 2024

skan-0.12.2 published which includes the bug fix I made to address the above issue.

This means we should be able to...

  1. unpin dependency of skan in pyproject.toml.
  2. Remove the code that was added to handle . in variable names pending the release of v0.12.0 which included my switch to using _.

@ns-rse ns-rse added this to the v2.3.0 milestone Dec 16, 2024
ns-rse added a commit that referenced this issue Dec 16, 2024
Closes #986

With the release of `skan-0.12.2` we can remove the code which converts `-` in the variable names of Pandas DataFrames
to instead use `_` which allows the use of "dot-notation" when referring to the variables of data frames.

We therefore unpin the `skan` version in `pyproject.toml`, explicitly state the `separator="_"` argument to
`skan.summarize()` and adapt the hard-coded variable names appropriately. As noted in the [release
notes](https://github.com/jni/skan/releases/tag/v0.12.2) the default will be `_` from `v0.13.0` and the flag allows
backwards compatibility but since we have gone with the use of `_` in variable names it will do no harm leaving these in
place (although they could be removed when `skan-v0.13.0` is released).
ns-rse added a commit that referenced this issue Dec 16, 2024
In addressing #986 I tested in a fresh virtual environment. This pulled in
[matplotlib-3.10.0](https://github.com/matplotlib/matplotlib/releases/tag/v3.10.0) which was released 2024-12-14 (two
days ago!). Our dependency on [topoly](topoly.cent.uw.edu.pl/documentation.html) doesn't work with this new version
because of the removal of `inset_location.InsetPosition` in favour of `inset_axes`.

This has been reported to the Topoly developers, see [ImportError: cannot import name 'InsetPosition' from
'mpl_toolkits.axes_grid1.inset_locator' · Issue #5 ·
ilbsm/topoly_tutorial](ilbsm/topoly_tutorial#5) and hopefully won't take long to resolve. For
now though we pin our dependency `matplotlib~=3.9.4` and all tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DNATracing Issues pertaining to the DNATracing class Medium Priority v2.3.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant