Design solution for sweep and cell-level QC for ephys

[wbwakeman]

Plan and document a workflow for ephys sweep- and cell-level QC, and for feature extraction.
Present to team and stakeholders for approval.

[wbwakeman]

Jim helped Sergey and Wayne understand the background for this better. Very related to #278 .

QC can happen at the sweep, feature, and cell(ephys_roi_result) level.
Valid sweeps are used to compute features
Features are subject to the LIMS ephys_qc_criterias to determine if they pass or fail
Cells (ephys_roi_results) may be auto_failed, auto_passed based on the QC results for each feature.
A human could override this with: manual_failed, manual_passed.

In the pre-PatchSeq days, if a cell did not have all the passing features, it was essentially useless. Some of our infrastructure still reflects this idea.

In the PatchSeq world, since the cells will have gene expression data too, it is less critical that a cell has ALL passing features to be released. i.e. there is still valuable information to be released so long as the cell has AT LEAST ONE passing feature.

So, for the June data release, we should not release any cell with failed features, just because it is too late to make significant changes to the pipeline. However, in preparation for October data release, we should update the pipeline so that cells are not failed as long as they have at least one passing feature.

[wbwakeman]

Investigate whether adding an optional warning parameter instead of outright failing is straightforward, such that it would enable users (external and LIMs) to use the run_pipeline to run through to completion.

Report back to group if that appears to not be the cases.

[NileGraddis]

OK, I've written up some thoughts on this issue, the current system, and potential solutions here & in LIMS:
ipfx_fx_graceful_failure.zip

[sgratiy]

The decorator approach seems reasonable way of reusing the current system. Not sure why you define such a nested structure in the decorator though:

def safe_fn(fvalue_on_failure=None, strict=False):
    def safe_fn_decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
               ...

Should not the the following be enough:

def safe_fn(fn, value_on_failure=None, strict=False):
        def wrapper(*args, **kwargs):
               ...

[NileGraddis]

The problem is that we want to argue to the decorator, like this:

@my_deco(fish="fowl")
def my_fn(...

This syntax is equivalent to:

my_fn = my_deco(fish="fowl")(my_fn)

which means the return value of my_deco needs to itself be a decorator.

If we didn't want to support the decorator syntax we could indeed simplify.