Bypass bessel filter for Chirp sweeps sampled at 12.5K

[ru57y34nn]

Describe the use case that is addressed by this feature.
The inclusion of our new Chirp stimulus sets is causing feature extraction to fail because we are collecting them at a sampling rate of 12.5 K and IPFX has the Bessel filter set to 10 K for all current clamp sweeps.

Detailed description:

We have new chirp stimulus sets with slightly different names than the previous names; C2CHIRPA180503, C2CHIRPB180503, C2CHIRPC180503, C2CHIRPD180503 (we have added ‘A’, ‘B’, ‘C’, and ‘D’ between C2CHIRP and the date, 180503 as they are each ran at different holding potentials.

We are running these chirp sweeps at a sampling rate of 12.5 K with our bessel filter set to 6 K, but this is causing issues with IPFX and causing feature extraction to crash. The issue appears to stem from the call to data_set_features.extract_data_set_features(), which collects all current clamp sweeps and then runs them through data_set_features.extract_sweep_features() to get sweep_features (dictionary) and features_states[‘sweep_features_state’] (dictionary). The extract_sweep_features() function calls data_set_features.extractors_for_sweeps() to create a feature_extractor.SpikeFeatureExtractor object for each sweep then and calls the process() method from the class instance which calls time_series_utils.calculate_dvdt(), and this is where the issue is coming from.

The filter parameter is preset to 10.0 in the SpikeFeatureExtractor class, so the filter coefficient (filter_coeff in time_series_utils.calculate_dvdt()) is being determined to be outside of the acceptable range, which is 0 < filter_coeff <= 1, and throws a ValueError with the message “Bessel coeff 1.6 is outside of valid range [0,1); cannot filter sampling frequency 12.5 KHz with cutoff frequency 10 KHz”. This leaves sweep_features as an empty dictionary and the feature_states[‘sweep_features_state’] dictionary has ‘failed_fx’ set to True and the ‘fail_fx_message’ contains the ValueError message.

The filter_coeff is calculated as (filter * 1000) / (samp_rate / 2), which with a bessel fitler of 10,000 and a sampling rate of 12.5K comes to a filter_coeff of 1.6. The solution here is to just bypass the bessel filter by setting the filter parameter to None when creating the SpikeFeatureExtractor object for each of the chirp sweeps. This should not cause any issues as the sweeps are already filtered at 6kHz during data collection.

We will probably want to also update the chirp_stimulus_codes in the run_chirp_fv_extraction.CollectChirpFeatureVectorParameters class to include the new chirp stimulus codes.

Currently this issue is causing all of feature extraction to crash with a KeyError because the sweep_features dictionary is being returned as an empty dictionary from the extract_sweep_features() function after encountering the ValueError thrown by caluculate_dvdt(). After calling extract_sweep_features, extract_data_set_features tries to iterate over the sweep numbers in cell_features[‘long_squares’][‘subthreshold_sweeps’] to get the ‘peak_deflect’ values and assign it to the corresponding sweep number in the sweep_features dictionary, but because that dictionary has been returned back as an empty dictionary feature extraction crashes with a KeyError on the first attempt through the loop.

Describe the solution you'd like
A solution to this feature request will prevent feature extraction from crashing due to the sampling rate of the new Chirp sweeps by setting the filter None for the chirp sweeps.
We would like the Bessel filter to be bypassed for all for all new chirp sweeps that are filtered at 12.5 K

Describe alternatives you've considered
I don’t see any other solution to this issue other than sampling at a higher rate, but that will produce larger file sizes and we have already started collecting data at 12.5 K for the new chirp sweeps

Additional context
I have attached a jupyter notebook that I used to troubleshoot this issue which contains all of the outputs from the cells which will hopefully make it easy to follow along)

Do you want to work on this issue?
I can help out with this issue but I don’t have the time to develop a solution entirely on my own.

new_chirp_fx_error.zip

[tmchartrand]

I think I have code for this in my chirp update, not positive though. I'll update here as I get that merged in.

[ru57y34nn]

@MatthewAitken even after adding the new chirp stimuli to the EphysRawStimulusNames table we are still encountering the same issue and crash during feature extraction for any experiments that contain these new chirp sweeps. I checked the stimulus_ontology files since adding the new chirps to the EphysRawStimulusNames table and they are now present in the stimulus_ontology file so I am not sure why we still have the same issue. I haven't had a chance to look into this very much right now, other than to just confirm that the issue still exists. Let me know if you have any questions.

Here is a recent example experiment log file:
\allen\programs\celltypes\production\mousecelltypes\prod3670\Ephys_Roi_Result_1239292125\202301121701_EPHYS_FEATURE_EXTRACTION_V3_QUEUE_1239292125.log

error message:
File "/allen/aibs/technology/conda/production/fx/lib/python3.6/site-packages/ipfx/data_set_features.py", line 388, in extract_data_set_features
sweep_features[s['sweep_number']]['peak_deflect'] = s['peak_deflect']
KeyError: 5

[ru57y34nn]

@MatthewAitken I am now unable to install or update ipfx since implementing the fix for this issue and updating to 1.07. Here is the error I am getting:

` Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
C:\Users\rustym\AppData\Local\Temp\pip-install-6_feukf4\statsmodels_fcefc8420f4941de8f614a916fa05115\setup.py:477: SyntaxWarning: "is" with a literal. Did you mean "=="?
if i is "tests":
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "C:\Users\rustym\AppData\Local\Temp\pip-install-6_feukf4\statsmodels_fcefc8420f4941de8f614a916fa05115\setup.py", line 347, in
from numpy.distutils.misc_util import get_info
ModuleNotFoundError: No module named 'numpy'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.`

Is anyone else encountering this issue when attempting to install or update ipfx? Also, let me know if you need more information about the issue.

[smestern]

Is anyone else encountering this issue when attempting to install or update ipfx? Also, let me know if you need more information about the issue.

Apologies coming in here as an external user of ipfx. I actually encountered this issue recently when trying to deploy ipfx to a fresh python install.
The issue is with the dependency 'statsmodels', which requires numpy to be previously installed in the python distro. See here statsmodels/statsmodels#4635. Unfortunately trying to install numpy at the same time as ipfx, e.g. via "pip install -r requirements.txt" causes this issue. If you are able, install numpy prior to trying to install ipfx.

[ru57y34nn]

@smestern thanks for the tip! For some reason this still didn't work with a conda environment for me but switching to a venv environment it did.