Implement Quality Control metrics

[alexpiet]

• Determine what quality control metrics we want to compute and track (hardware, software, mouse behavior)
• Use the AIND Quality Control system to track them: https://aind-data-schema.readthedocs.io/en/latest/quality_control.html

Keeping a compiled list here

QA (Done before data is collected "assure data will be good")

Cameras

• Are there dropped frames?
• Histograms to quantify out-of-focus images

Harp

• Do boards all share the same sync clock?
• Harp logs and firmware?

Harp treadmill

• Encoder captures wheel movement
• Brake default to 0. Can modify input torque.
• Torque sensor baseline.

Harp sniff sensor

• Baseline signal
• Reads temperature changes.

Harp clock

• All boards synch to clock
• Repeater in repeater mode

Harp behavior

• Sound triggers: # Softcode = # Harp triggers
• Water triggers: # Softcode = # Harp triggers
• Sound and water within odor sites.

Harp olfactometer

• Minimum flow met.
• Odor valve open -> End valve open > End valve close> Odor valve close.

Harp lick sensor

• Licks are detected, on off times are realistic.

Speaker

• Verify the speaker amplitude is correct

QC (Done after data is collected "Control for bad data")

• Number of licks in the session
• Histogram of lick response times
• Double-dipping
• Number of trials
• Software: development/correction of bias over past sessions - metric to quantify
• Behavior: Simple RL index to track behavior improvement
• session length (in time and trials)
• trials completed (The proportion of ignored trials throughout a session (may exclude ignored trials at the end of the session)
• cross talk licking
• see trends in licking, to asses for lick detection problems
• Camera illumination, orientation, frame drops

FIP

• data complete? all csv, all binary data complete?
• data length correct? all same lenght?
• ROI looks ok?
• no obvious artefact? e.g. fiber patch-cable disconnection, movement, etc
• Implement QC Capsule aind-fip-dff#18

VR foraging

• Total distance traveled
• Min and max velocity
• Session length
• Minimum sniff cycles

[ellahiltonvano]

Somewhat related to https://github.com/AllenNeuralDynamics/aind-behavior-blog/issues/599

[JeremiahYCohen]

QA:

1. Cameras
   o Are there dropped frames?
   o Histograms to quantify out-of-focus images
2. Harp
   o Do boards all share the same sync clock?
   o Harp logs and firmware?

QC:

1. Number of licks in the session
2. Histogram of lick response times
3. Double-dipping
4. Number of trials

[hagikent]

For FP

QA:
1.data complete? all csv, all binary data complete?
2.data length correct? all same lenght?
3.ROI looks ok?
4.no obvious artefact? e.g. fiber patch-cable disconnection, movement, etc
5.

@alexpiet @hanhou @JeremiahYCohen
How would you see QA to be organized? I can naively picture putting all into one light-weight .py file that generates plots+metrics, and GUI triggers it upon completion of each session?

__
QC: Managed here:
https://github.com/orgs/AllenNeuralDynamics/projects/75/views/1?filterQuery=QC&pane=issue&itemId=86115745
https://github.com/AllenNeuralDynamics/aind-physio-arch/issues/53

[ellahiltonvano]

Hi all, as listed on our action items please comment on this issue and see the related RACI chart (mentioned above in issue 599). Comment on this issue and check off your name on the action items discussion.
@ZhixiaoSu @KanghoonJ @galenlynch @jasonyslee @bwtan-allen @XX-Yin

[galenlynch]

Looks good...

[bwtan-allen]

QC:

Software: development/correction of bias over past sessions - metric to quantify
Behavior: Simple RL index to track behavior improvement

[alexpiet]

QC -

• session length

• trials completed

[tiffanyona]

QA before data acquisiton, QC after data acquisition (assumes QA was passed)

QA
Harp treadmill

• Encoder captures wheel movement
• Brake default to 0. Can modify input torque.
• Torque sensor baseline.

Harp sniff sensor

• Baseline signal
• Reads temperature changes.

Harp clock

• All boards synch to clock
• Repeater in repeater mode

Harp behavior

• Sound triggers: # Softcode = # Harp triggers
• Water triggers: # Softcode = # Harp triggers
• Sound and water within odor sites.

Harp olfactometer

• Minimum flow met.
• Odor valve open -> End valve open > End valve close> Odor valve close.

Harp lick sensor

• Licks are detected, on off times are realistic.

QC

• Total distance traveled
• Min and max velocity
• Session length
• Minimum sniff cycles

[KanghoonJ]

QC

• The proportion of ignored trials throughout a session (may exclude ignored trials at the end of the session).
• Total run time

[jasonyslee]

I haven't run mice yet and can't honestly comment critically.
When I get Force Foraging behavior running I will come back and add QC metrics for that.
So for now... LOOKS GOOD.

[XX-Yin]

QC:
early licking rate in ITI, delay

[ellahiltonvano]

Linking https://github.com/AllenNeuralDynamics/aind-behavior-blog/issues/700