New Bonsai User, Please Help ASAP - analyzing videos

[steph-ncu]

Hello all,

In my lab, we are running a social interaction/preference/avoidance test with our mice. Essentially what this entails is we have a "bully mouse" that is restricted to a mesh cage against a wall in an arena. Our test mice are allowed to run freely in the arena. There is an interaction zone surrounding the mesh cage and two "corners" that are in the corners opposite the mesh cage (see image below for diagram of experiment).

We recorded a webcam video of the experiments and recorded neuron activity using fiber photometry all through a Bonsai workflow. We used the KeyDown feature to note when the mice went into the arena and the experiment ended. I have the following questions:

The KeyDown file tells us at what second time we pressed the respective key. Because our program marks time continuously throughout each program (as in if one experiment ends at 15.5 s, the next experiment will continue counting from this time), it is hard to match this time up with the video footage. Any recommendations on how to sync our fiber photometry data and the video footage?
With the KeyDown feature, when imputing my fiber photometry data into the Python program I am using to analyze it, should I truncate the data according to the times listed on the KeyDown file?
When trying to analyze the video data, we need to mark the different types of zones so the program can tell us how many times the mouse goes into each zone (two corners and interaction zone). How can I figure out the pixel coordinates to input into my Bonsai program? (see the Bonsai workflow: https://drive.google.com/file/d/1-b0rlSdyaaiVTPYLZ1qOXY9QVbsq5XDe/view?usp=sharing)
In each video, I was recording two arenas with the same webcam simultaneously. Thus, each video contains two separate experimental mice that I want to analyze individually. Is it possible to analyze each experiment/arena all at once? To do this, would I need to add three more "Regions of Interest"?
How do I program the Bonsai workflow to recognize the mouse in the arenas?
When I tried inputting coordinates previously with a test video, the Bonsai program generated an excel file with two columns of numbers and columns of "true/false;" however there were no column titles (attached). Does anyone know how to interpret this with respect to my Bonsai workflow?
Thank you all in advance for your help!
Best,
Stephanie Urod

4460L_behR_FP-video-testrun-062722.2_bonsai.csv
a

[bruno-f-cruz]

Hey welcome to Bonsai :)

Just a preface, next time try to divide your post into multiple smaller ones. It will likely increase the chances of people answering individual questions much quicker. It will also make recycling content easier for the community.

The KeyDown file tells us at what second time we pressed the respective key. Because our program marks time continuously throughout each program (as in if one experiment ends at 15.5 s, the next experiment will continue counting from this time), it is hard to match this time up with the video footage. Any recommendations on how to sync our fiber photometry data and the video footage?

Can you include this workflow? How are you recording the photometry signal? Does this piece of hardware have access to an external digital input? Would a jitter of a couple of dozen of milliseconds be acceptable in your alignment? What camera are you using?

With the KeyDown feature, when imputing my fiber photometry data into the Python program I am using to analyze it, should I truncate the data according to the times listed on the KeyDown file?
Depends on the answer to the question above :P

When trying to analyze the video data, we need to mark the different types of zones so the program can tell us how many times the mouse goes into each zone (two corners and interaction zone). How can I figure out the pixel coordinates to input into my Bonsai program?
Not sure exactly why you need RegionOfInterest. From what you are describing I would just delete it and define the ROIs in the crop node directly. While Bonsai is running you can change the Crop RegionOfInterest property by clicking on the "...". This will open a window with your full image where you can draw a crop.

In each video, I was recording two arenas with the same webcam simultaneously. Thus, each video contains two separate experimental mice that I want to analyze individually. Is it possible to analyze each experiment/arena all at once? To do this, would I need to add three more "Regions of Interest"?
There are multiple post in the forum regarding this topic. I would probably advise you to take a quick look and if you still have doubts feel free to come back here. Here's the first one that appeared on a quick search

How do I program the Bonsai workflow to recognize the mouse in the arenas?
Same as the previous one. Here's one post that might help you out.

When I tried inputting coordinates previously with a test video, the Bonsai program generated an excel file with two columns of numbers and columns of "true/false;" however there were no column titles (attached). Does anyone know how to interpret this with respect to my Bonsai workflow?

CSVWriter will use the order of the tuple coming from Zip operator. In your case it will be a Point2f.X, Point2f.Y Bool Bool Bool, looking at your workflow these will correspond to the centroid position, and the result of your GreaterThan comparison.

If you want to add an header to your csv make sure to check the IncludeHeader property in the CsvWriter node. Additionally, if you want to change the name of the columns you can add an ExpressionTransform with the following code:

new(Item1 as Centroid, Item2 as condition1, Item3 as condition2, Item4 as condition3)
Hope it helps!

[steph-ncu]

Hi Bruno, thank you for your response!

The KeyDown file tells us at what second time we pressed the respective key. Because our program marks time continuously throughout each program (as in if one experiment ends at 15.5 s, the next experiment will continue counting from this time), it is hard to match this time up with the video footage. Any recommendations on how to sync our fiber photometry data and the video footage?

1. workflow: https://drive.google.com/file/d/1iCVy0kNryJF5e-m_A0iA8AKAJORTw5E4/view?usp=sharing
2. we are using a Neurophotometrics F3002 instrument to measure fiber photometry signal.
3. this device is connected to the fiber optic patch cables, as well as our computer, I am unsure if that is what you were asking.
4. ideally, we need the milliseconds to line up exactly so that we can compare the placement of the mouse in the arena to the fiber photometry signal.
5. we are using a Logitech HD Laptop Webcam C615 with Fold-and-Go Design, 360-Degree Swivel, 1080p Camera.

With the KeyDown feature, when imputing my fiber photometry data into the Python program I am using to analyze it, should I truncate the data according to the times listed on the KeyDown file?

1. **based on my answers above

In each video, I was recording two arenas with the same webcam simultaneously. Thus, each video contains two separate experimental mice that I want to analyze individually. Is it possible to analyze each experiment/arena all at once? To do this, would I need to add three more "Regions of Interest"?

1. thank you for the link, it appears i can use the "crop" feature. would you be able to show me a screenshot of how i would use this feature to organize my workflow while still using the "region of interest" to mark the arena?

[steph-ncu]

When I tried inputting coordinates previously with a test video, the Bonsai program generated an excel file with two columns of numbers and columns of "true/false;" however there were no column titles (attached). Does anyone know how to interpret this with respect to my Bonsai workflow?

1. could you elaborate on how to include the coding to edit the column headers?
2. ahead of the true/false columns, there are two sets of numbers. are you suggesting that these are the coordinates of the mouse at each video frame and the true/false at each time point represent if the mouse is within the 3 specific regions of interest?

[steph-ncu]

one more question, is there a way to get a total sum of the number of times when the mouse enters each region of interest, rather than a frame by frame count?

[bruno-f-cruz]

RE: the synchronization of your fiber photometry setup I would need to be familar with the hardware specs which I am not. However, if they are able to record an external signal together with the photometry data I would align everything by sending a short pulse once in a while during the task. The easiest way is to probably generate the pulse using an Arduino or some other microcontroller you might have around. This wire would then connect straight to the fiber photometry setup and, additionally, connect to an LED that would be visible in the camera. Post-hoc, you can then find the same event in the fiber photometry data (i.e. pulse rising edge) and in the camera feed (i.e. LED turning on). Some posts in the forum about this you might want to explore.

RE: Adding keypresses. If you get the last point working, I would just save, for each key press, the closest video frame acquired. In Bonsai this would look like:

RE: Region of interest. Not sure I fully understand. If you have 2 subjects in the arena, you could set two crops, and run the analysis in parallel for each crop. The simplest form of this would be:

SubjectXAnalysis just stands for whatever analysis pipeline you might want to use.

RE: Renaming header. Add an ExpressionTransform node to your workflow. Under properties find the Expression property. Click the ... icon on said field. A new text editor will open and you can add the previous code to this window:

RE: ahead of the true/false columns, there are two sets of numbers. are you suggesting that these are the coordinates of the mouse at each video frame and the true/false at each time point represent if the mouse is within the 3 specific regions of interest? That appears to be the case, yes.

RE one more question, is there a way to get a total sum of the number of times when the mouse enters each region of interest, rather than a frame by frame count?
Yes, one way to detect rising edge events (false->true events) is the following logic:.