Synching Arduino AnalogInput with Video?

[nghess]

Hi,
I've got a workflow going where I need to sample Arduino analog inputs in time with my video frames, then send the combined outputs to a PythonTransform (not shown). I've got it working fairly well, but I'm wondering if there is a cleaner way, that would also make sure I'm sampling the Arduino once per frame. Currently, I'm manually setting a SampleInterval node, and if I sample too frequently, it produces bad frame drops for video capture. So I'm currently setting the interval to something above my video frame rate. Ideally, I'd sample the Arduino only once per frame. I've tried using a Sample node, connected to one of the camera-produce nodes as a trigger it, but that also produces frame drops.

The workflow below works pretty well, but it feels a bit sloppy, so I'm wondering if there are suggestions as to a better way. --In case it's not clear, the nodes on the upper branch are coming from the camera, bottom branch is coming from Arduino. The combined outputs are then sent to a PythonTransform to the right (not pictured).

Thanks for the help!
-Nate

[bruno-f-cruz]

Can you expand a bit what you mean by synchronizing? What jitter are you looking for? Are you just looking to check what is the state of your input line when each frame arrive?

Finally, please include the workflow file.

[nghess]

Hi Bruno,
The goal is to produce a single sample of the AnalogInputs per video frame. I had tried using a Sample node, triggered by the incoming frame, but that didn't work well. the current workflow works for the most part, but the approach is sloppy and I am still experiencing some frame drop.

The frame drops present as large chunks of consecutive dropped frames. The problem is more apparent when the SampleInterval interval is smaller than the framerate. --I'm assuming that the large number of AnalogInput samples is gumming the works. This frame drop issue also shows up for the current SampleInterval interval (00:00:00.1), but not to the extent that it does when the interval is smaller.

I think that the asynchronous sampling between the video frames and Arduino analog samples is creating the problem, but I don't know a good method to make them sample as a team.

Workflow file is attached. Thanks for your help!

task.bonsai.zip

[bruno-f-cruz]

Hi Bruno, The goal is to produce a single sample of the AnalogInputs per video frame. I had tried using a Sample node, triggered by the incoming frame, but that didn't work well. the current workflow works for the most part, but the approach is sloppy and I am still experiencing some frame drop.

Assuming I am understanding this currently, would you simply want to use a WithLatestFrom instead? E.g.:

The frame drops present as large chunks of consecutive dropped frames. The problem is more apparent when the SampleInterval interval is smaller than the framerate. --I'm assuming that the large number of AnalogInput samples is gumming the works. This frame drop issue also shows up for the current SampleInterval interval (00:00:00.1), but not to the extent that it does when the interval is smaller.

Your workflow has a large dependency tree on top of the camera stream that may be leading to some backpressure. A quick look at your "Task" python transform is raising some red flags. Specifically you seem to be creating/allocating new arrays at a potentially large frequency (looking at canvas = create_blank_canvas(img_dims[0], img_dims[1]) and drawing a large amount of primitives on top of it. This is usually ok for most purposes but given your "CombineLatest" operator, the frequency of that stream can be very large. The aforementioned solution may alliviate this backpressure problem.

I think that the asynchronous sampling between the video frames and Arduino analog samples is creating the problem, but I don't know a good method to make them sample as a team.

I dont think that there is a "right way" to go about this. You need to first establish the constrains of your assynchronous streams. Eg. are they hiererchicaly similar? or should one stream empose the sampling frequency of the output stream? Which sequence should determine the sampling? Is it ok to have repeated values from the subordinate sequence? etc... Bonsai provides a rich lexicon to express most these relatioships.

Anyway hope it helps, and happy to keep iterating!

[nghess]

Hi Bruno,
Sorry for the slow reply. I wanted to make sure that the fix was working. But we've now run about a dozen sessions, and the data looks good.

I did implement WithLatest, essentially just as in your illustration, and it solved the issue. -This way the incoming frame gates when the AnalogInput is sampled, and there's no bottleneck.

Apologies for the overcomplicated question and file, for a very simple question.

Appreciate your help, as always!